Merge pull request #701 from Shopify/vs/add_semantic_highlighting_doc…

…umentation

Add semantic highlighting documentation

README.md

@@ -39,6 +39,9 @@ end
 See the [documentation](https://shopify.github.io/ruby-lsp) for more in-depth details about the
 [supported features](https://shopify.github.io/ruby-lsp/RubyLsp/Requests.html).
 
+For creating rich themes for Ruby using the semantic highlighting information, see the [semantic highlighting
+documentation](SEMANTIC_HIGHLIGHTING.md).
+
 ## Learn More
 
 * [RubyConf 2022: Improving the development experience with language servers](https://www.youtube.com/watch?v=kEfXPTm1aCI) ([Vinicius Stock](https://github.com/vinistock))

SEMANTIC_HIGHLIGHTING.md

@@ -0,0 +1,37 @@
+# Semantic highlighting
+
+The Ruby LSP supports semantic highlighting. This feature informs editors about the right token types for each part of
+the code to allow for rich and accurate highlighting. The strategy taken by the Ruby LSP is to only return tokens for
+syntax that is ambiguous in Ruby (as opposed to all existing tokens) to optimize for performance.
+
+An example of ambiguous syntax in Ruby are local variables and method calls. If you look at this line in isolation:
+```ruby
+foo
+```
+it is not possible to tell if `foo` is a local variable or a method call. It depends on whether `foo` was assigned to
+something before or not. This is one scenario where semantic highlighting removes the ambiguity for themes, returning
+the correct token type by statically analyzing the code.
+
+To enhance a theme's Ruby syntax highlighting using the Ruby LSP, check the information below. You may also want to
+check out the [Spinel theme](https://github.com/Shopify/vscode-shopify-ruby/blob/main/themes/dark_spinel.json) as an
+example, which uses all of the Ruby LSP's semantic highlighting information.
+
+## Token types
+
+According to the LSP specification, language servers can either use token types and modifiers from the [default
+list](https://microsoft.github.io/language-server-protocol/specification#semanticTokenTypes) or contribute new semantic
+tokens of their own. Currently, the Ruby LSP does not contribute any new semantic tokens and only uses the ones
+contained in the default list.
+
+## Token list
+
+| Syntax  | Type.Modifier | Note |
+| ------------- | ------------- | ------------- |
+| Sorbet annotation methods such as `let` or `cast`  | type | Not every annotation is handled |
+| Method calls with any syntax  | method | |
+| Constant references (including classes and modules)  | namespace | We don't yet differentiate between module and class references |
+| Method definition  | method.declaration | |
+| self  | variable.default_library | |
+| Method, block and lambda arguments | parameter | |
+| Class declaration | class.declaration | |
+| Module declaration | class.declaration | |

lib/ruby_lsp/requests/semantic_highlighting.rb

@@ -347,7 +347,7 @@ def visit_class(node)
       def visit_module(node)
         return super unless visible?(node, @range)
 
-        add_token(node.constant.location, :class, [:declaration])
+        add_token(node.constant.location, :namespace, [:declaration])
         visit(node.bodystmt)
       end
 

test/expectations/semantic_highlighting/module_and_reference.exp.json

@@ -0,0 +1,19 @@
+{
+    "params": [],
+    "result": [
+        {
+            "delta_line": 0,
+            "delta_start_char": 7,
+            "length": 3,
+            "token_type": 0,
+            "token_modifiers": 1
+        },
+        {
+            "delta_line": 3,
+            "delta_start_char": 0,
+            "length": 3,
+            "token_type": 0,
+            "token_modifiers": 0
+        }
+    ]
+}