Allow for indexing enhancements through included hooks

[vinistock]

Motivation

This PR is the first step for allowing addons to enhance the indexing mechanism and a proposal of how we could handle included hooks.

Example usages Shopify/ruby-lsp-rails#422, Shopify/ruby-lsp-rails#423

What does a concern do?

For context, in order to make a module into a concern one has to extend ActiveSupport::Concern. Upon extending, the concern module will then dynamically define an included hook into the module that has extended it. Finally, the included hook will dynamically extend any ClassMethods module that exists inside of the original module.

module Concern
  def self.extended(mod)
    mod.define_singleton_method(:included) do |other_module|
      other_module.extend(mod.const_get(:ClassMethods))
    end
  end
end

So we need a way to

• Handle the extend event, so that we can remember that a module has been made a concern
• Provide a way to run included callbacks during ancestor linearization, so that the ancestor chain can be modified dynamically

Other use cases

I have also considered other use cases, like creating a new instance method during an extend or handling something like belongs_to (which creates new methods on the receiver).

module Concern
  def self.extended(base)
    base.define_method(:new_method) {}
  end
end

Naming

I ended up going with "indexing augmentations" for the name of concept. The idea is to avoid using extension and plugin (too associated with editors) or addon (now associated with the language server).

Do we agree with the naming? Is indexing enhancement better? Indexing contribution?

Implementation

The implementation has a few parts:

1. We need a way to register for augmentations and those have to be passed down to the declaration listener, so that we can invoke the events as needed
2. As with most things, we want to expose the minimum amount of methods to addons to help guide authors in their implementations. I created an interface that any object can implement, containing the only events we want to allow
3. Finally, the bulk of the logic and decisions are part of the included hooks. The augmentations will need to be able to register lazy included hooks, because during indexing we have no guarantees that we won't find an include for a concern that hasn't been found yet. We also have no guarantees that we will know which module an include refers to before we finished indexing. The idea is to allow registering blocks that get invoked once per linearization. These hooks should mutate the index with the extra information they want to include. The only time we run these blocks again is if the ancestor cache gets busted, which is exactly what we want otherwise we would lose the information forever

Notes

I believe we do not have to support lazy extended, prepended or inherited hooks. The reason is because the concern pattern is broadly used to mix instance and class methods in a single operation. You include one module and then you get everything you need.

The opposite pattern of extending a module and then getting a dynamic include is not really common and would essentially just be the exact same thing as a concern.

And inheriting from a class already carries both instance and class methods, so there's also reduced need for meta-programming on that side.

We may discover other scenarios later, but I think it should be fine to handle included hooks only for now.

Automated Tests

Added tests.

Manual Tests

There's no way to test this without the changes being made to the Rails addon too. I did it locally and recorded it in action in a toy app.

augmentation_demo.mov

[jacobdaddario]

Fantastic bit of technical writing as always. Excited to see this go.

Do we agree with the naming? Is indexing enhancement better? Indexing contribution?

I think that one of the latter two choices may be better depending on the technique used. If customizations are modifying the indexing techniques, then I think enhancement feels more appropriate. If the customizations are adding directly on to the Index instance, then contribution feels more appropriate. Based on your PR description, it sounds like the first option is closer to the case.

EDIT:
Hmmm, looks like it can do both. augmentation might be the best name in that case.

[andyw8]

Another possible name - supplement. Here's what GPT says:

---

The terms "supplement" and "augmentation" both refer to the process of adding something, but they are used in slightly different contexts and have nuanced meanings.

Supplement

• Definition: To supplement something means to add to it in order to complete it, enhance it, or make up for a deficiency.
• Context: Often used when referring to adding something that is lacking or to improve the overall quality or completeness of something.
• Example: Taking vitamin supplements to ensure you get all the necessary nutrients that might be missing from your diet.

Augmentation

• Definition: To augment something means to increase its size, value, or effectiveness by adding something to it.
• Context: Typically used when the goal is to make something larger or more effective, rather than just completing it.
• Example: Augmenting a software application with new features to improve its functionality.

Key Differences

• Purpose: Supplementing is often about filling gaps or enhancing completeness, while augmenting is about increasing or amplifying.
• Usage: Supplement is more commonly used in contexts like nutrition, education, and information. Augmentation is often used in contexts like technology, medical procedures, and performance enhancement.

In summary, while both terms involve adding something, "supplement" focuses on completion and enhancement, whereas "augmentation" focuses on increasing and amplifying.

[st0012]

I'll test the API with the RSpec addon too.

ruby-lsp-rspec-definition-demo.mp4

[st0012]

For naming, it would be great to have a more approachable name. I think supplement is better than augmentation, but it's a bit rare to see it in software context too (from my limited experience). So enhancement will be my favorite.

[vinistock]

Switched the name to enhancements.

[st0012]

I added basic error reporting around enhancement so when it encounters an error, the file's indexing can still be completed and we can get some messages for debugging, like:

2024-08-05 15:22:46.219 [info] (ruby-lsp-rspec) Error occurred when indexing /Users/hung-wulo/src/github.com/st0012/ruby-lsp-rspec/spec/ruby_lsp_rspec_spec.rb with 'RubyLsp::RSpec::IndexingEnhancement' enhancement: 'undefined method `foo' for an instance of Prism::ArgumentsNode'

Backtrace: /Users/hung-wulo/src/github.com/st0012/ruby-lsp-rspec/lib/ruby_lsp/ruby_lsp_rspec/indexing_enhancement.rb:29:in `on_call_node'  
/Users/hung-wulo/src/github.com/Shopify/ruby-lsp/lib/ruby_indexer/lib/ruby_indexer/declaration_listener.rb:295:in `block in on_call_node_enter'  
/Users/hung-wulo/src/github.com/Shopify/ruby-lsp/lib/ruby_indexer/lib/ruby_indexer/declaration_listener.rb:294:in `each'  
/Users/hung-wulo/src/github.com/Shopify/ruby-lsp/lib/ruby_indexer/lib/ruby_indexer/declaration_listener.rb:294:in `on_call_node_enter'  
/Users/hung-wulo/.gem/ruby/3.3.3/gems/prism-0.30.0/lib/prism/dispatcher.rb:216:in `block in visit_call_node'  
/Users/hung-wulo/.gem/ruby/3.3.3/gems/prism-0.30.0/lib/prism/dispatcher.rb:216:in `each'  
/Users/hung-wulo/.gem/ruby/3.3.3/gems/prism-0.30.0/lib/prism/dispatcher.rb:216:in `visit_call_node'  
/Users/hung-wulo/.gem/ruby/3.3.3/gems/prism-0.30.0/lib/prism/node.rb:2478:in `accept'  
/Users/hung-wulo/.gem/ruby/3.3.3/gems/prism-0.30.0/lib/prism/visitor.rb:31:in `block in visit_child_nodes'