Current Approach

Currently, we use matchpy to support pipeline optimization. A set of replacement rules are defined in pyterrier.transformer.rewrite_rules, which all get executed when Transformer.compile() is called.

Transformer.compile is not used heavily, though it can be important for getting the most out of pipelines.

Problems with the current approach

The current approach has several downsides:

• It decouples the logic for applying combination rules from the transformers themselves. I.e., I'd expect a ComposedPipeline to define how it can be combined/optimized, not some other place in the code.
• It requires maintaining the global rewrite_rules variable.
• It introduces additional complexity into the class hierarchy, with certain transformers defined as matchpy.Operations with an arity attribute.
• Overall, matchpy feels like overkill for our needs, which are relatively simple combination rules. Using it introduces a dependency that we probably don't need.

Proposed Solution

I propose dropping matchpy in favor of defining optimization logic in individual transformers. This will be faciliated with three methods:

• Transformer.compile(self) -> Transformer -- existing method, but now individual transformers are responsible for implementing it. By default, it returns the transformer itself, but can be overridden with other functionality, such as merging ComposedPipeline together.
• Transformer.fuse_left(self, left: Transformer) -> Optinal[Transformer] -- Fuses this transformer with a transformer that immediately precedes this one in a pipeline. The new transformer should have the same effect as performing the two transformers in sequence, i.e., pipeline_unfused and pipeline_fused in the following example should provide the same results:

pipeline_unfused = left >> self
pipeline_fused = self.fuse_left(left)

• Transformer.fuse_right(self, right: Transformer) -> Optinal[Transformer] -- Similar to fuse_left but for a transformer that immediately follows this one.

I'd expect that most pipelines wouldn't need to implement any of these methods. Perhaps only ComposedPipeline would define compile, which would be responsible for performing the merging logic, something like:

def compile(self) -> Transformer:
    out = deque()
    inp = deque(tuple(chain.from_iterable((t.models if isinstance(t, ComposedPipeline) else (t,)) for t in self.models)))
    while inp:
        right = inp.popleft()
        if out and (fused_right := out[-1].fuse_right(right)):
            out.pop()
            inp.appendleft(fused_right)
        elif out and (fused_left := right.fuse_left(out[-1])):
            out.pop()
            inp.appendleft(fused_left)
        else:
            out.append(right)
    return ComposedPipeline(*out)

An example implemetnation of fuse_left that merges a BatchRetrieve into a FeaturesBatchRetrieve, using the current logic:

class FeaturesBatchRetrieve:
  ...
  def fuse_left(self, left: pt.Transformer) -> pt.Transformer:
      if isinstance(left, BatchRetrieve) and if self.indexref == left.indexref:
          controls = dict(controls)
          controls['wmodel'] = left.wmodel
          return FeaturesBatchRetrieve(
              self.indexref,
              self.features,
              controls,
              self.properties,
              self.threads,
          )