AEP2: Scope rework with @state & deterministic

[Finistere]

Current Problem

Current scopes are non-trivial to use as the scope must be propagated to all downstream dependencies. It becomes really complex when manipulating multiple scopes...

from antidote import lazy, injectable, world

REQUEST_SCOPE = world.scopes.new(name='request')

@lazy(scope=REQUEST_SCOPE)
def current_request() -> object:
    return object()

@injectable(scope=REQUEST_SCOPE)
class Dummy:
    def __init__(self, request: object = current_request()) -> None:
        self.request = request

Proposal

This could be simplified by introducing two new concepts instead: a state dependency which Antidote could track for changes and deterministic dependencies only re-computed when their own state dependencies changed.

Example

from antidote import state, inject, injectable

# request will be cached until reset() is called.
# the decorated function will be used as a factory.
@state
def current_request() -> object:
    ...

current_request.reset()

@lazy(deterministic=True)
def f(req = current_request()) -> object:
    return req

@injectable(deterministic=True)
class Dummy:
    def __init__(self, req = current_request()) -> None:
        ...

current_request.call()  # actual call to the wrapped function like @lazy, for testing purposes.
current_request.__wrapped__  # wrapped function

Spec

from typing import Callable, ParamSpec, Protocol, TypeVar

T = TypeVar("T")
P = ParamSpec("P")


class StateWrapper(Protocol[P, T]):
    @property
    def __wrapped__(self) -> Callable[P, T]:
        ...

    def __call__(self) -> T:
        ...

    def call(self, *args: P.args, **kwargs: P.kwargs) -> T:
        ...
    
    def reset(self) -> None:
        ...


def state(__func: Callable[P, T]) -> StateWrapper[P, T]:
    ...

• Current scopes won't exist anymore. They will be deprecated and replaced by the notion of state.
• A function decorated with @state defines a dependency which is cached until reset() is called.
• A dependency can be declared as being deterministic, mutually exclusive with singleton. If so, it'll be cached and only re-computed whenever any of its state dependencies change.
• A dependency declared as deterministic without any state dependencies will raise an error.
• @lazy(deterministic=True) will check all arguments of the underlying function.
• @injectable(deterministic=True) will check all arguments of __init__() and of the factory_method if defined.
• A singleton cannot depend on any state nor any deterministic dependencies.
• @state will apply @inject to the underlying function
• No arguments can be provided to the @state decorated function, so no parametrization like @lazy is possible.

[pauleveritt]

This is pretty wild. In a good way. 😀

1. Any nesting? I was using parent registries to allow overriding something for the duration of a request. But I'm not sure if I really needed it.

2. Is the forward-relationship available? Meaning, if I knew current_request changed, could I recurse a tree and regenerate dependents?

[Finistere]

Any nesting? I was using parent registries to allow overriding something for the duration of a request. But I'm not sure if I really needed it.

Supposing in your registry you were overriding A with B for requests. In Antidote, at first sight, you would use a common interface between A and B and use predicate/constraint to determine given the request which one to use.

I've re-read your explanation in #46 and being able to override dependencies on the fly doesn't look easily maintainable to me intuitively. It's good for testing, but it seems too easy to hide logic deep within functions. in #45 there is a discussion on how to manage multiple registries/catalogs/containers. But more in the spirit of library/framework authors defining a set of dependencies which you may include or not in a (more) static way. #45 can also obviously become quite complex, so I hope to limit that complexity whenever possible.

Is the forward-relationship available? Meaning, if I knew current_request changed, could I recurse a tree and regenerate dependents?

If a deterministic dependency A depends on current_request, then whenever current_request resets, Antidote will automatically recompute A. This behavior is also enforced for any deterministic dependencies B depending on A.

Given a deterministic dependency R, we have a tree of dependencies of which it's the root. @state are necessarily leaf nodes that Antidote will track, and deterministic dependencies will be re-computed whenever necessary. So in the following example, whenever state S1 changes, D11, D1, D31, D3, and R would be re-computed at the next injection.

   ┌──► D1 ───► D11 │
   │                │
   ├──► D2          ├──► S1
R  │                │
   │       ┌──► D31 │
   └──► D3 │
           └──► S2

For any dependency, Antidote would track all of the leaf nodes, @state, it relies upon. So S1 and S2 for R. When any of those changed since the last computation of R, Antidote would traverse the dependency tree to re-compute whatever is necessary and R.

[pauleveritt]

1. I agree, dynamic "nested" registries and overrides are dangerous. Static, compile-time is the way to go.
2. That is so much more than I could have asked. I'm legit excited for what you are thinking of.

[Finistere]

Is it enough for the reactive components you're thinking of though? Obviously, you would need to play with an actual implementation, but as presented do you see cases that would not be addressed?

[pauleveritt]

It probably is, but I will need some design advice from you to make it fit.

First simple question: triggering the injection. For example, a Markdown file changes, file watcher notices, reads the file, sees it has changed. a MarkdownFile instance was depending on that file. And after that, tons of things that ultimately generate a .html build file.

What's the way to trigger the first injection?

Larger question...these reactive components might have a lot in between observer and observable. E.g. persisting to disk, caching, validation. People might register custom before/after hooks. Is this like a custom Provider in Antidote?

The component writer doesn't write such things. It could be in a Component base class. OTOH, I'm hoping with protocols to not really do subclassing.

[Finistere]

It could look like this:

from antidote import inject, state, State


class Watcher:
    _path_to_state: dict[str, State]

    def __getitem__(self, path: str) -> State:
        try:
            return self._path_to_state[path]
        except KeyError:
            @state
            def read_content() -> str:
                with open(path) as f:
                    return f.read()

            self._path_to_state[path] = state
            return state

    def _on_change(self, path: str) -> None:
        self._path_to_state[path].reset()


watcher = Watcher()


@inject
def f(my_file=watcher['/tmp/my-file.md']) -> None:
    ...

For something like Sphinx which parses files you would probably define a deterministic injectable class for each file with its content injected. You would need to keep track of the association between each class and each file.

class MarkdownFile:
    pass


files: dict[str, Type[MarkdownFile]] = {}

for path in []:
    @injectable(deterministic=True, factory_method='load')
    @dataclass
    class CurrentFile(MarkdownFile):
        @classmethod
        def load(cls, content: object = watcher['tmp/my-file.md']) -> CurrentFile:
            ...
        
        _references: list[str] = ...  # parsed from the content
        
        @property
        def references(self) -> list[MarkdownFile]:
            return [world.get(files[ref]) for ref in self._references]

    files[path] = CurrentFile

To be clear, I've no idea if it really solves your issue and whether this API would be fine to work with as a developer and library author. It's a bit unclear to me how you intend to work with files in your project in terms of API. So feel free to create some science-fiction on how you would like to use it.^^

[pauleveritt]

Hmm, I worry that my need for persistent will be a deal-breaker.

When you say: So in the following example, whenever state S1 changes, D11, D1, D31, D3, and R would be re-computed

If S1 relates to the object at D11, restarting the process breaks that relation. This means some kind of path string is needed for that, and some kind of resolver from path to state. I have a feeling that's not what you mean, which would be reasonable.

[Finistere]

You do need a resolver from the path to state with the current proposal yes. Persistence is not part of this proposal. I don't know what to think of it currently. The persistence of state is either complex because of all the gotchas of object serialization or domain-specific. Intuitively a long-lived watcher/daemon/service keeping the state alive and responding to requests would be more effective.

[pauleveritt]

Regarding science fiction...where on the spectrum between:

1. As tiny as possible, so you can quickly load into your brain, only a couple of concepts.

2. Larger, covering more practical usages, which combine with each other.

As a note, Sphinx gets around this through the magic of pickle. 😀

[Finistere]

Probably more 1). The goal for me would be to understand your use case when working with external states such as files. I didn't see any example in your library on how you would interact with files typically. For Flask request or equivalent, the current proposal would work fine for example.