my subclass returns a different type for a method that's only on the superclass. how to type?

[zzzeek]

edit: ignore this Q. or not, if helpful for someone. the answer is use TYPE_CHECKING blocks. see below

Hey there -

So I think I'm sort of looking at #269 but it's not clear. The examples shown are micro versions of what we are trying to do in SQLAlchemy, but even these examples are still a little complicated.

the problem is the pattern works completely great in pyright and pylance, but mypy not at all, not any flags I can find that make it accept it, but mypy still returns the correct answer at the end, suggesting it does understand what I'm doing. it makes sense and I dont see any other way to do it, other than just keeping .pyi files around so that we can have all methods annotated as what we want them to be without having to actually re-implement them everywhere, which would cause measurable performance impact by having hundreds of methods re-implemented, calling upon super().method() and cast() for every call, which is very wasteful.

First is the code without the additional method annotations. Issues are that subclasses need to use cast() when they redefine a method, methods that return a subtype as a result of being on the subclass aren't able to report that they return this more specific type unless we were to provide all new concrete implementations, of which there would be many dozens of them all performing more poorly.

Also note that the "Self" pattern described at pep-673 is not appropriate here - the actual subtypes that are returned by the methods here are not the same type as "self" (example in SQLAlchemy: Column.__add__(5) returns BinaryExpression, not Column).

# example 1 - minimal typing, doesn't do what we want very well

import typing
from typing import Any


class Operator:
    """represents a SQL or Python operator.

    Like the + sign (__add__()), == sign (__eq__()), a method like
    ".concat()", etc.

    """
    pass


add = Operator()
concat_op = Operator()
# ... lots more operators

class AbstractOperatorThing:
    """base of the operator system.

    Has a few operators on it in the real version.

    """


    def operate(self, op: Operator, other: Any) -> "AbstractOperatorThing":
        raise NotImplementedError()


class MoreAbstractOperatorThing(AbstractOperatorThing):
    """second level class.

    Has all the rest of the operators defined in an abstract way,
    all going through the .operate() method.

    """
    def __add__(
        self: "MoreAbstractOperatorThing", other: Any
    ) -> "MoreAbstractOperatorThing":
        # fails to type correctly without an explicit cast, which we'd rather not have
        return self.operate(add, other)

    def concat(self, other: Any) -> "MoreAbstractOperatorThing":
        # fails to type correctly without an explicit cast, which we'd rather not have
        return self.operate(concat_op, other)

    # ... lots more operations

class ConcreteThing(MoreAbstractOperatorThing):
    """concrete implementation.

    Only defines a new operate() method, that sends off the given Operator
    and arguments into some other internal system to produce a new object.

    This is like your Column() in SQLAlchemy (and everything else that
    is a SQL expression).

    """
    # ... no operations are redefined here.  We get __add__, concat, etc
    # from the superclass.

    def operate(self, op: Operator, other: Any) -> "ConcreteThing":
        # assume something happens here with Operator so that we
        # return a ConcreteThing
        return ConcreteThing()


s1 = ConcreteThing()

# ConcreteThing.__add__(5) -> ConcreteThing
expr = s1 + 5

if typing.TYPE_CHECKING:
    # We want the revealed type is "test5.ConcreteThing".
    # with no annotations as above, it's a MoreAbstractOperatorThing.
    # that's the wrong answer (not specific enough).
    reveal_type(expr)

What I'd want in an amazing world, but I dont think this is possible, is to have annotations in ConcreteThing that show the method overrides, but without having to implement them, like this:

# amazing syntax that I dont think exists version

class ConcreteThing(MoreAbstractOperatorThing):

    # like a .pyi annotation in the source code, but doesn't actually get run
    def __add__(
        self: "ConcreteThing", other: Any
    ) -> "ConcreteThing":
        ...

    def concat(self: "ConcreteThing", other: Any) -> "ConcreteThing":
        ...

    def operate(self, op: Operator, other: Any) -> "ConcreteThing":
        # assume something happens here with Operator so that we
        # return a ConcreteThing
        return ConcreteThing()

but the above doesn't work. __add__() is a real method and apparently ... means the same as pass, it returns None.

So here's what we are trying! and it works totally great in pylance and pyright. mypy does not like it. So...is this we're doing it wrong and pylance will eventually report as an error, or is mypy coming up short?

# example 3 - works completely in pylance.  however has the unusual pattern of 
# putting @overrides for a subclass on the superclass.  this is weird but I see no other way
# to do this without changing the actual implementation, or just keeping .pyi files around.

import typing
from typing import Any
from typing import overload


class Operator:
    pass


add = Operator()
concat_op = Operator()


class AbstractOperatorThing:
    # pylance / pyright are totally OK with these - no errors, correct answer
    #
    #
    # mypy says:
    # dev/sqlalchemy/test5.py:16: error: The erased type of self "test5.ConcreteThing" is not a supertype of its class "test5.AbstractOperatorThing"
    # dev/sqlalchemy/test5.py:20: error: The erased type of self "test5.MoreAbstractOperatorThing" is not a supertype of its class "test5.AbstractOperatorThing"
    # dev/sqlalchemy/test5.py:29: error: Self argument missing for a non-static method (or an invalid type for self)

    # I have tried other ideas here but all produce errors of some kind;
    # if i remove the @overloads here, then I get
    # "error: Self argument missing for a non-static method (or an invalid
    # type for self)" for operate() here, due to the existence of the subclass.

    # however, mypy *understands the intent* of the overrides by getting
    # the correct answer at the end.

    @overload
    def operate(
        self: "ConcreteThing", op: Operator, other: Any
    ) -> "ConcreteThing":
        ...

    @overload
    def operate(
        self: "MoreAbstractOperatorThing", op: Operator, other: Any
    ) -> "MoreAbstractOperatorThing":
        ...

    @overload
    def operate(
        self: "AbstractOperatorThing", op: Operator, other: Any
    ) -> "AbstractOperatorThing":
        ...

    def operate(self, op: Operator, other: Any) -> "AbstractOperatorThing":
        raise NotImplementedError()


class MoreAbstractOperatorThing(AbstractOperatorThing):
    @overload
    def __add__(self: "ConcreteThing", other: Any) -> "ConcreteThing":
        ...

    @overload
    def __add__(
        self: "MoreAbstractOperatorThing", other: Any
    ) -> "MoreAbstractOperatorThing":
        ...

    def __add__(
        self: "MoreAbstractOperatorThing", other: Any
    ) -> "MoreAbstractOperatorThing":
        # no cast() is needed here because we have the @overload on
        # AbstractOperatorThing
        return self.operate(add, other)

    @overload
    def concat(self: "ConcreteThing", other: Any) -> "ConcreteThing":
        ...

    @overload
    def concat(self: "MoreAbstractOperatorThing", other: Any) -> "MoreAbstractOperatorThing":
        ...

    def concat(self, other: Any) -> "MoreAbstractOperatorThing":
        return self.operate(concat_op, other)

    # many more operations

    # would prefer not to re-define operate() here, like this


class ConcreteThing(MoreAbstractOperatorThing):

    # all the MoreAbstractOperatorThing operations need to work

    # can't define them here without re-implementing them all (!)
    # using either super() or copying the implementation + a cast(), this
    # is extremely verbose and adds measurable performance overhead as well

    # cant define overrides for them here either because they are not
    # part of this class

    def operate(self, op: Operator, other: Any) -> "ConcreteThing":
        # assume something happens here with Operator so that we
        # return a ConcreteThing
        return ConcreteThing()


s1 = ConcreteThing()

# ConcreteThing.__add__(5) -> ConcreteThing
expr = s1 + 5

if typing.TYPE_CHECKING:
    # type checker knows this is ConcreteThing. so while it errors for
    # the annotations, it still understands them?

    # Revealed type is "test5.ConcreteThing"
    reveal_type(expr)

so what am I trying to do here? any ideas?

[zzzeek]

oh wow, I can put those just in TYPE_CHECKING blocks, and be done with it. cant I. OK.

for anyone reading, just like this:

# amazing syntax version

class ConcreteThing(MoreAbstractOperatorThing):

    if typing.TYPE_CHECKING:
        def __add__(
            self: "ConcreteThing", other: Any
        ) -> "ConcreteThing":
            ...

        def concat(self: "ConcreteThing", other: Any) -> "ConcreteThing":
            ...

    def operate(self, op: Operator, other: Any) -> "ConcreteThing":
        return ConcreteThing()

no @overload needed or any breakage of abstraction.