runtime inspection of GenericAlias / typing._GenericAlias as an acceptable practice

[zzzeek]

towards trying to finalize all the typing approaches we want to use with SQLAlchemy, we can come up with things that work but sometimes it's not clear if it's a good idea based on the intent of the constructs in use.

Ill present the API in a neutral way, suppose a function that is given a type, that is, a Python class, and returns some kind of object (here CollectionThing) which refers to a collection of objects of those types:

from typing import Any
from typing import Generic
from typing import List
from typing import Type
from typing import TypeVar


_T = TypeVar("_T", bound=Any)


class CollectionThing(Generic[_T]):
    ...

def collection_of_objects_one(cls: Type[_T]) -> CollectionThing[List[_T]]:
    ...


class SomeClass:
    pass

# (variable) thing: CollectionThing[List[SomeClass]]
thing = collection_of_objects_one(SomeClass)

however, collection_of_objects also accepts an optional parameter collection_cls where we can change the return type to hold onto another kind of collection inside of CollectionThing, like a Set. So without doing much w the typing, we can have something that is not very easy to type like this:

def collection_of_objects_two(
    cls: Type[_T], collection_cls: Type = None
) -> CollectionThing[Collection[_T]]:
    ...

# (variable) thing: CollectionThing[Collection[SomeClass]]
thing = collection_of_objects_two(SomeClass, collection_cls=set)

I'm using Collection because it's common to List, Set, Dict, and most any other kind of collection thing. But to get an exact type, like CollectionThing[Set[SomeClass]], with the above approach there's not too easy of a way. I can kind of get it sort of using a lot of @overload functions, but I can't get it to work fully, and I don't like how it looks anyway, and the overload approach can only work for a fixed series of functions. The "not possible" way would be to use higher kinded types discussed in #548:

_T = TypeVar("_T", bound=Any)
_TC = TypeVar("_TC", bound=Any)

# not possible w/ current typing
def collection_of_objects_three(
    cls: Type[_T], collection_cls: Type[_TC] = None
) -> CollectionThing[_TC[_T]]:
    ...

so without the @overload approach and without higher kinded types, or with the @overload approach (if it could be made to work) but the user wants to use custom collections, you are stuck with code like this:

# use a cast.  will not be popular
foo = cast(
    CollectionThing[Set[MyClass]], 
    collection_of_objects_two(MyClass, collection_class=set)
)

So as the title implies, there's a neato nifty way to make a new API here that is totally succinct, eliminates the need for the user to be aware of the CollectionThing class which is a very desirable goal for the general case, and it's easier to read. but it breaks one rule, or at least, does a thing that I have not seen anyone doing even though it's not very hard, and that is:

• use a GenericAlias , or for Python 3.7 compat a typing._GenericAlias inline within the API

that is, the typing API is just this:

def collection_of_objects_four(cls: Type[_T]) -> CollectionThing[_T]:
    ...

# (variable) foo3: CollectionThing[set[SomeClass]]
foo3 = collection_of_objects_four(set[SomeClass])

# (variable) bar3: CollectionThing[List[SomeClass]]
bar3 = collection_of_objects_four(List[SomeClass])

so above, we just forget about "collection_class" and consider that since collection_of_objects accepts a type as the argument, List[SomeClass] is also a type. The runtime that previously looked at collection_class now peeks at __origin__ and __args__ of the incoming type, as this is in reality a SQLAlchemy relationship() which accepts a string class name, it looks for a ForwardRef also and gets that string from __forward_arg_ if that's what was sent.

When i prototyped that, I noticed that it's not super friendly to detect typing._GenericAlias or types.GenericAlias, both because there's some API change from Py 3.7 on forward, I dont think I want to be using isinstance() here, I notice there is no inspect.is_generic_alias(obj), so while I can get by with peeking for __origin__ and making some other guesses, there's kind of a feeling of "thou shouldst not be here". SQLAlchemy is not a typing tool. But it very much deals with types and getting information about types.

Basically we're looking at I would be publishing an API that's either like, "hey now you can do relationship(List["Address"]) or maybe more specialized like related_collection(List["Address"]) to get the correct typing for the left side automaticaly (which looks like Mapped[List[Address]]).

other ways to go would be the pydantic / dataclasses thing, where we get the typing information almost exclusively from the left hand "typed" side, but for us that is a more dramatic API change and while I might look into that, I dont want that to be the "canonical" way this works. I want folks to stick as much with the same SQLAlchemy API they always had, but introduce small syntactical changes they can make so that their mapped classes are fully typed without any plugins.

So basically the Q is am I going against the spirit of GenericAlias by using them at runtime in this way. thanks for reading!

[Akuli]

If most of the time people are going to pass one of a few classes for collection_cls, you can special-case those and return Any in other cases. For example, if the most common collection_cls arguments are set, list and tuple, you can do this:

from collections import deque
from typing import TypeVar, Generic, overload, Any

_T = TypeVar("_T")

class CollectionThing(Generic[_T]):
    pass

@overload
def collection_of_objects(
    cls: type[_T], collection_cls: type[set]
) -> CollectionThing[set[_T]]: ...
@overload
def collection_of_objects(
    cls: type[_T], collection_cls: type[list]
) -> CollectionThing[list[_T]]: ...
@overload
def collection_of_objects(
    cls: type[_T], collection_cls: type[tuple]
) -> CollectionThing[tuple[_T, ...]]: ...
@overload
def collection_of_objects(
    cls: type, collection_cls: type
) -> CollectionThing[Any]: ...

def collection_of_objects(cls: type, collection_cls: type) -> CollectionThing[Any]:
    return CollectionThing()

reveal_type(collection_of_objects(str, collection_cls=set))  # CollectionThing[set[str]]
reveal_type(collection_of_objects(str, collection_cls=list))  # CollectionThing[list[str]]
reveal_type(collection_of_objects(str, collection_cls=tuple))  # CollectionThing[tuple[str, ...]]
reveal_type(collection_of_objects(str, collection_cls=deque))  # CollectionThing[Any]

This isn't ideal, because passing in collection_cls=deque didn't work. If someone wants to make a CollectionThing with a deque, they will have to just let the type be Any, or use an explicit type annotation:

thing: CollectionThing[deque[str]] = collection_of_objects(str, collection_cls=deque)
reveal_type(thing)   # Revealed type is "a.CollectionThing[collections.deque[builtins.str]]"

Also, CollectionThing[_TC[_T]] doesn't look correct to me. If you want to use a tuple, it would evaluate to something like tuple[str], which somewhat surprisingly means a tuple that contains exactly one element. An arbitrary-length tuple is tuple[str, ...].

[Akuli]

A slight improvement would be to change the last overload to this:

@overload
def collection_of_objects(
    cls: type, collection_cls: type[_T]
) -> CollectionThing[_T]: ...

Now the type of collection_of_objects(str, collection_cls=deque) will be CollectionThing[deque[Any]], which is still not perfect, but better than CollectionThing[Any].

[zzzeek]

so this implies the answer to my question, "is it OK to do runtime inspection of GenericAlias", is "no". is there consensus on this and what's the rationale ?

[zzzeek]

tuple is not one of the collection types we're targeting - has to be a mutablecollection, so assume mutablecollection, not collection.

[Akuli]

I think it's ok to inspect GenericAlias objects, just like e.g. pydantic and dacite do. But as you noticed, it easily becomes messy.

[zzzeek]

right but pydatnic always has the thing it's inpspecting on the left side of the equals sign:

foo: Annotated[... etc] = Field

and never:

foo = Field(Annotated[...])

now that's because that's what Pydantic does. sqlalchemy does something different. at the same time i think im just not that comfortable yet introducing a pattern that is nowhere in Python right now, because ive done that enough :)

[zzzeek]

pep 593 directly talks about consuming annotation data at runtime, though doesn't mention if it's in the annotated side or not. certainly what pydantic is doing here is completely a runtime thing.

certainly I should try to get the overloads for collection_class working also as that's the traditional API. but im going to poke around Annotated a bit

[zzzeek]

OK folks, since I know this discussion forum is very active, and while @Akuli has been generous in their help for which I thank them, I think the answer I'm getting for my ultimate question about using generics on the right hand side for things is:

** awkward silence **

and that is a very good answer! it means everyone's more or less on the same page as I am , which is, "well, that's just not what we had in mind, though who can say...." and as Ive spent today thinking about this I will be doing it in some more traditional ways. for relationship() we will stick with the @overload thing and judicious use of Any to make typing it simple (thanks @Akuli for the Any tip) and for the "zippy new thing" I have an approach that will read __annotations__ just like all the other libraries do which looks really nice.

[AlexWaygood]

Actually, they still haven't made from __future__ import annotations the default for 3.11 — they looked at maybe doing so, but decided to once again punt on the issue until 3.12. And PEP 649 now seems to be gaining traction over PEP 563.

[zzzeek]

so for strings representing types that aren't imported outside of TYPE_CHECKING, it keeps them as ForwardRef ?

[JelleZijlstra]

Not sure what you're referring to exactly, but there's a bug where get_type_hints() doesn't resolve strings in list["T"]. That's being fixed in the next versions of CPython.

[zzzeek]

trying out 3.11, they aren't stringed unless I do the "from future import annotations" part

[zzzeek]

OK get_type_annotations evaluates the forward refs, so isn't actually useful at runtime unless you pass it a specialized global namespace.

pydantic has their own function that's a little better, sort of

[JelleZijlstra]

My perspective is that using GenericAlias at runtime works right now, and CPython cares enough about backward compatibility for this API that if you don't do anything too crazy, it will continue to work in the foreseeable future.

However, static type checkers are probably not going to like it for a while.

[zzzeek]

thanks for the advice. mypy and pylance were fine with it but I am going to spend more time to try to keep things "normal".