Initial check-in of type specification conformance suite.

.flake8

@@ -1,6 +1,7 @@
 [flake8]
 
 max-line-length = 90
+exclude = conformance
 ignore =
   # irrelevant plugins
   B3,

conformance/.gitignore

@@ -0,0 +1,47 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# Environments
+.env
+.venv
+
+# Tools
+.mypy_cache
+.pyre_configuration
+.pyre
+.coverage
+htmlcov
+
+# General
+.DS_Store
+
+# Editor temp files
+.*.swp
+
+# Workspace configurations
+.vscode

conformance/README.md

@@ -0,0 +1,69 @@
+# Type System Conformance
+
+## Motivation
+
+[PEP 729](https://peps.python.org/pep-0729/) provides a structured and documented way to specify and evolve the Python type system. In support of this effort, an official [Python typing spec](https://github.com/python/typing/tree/main/docs/spec) has been drafted. This spec consolidates details from various historical typing-related PEPs. The spec will be modified over time to clarify unspecified and under-specified parts of the type system. It will also be extended to cover new features of the type system.
+
+Accompanying the typing specification is this conformance test suite which validates the behavior of static type checkers against the specification.
+
+## Structure & Name
+
+This project contains test cases for behaviors defined in the Python typing spec. Tests are structured and grouped in accordance with the specification's chapter headings.
+
+* [concepts](https://typing.readthedocs.io/en/latest/spec/concepts.html)
+* [annotations](https://typing.readthedocs.io/en/latest/spec/annotations.html)
+* [specialtypes](https://typing.readthedocs.io/en/latest/spec/special-types.html)
+* [generics](https://typing.readthedocs.io/en/latest/spec/generics.html)
+* [qualifiers](https://typing.readthedocs.io/en/latest/spec/qualifiers.html)
+* [classes](https://typing.readthedocs.io/en/latest/spec/class-compat.html)
+* [aliases](https://typing.readthedocs.io/en/latest/spec/aliases.html)
+* [literals](https://typing.readthedocs.io/en/latest/spec/literal.html)
+* [protocols](https://typing.readthedocs.io/en/latest/spec/protocol.html)
+* [callables](https://typing.readthedocs.io/en/latest/spec/callables.html)
+* [overloads](https://typing.readthedocs.io/en/latest/spec/overload.html)
+* [dataclasses](https://typing.readthedocs.io/en/latest/spec/dataclasses.html)
+* [typeddicts](https://typing.readthedocs.io/en/latest/spec/typeddict.html)
+* [narrowing](https://typing.readthedocs.io/en/latest/spec/narrowing.html)
+* [directives](https://typing.readthedocs.io/en/latest/spec/directives.html)
+* [distribution](https://typing.readthedocs.io/en/latest/spec/distributing.html)
+* [historical](https://typing.readthedocs.io/en/latest/spec/historical.html)
+
+A test file is a ".py" file. The file name should start with one of the above names followed by a description of the test (with words separated by underscores). For example, `generics_paramspec_basic_usage.py` would contain the basic usage tests for `ParamSpec`. Each test file can contain multiple individual unit tests, but these tests should be related to each other. If the number of unit tests in a single test file exceeds ten, it may be desirable to split it into separate test files. This will help maintain a consistent level of granularity across tests.
+
+## Notes About Tests
+
+Tests are designed to run on all current and future static type checkers. They must therefore be type-checker agnostic and should not rely on functionality or behaviors that are specific to one type checker or another.
+
+Test cases are meant to be human readable. They should include comments that help explain their purpose (what is being tested, whether an error should be generated, etc.). They should also contain links to the typing spec, discussions, and issue trackers.
+
+The test suite focuses on static type checking not general Python semantics. Tests should therefore focus on static analysis behaviors, not runtime behaviors.
+
+## Running the Conformance Test Tool
+
+To run the conformance test suite:
+* Clone the https://github.com/python/typing repo.
+* Create and activate a Python 3.12 virtual environment.
+* Switch to the `conformance` subdirectory and install all dependencies (`pip install -r requirements.txt`).
+* Switch to the `src` subdirectory and run `python main.py`.
+
+## Reporting Conformance Results
+
+Different type checkers report errors in different ways (with different wording in error messages and different line numbers or character ranges for errors). This variation makes it difficult to fully automate test validation given that tests will want to check for both false positive and false negative type errors. Some level of manual inspection will therefore be needed to determine whether a type checker is fully conformant with all tests in any given test file. This "scoring" process is required only when the output of a test changes — e.g. when a new version of that type checker is released and the tests are rerun. We assume that the output of a type checker will be the same from one run to the next unless/until a new version is released that fixes or introduces a bug. In this case, the output will need to be manually inspected and the conformance results re-scored for those tests whose output has changed.
+
+Conformance results are reported and summarized for each supported type checker. Initially, results will be reported for mypy and pyright. It is the goal and desire to add additional type checkers over time.
+
+## Adding a New Test Case
+
+To add a new test, create a new ".py" file in the `tests` directory. Its name must begin with one of the above test group names followed by an underscore. Write the contents of the test including a module docstring describing the purpose of the test. Next, run the conformance test tool. This will generate a new `.toml` file in the `results` subdirectory corresponding each type checker. Manually review the output from each type checker and determine whether it conforms to the specification. If so, add `conformant = "Pass"` to the `.toml` file. If it does not fully comply, add `conformant = "Partial"` and a `notes` section detailing where it is not compliant. If the type checker doesn't support the feature in the test add `conformant = "Unsupported"`. Once the conformance status has been updated, rerun the conformance test tool to regenerate the summary report.
+
+## Updating a Test Case
+
+If a test is updated (augmented or fixed), the process is similar to when adding a new test. Run the conformance test tool to generate new results and manually examine the output of each supported type checker. Then update the conformance status accordingly. Once the conformance status has been updated, rerun the conformance test tool to regenerate the summary report.
+
+## Updating a Type Checker
+
+If a new version of a type checker is released, re-run the test tool with the new version. If the type checker output has changed for any test cases, the tool will supply the old and new outputs. Examine these to determine whether the conformance status has changed. Once the conformance status has been updated, re-run the test tool again to regenerate the summary report.
+
+## Contributing
+
+Contributions are welcome!

conformance/requirements.txt

@@ -0,0 +1,6 @@
+tomli
+tomlkit
+pyright
+mypy
+pyre-check
+pytype

conformance/results/mypy/annotations_typeexpr.toml

@@ -0,0 +1,20 @@
+conformant = "Pass"
+output = """
+annotations_typeexpr.py:77: error: Invalid type comment or annotation  [valid-type]
+annotations_typeexpr.py:77: note: Suggestion: use eval[...] instead of eval(...)
+annotations_typeexpr.py:78: error: Bracketed expression "[...]" is not valid as a type  [valid-type]
+annotations_typeexpr.py:79: error: Syntax error in type annotation  [syntax]
+annotations_typeexpr.py:79: note: Suggestion: Use Tuple[T1, ..., Tn] instead of (T1, ..., Tn)
+annotations_typeexpr.py:80: error: Invalid type comment or annotation  [valid-type]
+annotations_typeexpr.py:81: error: Invalid type comment or annotation  [valid-type]
+annotations_typeexpr.py:82: error: Invalid type comment or annotation  [valid-type]
+annotations_typeexpr.py:83: error: Invalid type comment or annotation  [valid-type]
+annotations_typeexpr.py:84: error: Invalid type comment or annotation  [valid-type]
+annotations_typeexpr.py:85: error: Variable "annotations_typeexpr.var1" is not valid as a type  [valid-type]
+annotations_typeexpr.py:85: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
+annotations_typeexpr.py:86: error: Invalid type: try using Literal[True] instead?  [valid-type]
+annotations_typeexpr.py:87: error: Invalid type: try using Literal[1] instead?  [valid-type]
+annotations_typeexpr.py:88: error: Invalid type: try using Literal[-1] instead?  [valid-type]
+annotations_typeexpr.py:89: error: Invalid type comment or annotation  [valid-type]
+annotations_typeexpr.py:90: error: Invalid type comment or annotation  [valid-type]
+"""

conformance/results/mypy/callables_annotation.toml

@@ -0,0 +1,16 @@
+conformant = "Pass"
+output = """
+callables_annotation.py:13: error: Too few arguments  [call-arg]
+callables_annotation.py:14: error: Argument 2 has incompatible type "int"; expected "str"  [arg-type]
+callables_annotation.py:15: error: Too many arguments  [call-arg]
+callables_annotation.py:16: error: Unexpected keyword argument "a"  [call-arg]
+callables_annotation.py:16: error: Unexpected keyword argument "b"  [call-arg]
+callables_annotation.py:22: error: Too many arguments  [call-arg]
+callables_annotation.py:39: error: Please use "Callable[[<parameters>], <return type>]" or "Callable"  [misc]
+callables_annotation.py:40: error: The first argument to Callable must be a list of types, parameter specification, or "..."  [valid-type]
+callables_annotation.py:40: note: See https://mypy.readthedocs.io/en/stable/kinds_of_types.html#callable-types-and-lambdas
+callables_annotation.py:41: error: Bracketed expression "[...]" is not valid as a type  [valid-type]
+callables_annotation.py:41: note: Did you mean "List[...]"?
+callables_annotation.py:42: error: Please use "Callable[[<parameters>], <return type>]" or "Callable"  [misc]
+callables_annotation.py:43: error: Unexpected "..."  [misc]
+"""

conformance/results/mypy/callables_kwargs.toml

@@ -0,0 +1,23 @@
+conformant = "Pass"
+output = """
+callables_kwargs.py:22: note: "func1" defined here
+callables_kwargs.py:43: error: Missing named argument "v1" for "func1"  [call-arg]
+callables_kwargs.py:43: error: Missing named argument "v3" for "func1"  [call-arg]
+callables_kwargs.py:48: error: Unexpected keyword argument "v4" for "func1"  [call-arg]
+callables_kwargs.py:49: error: Too many positional arguments for "func1"  [misc]
+callables_kwargs.py:55: error: Argument 1 to "func1" has incompatible type "**dict[str, str]"; expected "int"  [arg-type]
+callables_kwargs.py:58: error: Argument 1 to "func1" has incompatible type "**dict[str, object]"; expected "int"  [arg-type]
+callables_kwargs.py:58: error: Argument 1 to "func1" has incompatible type "**dict[str, object]"; expected "str"  [arg-type]
+callables_kwargs.py:60: error: "func1" gets multiple values for keyword argument "v1"  [misc]
+callables_kwargs.py:61: error: "func2" gets multiple values for keyword argument "v3"  [misc]
+callables_kwargs.py:61: error: Argument 1 to "func2" has incompatible type "int"; expected "str"  [arg-type]
+callables_kwargs.py:62: error: "func2" gets multiple values for keyword argument "v1"  [misc]
+callables_kwargs.py:98: error: Incompatible types in assignment (expression has type "Callable[[KwArg(TD2)], None]", variable has type "TDProtocol3")  [assignment]
+callables_kwargs.py:98: note: "TDProtocol3.__call__" has type "Callable[[NamedArg(int, 'v1'), NamedArg(int, 'v2'), NamedArg(str, 'v3')], None]"
+callables_kwargs.py:99: error: Incompatible types in assignment (expression has type "Callable[[KwArg(TD2)], None]", variable has type "TDProtocol4")  [assignment]
+callables_kwargs.py:99: note: "TDProtocol4.__call__" has type "Callable[[NamedArg(int, 'v1')], None]"
+callables_kwargs.py:100: error: Incompatible types in assignment (expression has type "Callable[[KwArg(TD2)], None]", variable has type "TDProtocol5")  [assignment]
+callables_kwargs.py:100: note: "TDProtocol5.__call__" has type "Callable[[Arg(int, 'v1'), Arg(str, 'v3')], None]"
+callables_kwargs.py:109: error: Overlap between argument names and ** TypedDict items: "v1"  [misc]
+callables_kwargs.py:121: error: Unpack item in ** argument must be a TypedDict  [misc]
+"""

conformance/results/mypy/callables_protocol.toml

@@ -0,0 +1,34 @@
+conformant = "Pass"
+output = """
+callables_protocol.py:35: error: Incompatible types in assignment (expression has type "Callable[[VarArg(bytes), NamedArg(int | None, 'max_items')], list[bytes]]", variable has type "Proto1")  [assignment]
+callables_protocol.py:35: note: "Proto1.__call__" has type "Callable[[VarArg(bytes), DefaultNamedArg(int | None, 'max_len')], list[bytes]]"
+callables_protocol.py:36: error: Incompatible types in assignment (expression has type "Callable[[VarArg(bytes)], list[bytes]]", variable has type "Proto1")  [assignment]
+callables_protocol.py:36: note: "Proto1.__call__" has type "Callable[[VarArg(bytes), DefaultNamedArg(int | None, 'max_len')], list[bytes]]"
+callables_protocol.py:37: error: Incompatible types in assignment (expression has type "Callable[[VarArg(bytes), NamedArg(str | None, 'max_len')], list[bytes]]", variable has type "Proto1")  [assignment]
+callables_protocol.py:37: note: "Proto1.__call__" has type "Callable[[VarArg(bytes), DefaultNamedArg(int | None, 'max_len')], list[bytes]]"
+callables_protocol.py:67: error: Incompatible types in assignment (expression has type "Callable[[VarArg(bytes)], Any]", variable has type "Proto2")  [assignment]
+callables_protocol.py:67: note: "Proto2.__call__" has type "Callable[[VarArg(bytes), KwArg(str)], None]"
+callables_protocol.py:68: error: Incompatible types in assignment (expression has type "Callable[[VarArg(str), KwArg(str)], Any]", variable has type "Proto2")  [assignment]
+callables_protocol.py:68: note: "Proto2.__call__" has type "Callable[[VarArg(bytes), KwArg(str)], None]"
+callables_protocol.py:69: error: Incompatible types in assignment (expression has type "Callable[[VarArg(bytes), KwArg(bytes)], Any]", variable has type "Proto2")  [assignment]
+callables_protocol.py:69: note: "Proto2.__call__" has type "Callable[[VarArg(bytes), KwArg(str)], None]"
+callables_protocol.py:70: error: Incompatible types in assignment (expression has type "Callable[[KwArg(str)], Any]", variable has type "Proto2")  [assignment]
+callables_protocol.py:70: note: "Proto2.__call__" has type "Callable[[VarArg(bytes), KwArg(str)], None]"
+callables_protocol.py:97: error: Incompatible types in assignment (expression has type "Callable[[int], None]", variable has type "Proto4")  [assignment]
+callables_protocol.py:97: note: "function" is missing following "Proto4" protocol member:
+callables_protocol.py:97: note:     other_attribute
+callables_protocol.py:121: error: Incompatible types in assignment (expression has type "Callable[[VarArg(bytes), DefaultNamedArg(int | None, 'max_len')], list[bytes]]", variable has type "NotProto6")  [assignment]
+callables_protocol.py:169: error: Incompatible types in assignment (expression has type "Callable[[int], Any]", variable has type "Proto8")  [assignment]
+callables_protocol.py:169: note: "Proto8.__call__" has type overloaded function
+callables_protocol.py:186: error: Incompatible types in assignment (expression has type "str", variable has type "int")  [assignment]
+callables_protocol.py:187: error: "Proto9[P, R]" has no attribute "xxx"  [attr-defined]
+callables_protocol.py:197: error: "Proto9[[int], str]" has no attribute "other_attribute2"; maybe "other_attribute"?  [attr-defined]
+callables_protocol.py:238: error: Incompatible types in assignment (expression has type "Callable[[int, str], Any]", variable has type "Proto11")  [assignment]
+callables_protocol.py:238: note: "Proto11.__call__" has type "Callable[[int, Arg(str, 'y')], Any]"
+callables_protocol.py:260: error: Incompatible types in assignment (expression has type "Callable[[VarArg(Any), NamedArg(Any, 'kwarg0')], None]", variable has type "Proto12")  [assignment]
+callables_protocol.py:260: note: "Proto12.__call__" has type "Callable[[VarArg(Any), NamedArg(Any, 'kwarg0'), NamedArg(Any, 'kwarg1')], None]"
+callables_protocol.py:284: error: Incompatible types in assignment (expression has type "Callable[[str], str]", variable has type "Proto13_Default")  [assignment]
+callables_protocol.py:284: note: "Proto13_Default.__call__" has type "Callable[[DefaultArg(str, 'path')], str]"
+callables_protocol.py:311: error: Incompatible types in assignment (expression has type "Callable[[NamedArg(str, 'path')], str]", variable has type "Proto14_Default")  [assignment]
+callables_protocol.py:311: note: "Proto14_Default.__call__" has type "Callable[[DefaultNamedArg(str, 'path')], str]"
+"""

conformance/results/mypy/dataclasses_descriptors.toml

@@ -0,0 +1,8 @@
+conformant = "Partial"
+notes = """
+Does not correctly evaluate type of descriptor access.
+"""
+output = """
+dataclasses_descriptors.py:66: error: Expression is of type "Desc2[int]", not "int"  [assert-type]
+dataclasses_descriptors.py:67: error: Expression is of type "Desc2[str]", not "str"  [assert-type]
+"""

conformance/results/mypy/dataclasses_frozen.toml

@@ -0,0 +1,7 @@
+conformant = "Pass"
+output = """
+dataclasses_frozen.py:16: error: Property "a" defined in "DC1" is read-only  [misc]
+dataclasses_frozen.py:17: error: Property "b" defined in "DC1" is read-only  [misc]
+dataclasses_frozen.py:23: error: Cannot inherit non-frozen dataclass from a frozen one  [misc]
+dataclasses_frozen.py:33: error: Cannot inherit frozen dataclass from a non-frozen one  [misc]
+"""

conformance/results/mypy/dataclasses_hash.toml

@@ -0,0 +1,6 @@
+conformant = "Partial"
+notes = """
+Does not report when dataclass is not compatible with Hashable protocol.
+"""
+output = """
+"""

conformance/results/mypy/dataclasses_inheritance.toml

@@ -0,0 +1,5 @@
+conformant = "Pass"
+output = """
+dataclasses_inheritance.py:60: error: Cannot override instance variable (previously declared on base class "DC6") with class variable  [misc]
+dataclasses_inheritance.py:64: error: Cannot override class variable (previously declared on base class "DC6") with instance variable  [misc]
+"""

conformance/results/mypy/dataclasses_kwonly.toml

@@ -0,0 +1,18 @@
+conformant = "Partial"
+notes = """
+Incorrectly rejects kw_only field with default before positional field.
+"""
+output = """
+dataclasses_kwonly.py:23: error: Too many positional arguments for "DC1"  [misc]
+dataclasses_kwonly.py:29: error: Attributes without a default cannot follow attributes with one  [misc]
+dataclasses_kwonly.py:32: error: Too many positional arguments for "DC2"  [misc]
+dataclasses_kwonly.py:32: error: Too few arguments for "DC2"  [call-arg]
+dataclasses_kwonly.py:32: error: Argument 1 to "DC2" has incompatible type "str"; expected "int"  [arg-type]
+dataclasses_kwonly.py:35: error: "DC2" gets multiple values for keyword argument "b"  [misc]
+dataclasses_kwonly.py:35: error: Too few arguments for "DC2"  [call-arg]
+dataclasses_kwonly.py:35: error: Argument 1 to "DC2" has incompatible type "str"; expected "int"  [arg-type]
+dataclasses_kwonly.py:38: error: Too many positional arguments for "DC2"  [misc]
+dataclasses_kwonly.py:38: error: Argument 1 to "DC2" has incompatible type "str"; expected "int"  [arg-type]
+dataclasses_kwonly.py:38: error: Argument 2 to "DC2" has incompatible type "int"; expected "str"  [arg-type]
+dataclasses_kwonly.py:53: error: Too many positional arguments for "DC3"  [misc]
+"""

conformance/results/mypy/dataclasses_order.toml

@@ -0,0 +1,4 @@
+conformant = "Pass"
+output = """
+dataclasses_order.py:50: error: Unsupported operand types for < ("DC1" and "DC2")  [operator]
+"""

conformance/results/mypy/dataclasses_postinit.toml

@@ -0,0 +1,11 @@
+conformant = "Pass"
+output = """
+dataclasses_postinit.py:19: error: Argument 3 of "__post_init__" is incompatible with supertype "dataclass"; supertype defines the argument type as "str"  [override]
+dataclasses_postinit.py:28: error: "DC1" has no attribute "x"  [attr-defined]
+dataclasses_postinit.py:29: error: "DC1" has no attribute "y"  [attr-defined]
+dataclasses_postinit.py:36: error: Signature of "__post_init__" incompatible with supertype "dataclass"  [override]
+dataclasses_postinit.py:36: note:      Superclass:
+dataclasses_postinit.py:36: note:          def __post_init__(self: DC2, x: int, y: str) -> None
+dataclasses_postinit.py:36: note:      Subclass:
+dataclasses_postinit.py:36: note:          def __post_init__(self: DC2, x: int) -> None
+"""