Merge pull request #31 from p2p-ld/tests-combinatorics

[tests] `numpydantic.testing` - exposing helpers for 3rd-party interface development & combinatoric testing

.github/workflows/tests.yml

@@ -33,10 +33,10 @@ jobs:
     runs-on: ${{ matrix.platform }}
 
     steps:
-      - uses: actions/checkout@v2
+      - uses: actions/checkout@v3
 
       - name: Set up python
-        uses: actions/setup-python@v4
+        uses: actions/setup-python@v5
         with:
           python-version: ${{ matrix.python-version }}
 

docs/_static/css/notebooks.css

@@ -0,0 +1,31 @@
+div.cell.tag_hide-cell details.above-input > summary,
+div.cell.tag_hide-input details.above-input > summary,
+div.cell.tag_hide-output details.below-input > summary{
+    background-color: var(--color-admonition-title-background--admonition-todo);
+    color: var(--color-content-foreground);
+    border: unset;
+    border-left: 2px solid var(--mystnb-source-margin-color);
+    opacity: unset;
+    padding: 0.25em 0 0.25em 1em;
+}
+
+div.cell.tag_hide-cell details.above-input > summary > span,
+div.cell.tag_hide-input details.above-input > summary > span,
+div.cell.tag_hide-output details.below-input > summary > span
+{
+    opacity: unset;
+}
+
+div.cell details.above-input div.cell_input {
+    border: unset;
+    background-color: unset;
+    border-left: 2px solid var(--mystnb-source-margin-color);
+}
+
+div.cell details.above-input div.cell_input div.highlight {
+    background: var(--color-admonition-background);
+}
+
+.output.text_html pre {
+    font-size: 0.8em;
+}

docs/api/testing/cases.md

@@ -0,0 +1,7 @@
+# cases
+
+```{eval-rst}
+.. automodule:: numpydantic.testing.cases
+    :members:
+    :undoc-members:
+```

docs/api/testing/helpers.md

@@ -0,0 +1,7 @@
+# helpers
+
+```{eval-rst}
+.. automodule:: numpydantic.testing.helpers
+    :members:
+    :undoc-members:
+```

docs/api/testing/index.md

@@ -0,0 +1,19 @@
+# testing
+
+Utilities for testing and 3rd-party interface development.
+
+See also the [narrative testing docs](../../contributing/testing.md)
+
+```{toctree}
+:maxdepth: 2
+
+cases
+helpers
+interfaces
+```
+
+```{eval-rst}
+.. automodule:: numpydantic.testing
+    :members:
+    :undoc-members:
+```

docs/api/testing/interfaces.md

@@ -0,0 +1,7 @@
+# interfaces
+
+```{eval-rst}
+.. automodule:: numpydantic.testing.interfaces
+    :members:
+    :undoc-members:
+```

docs/changelog.md

@@ -4,6 +4,33 @@
 
 ### 1.6.*
 
+#### 1.6.4 - 24-10-11 - Combinatoric Testing
+
+PR: https://github.com/p2p-ld/numpydantic/pull/31
+
+
+We have rewritten our testing system for more rigorous tests,
+where before we were limited to only testing dtype or shape cases one at a time,
+now we can test all possible combinations together!
+
+This allows us to have better guarantees for behavior that all interfaces
+should support, validating it against all possible dtypes and shapes.
+
+We also exposed all the helpers and array testing classes for downstream development
+so that it would be easier to test and validate any 3rd-party interfaces
+that haven't made their way into mainline numpydantic yet - 
+see the {mod}`numpydantic.testing` module.
+
+See the [testing documentation](./contributing/testing.md) for more details.
+
+**Bugfix**
+- Previously, numpy and dask arrays with a model dtype would fail json roundtripping
+  because they wouldn't be correctly cast back to the model type. Now they are.
+- Zarr would not dump the dtype of an array when it roundtripped to json,
+  causing every array to be interpreted as a random integer or float type.
+  `dtype` is now dumped and used when deserializing.
+
+
 #### 1.6.3 - 24-09-26
 
 **Bugfix**

docs/conf.py

@@ -49,6 +49,7 @@
 
 html_theme = "furo"
 html_static_path = ["_static"]
+html_css_files = ["css/notebooks.css"]
 
 # autodoc
 autodoc_pydantic_model_show_json_error_strategy = "coerce"

docs/contributing/coc.md

@@ -0,0 +1,5 @@
+# Code of Conduct
+
+```{todo}
+jonny write the code of conduct
+```

docs/contributing/index.md

@@ -0,0 +1,8 @@
+# Contributing
+
+```{toctree}
+coc
+process
+interface
+testing
+```

docs/contributing/interface.md

@@ -0,0 +1,5 @@
+# Writing an Interface
+
+```{todo}
+Jonny write the interface contrib docs
+```

docs/contributing/process.md

@@ -0,0 +1,15 @@
+# Contribution Process
+
+```{todo}
+Jonny write the contribution docs
+```
+
+### Issues
+
+### Development Environment
+
+### Testing
+
+### Linting
+
+### Pull Requests

docs/contributing/testing.md

@@ -0,0 +1,213 @@
+---
+file_format: mystnb
+mystnb:
+    output_stderr: remove
+    render_text_lexer: python
+    render_markdown_format: myst
+myst:
+    enable_extensions: ["colon_fence"]
+---
+# Testing
+
+```{code-cell}
+---
+tags: [hide-cell]
+---
+
+from pathlib import Path
+from rich.console import Console
+from rich.theme import Theme
+from rich.style import Style
+from rich.color import Color
+
+theme = Theme({
+    "repr.call": Style(color=Color.from_rgb(110,191,38), bold=True),
+    "repr.attrib_name": Style(color="slate_blue1"),
+    "repr.number": Style(color="deep_sky_blue1"),
+    "repr.none": Style(color="bright_magenta", italic=True),
+    "repr.attrib_name": Style(color="white"),
+    "repr.tag_contents": Style(color="light_steel_blue"),
+    "repr.str": Style(color="violet") 
+})
+console = Console(theme=theme)
+
+```
+
+```{note}
+Also see the [`numpydantic.testing` API docs](../api/testing/index.md)
+and the [Writing an Interface](../interfaces.md) guide
+```
+
+Numpydantic exposes a system for combinatoric testing across dtypes, shapes,
+and interfaces in the {mod}`numpydantic.testing` module.
+
+These helper classes and functions are included in the distributed package
+so they can be used for downstream development of independent interfaces
+(though we always welcome contributions!)
+
+## Validation Cases
+
+Each test case is parameterized by a {class}`.ValidationCase`. 
+
+The case is intended to be able to be partially filled in so that multiple
+validation cases can be merged together, but also used independently
+by falling back on default values.
+
+There are three major parts to a validation case:
+
+- **Annotation specification:** {attr}`~.ValidationCase.annotation_dtype` and
+  {attr}`~.ValidationCase.annotation_shape` specifies how the
+  {class}`.NDArray` {attr}`.ValidationCase.annotation` that is used to test
+  against is generated
+- **Array specification:** {attr}`~.ValidationCase.dtype` and {attr}`~.ValidationCase.shape`
+  specify that array that will be generated to test against the annotation
+- **Interface specification:** An {class}`.InterfaceCase` that refers to
+  an {class}`.Interface`, and provides array generation and other auxilary logic.
+
+Typically, one specifies a dtype along with an annotation dtype or
+a shape along with an annotation shape (or implicitly against the defaults for either),
+along with a value for `passes` that indicates if that combination is valid.
+
+```{code-cell}
+from numpydantic.testing import ValidationCase
+
+dtype_case = ValidationCase(
+    id="int_int", 
+    dtype=int, 
+    annotation_dtype=int, 
+    passes=True
+)
+shape_case = ValidationCase(
+    id="cool_shape", 
+    shape=(1,2,3), 
+    annotation_shape=(1,"*","2-4"), 
+    passes=True
+)
+
+merged = dtype_case.merge(shape_case)
+console.print(merged.model_dump(exclude={'annotation', 'model'}, exclude_unset=True))
+```
+
+When merging validation cases, the merged case only `passes` if all the
+original cases do.
+
+```{code-cell}
+from numpydantic.testing import ValidationCase
+
+dtype_case = ValidationCase(
+    id="int_int", 
+    dtype=int, 
+    annotation_dtype=int, 
+    passes=True
+)
+shape_case = ValidationCase(
+    id="uncool_shape", 
+    shape=(1,2,3), 
+    annotation_shape=(9,8,7), 
+    passes=False
+)
+
+merged = dtype_case.merge(shape_case)
+console.print(merged.model_dump(exclude={'annotation', 'model'}, exclude_unset=True))
+```
+
+We provide a convenience function {func}`.merged_product` for creating a merged product of
+multiple sets of test cases.
+
+For example, you may want to create a set of dtype and shape cases and validate
+against all combinations
+
+```{code-cell}
+from numpydantic.testing.helpers import merged_product
+
+dtype_cases = [
+    ValidationCase(dtype=int, annotation_dtype=int, passes=True),
+    ValidationCase(dtype=int, annotation_dtype=float, passes=False)
+]
+shape_cases = [
+    ValidationCase(shape=(1,2,3), annotation_shape=(1,2,3), passes=True),
+    ValidationCase(shape=(4,5,6), annotation_shape=(1,2,3), passes=False)
+]
+
+iterator = merged_product(dtype_cases, shape_cases)
+
+console.print([i.model_dump(exclude_unset=True, exclude={'model', 'annotation'}) for i in iterator])
+
+```
+
+You can pass constraints to the {func}`.merged_product` iterator to
+filter cases that match some value, for example to get only the cases that pass:
+
+```{code-cell}
+iterator = merged_product(dtype_cases, shape_cases, conditions={"passes": True})
+console.print([i.model_dump(exclude_unset=True, exclude={'model', 'annotation'}) for i in iterator])
+```
+
+## Interface Cases
+
+Validation cases can be paired with interface cases that handle
+generating arrays for the given interface from the specification in the
+validation case.
+
+Since some array interfaces like Zarr have multiple possible forms
+of an array (in memory, on disk, in a zip file, etc.) an interface
+may have multiple cases that are important to test against.
+
+The {meth}`.InterfaceCase.make_array` method does what you'd expect it to,
+creating an array, and returning the appropriate input type for the interface:
+
+```{code-cell}
+from numpydantic.testing.interfaces import NumpyCase, ZarrNestedCase
+
+NumpyCase.make_array(shape=(1,2,3), dtype=float)
+```
+
+```{code-cell}
+ZarrNestedCase.make_array(shape=(1,2,3), dtype=float, path=Path("__tmp__/zarr_dir"))
+```
+
+Interface cases also define when an interface should skip a given test
+parameterization. For example, some array formats can't support arbitrary
+object serialization, and the video class can only support 8-bit arrays
+of a specific shape
+
+```{code-cell}
+from numpydantic.testing.interfaces import VideoCase
+
+VideoCase.skip(shape=(1,1), dtype=float)
+```
+
+This, and the array generation methods are propagated up into 
+a ValidationCase that contains them
+
+```{code-cell}
+case = ValidationCase(shape=(1,2,3), dtype=float, interface=VideoCase)
+case.skip()
+```
+
+The {func}`.merged_product` iterator automatically excludes any
+combinations of interfaces and test parameterizations that should be skipped.
+
+## Making Fixtures
+
+Pytest fixtures are a useful way to re-use validation case products.
+To keep things tidy, you may want to use marks and ids when creating them
+so that you can run tests against specific interfaces or conditions
+with the `pytest -m mark` system.
+
+```python
+import pytest
+
+@pytest.fixture(
+    params=(
+        pytest.param(
+            p, 
+            id=p.id, 
+            marks=getattr(pytest.mark, p.interface.interface.name)
+        )
+        for p in iterator
+    )
+)
+def my_cases(request):
+    return request.param
+```