django.test.TestCase not recorded when the tests are run by pytest

[dustinbyrne]

paperless-ngx uses django.test.TestCase for its tests. django.test.TestCase is a subclass of unittest.TestCase that adds functionality for testing Django applications.

When running these test cases with appmap-python, no AppMaps are emitted. It seems as though the subclasses of django.test.TestCase are not treated equally to those of unittest.TestCase.

[kgilpin]

@dustinbyrne can you provide more information to help Navie spec this issue? Then apply the label navie-plan.

[github-actions]

Title: Ensure django.test.TestCase Subclasses are Recorded by appmap-python When Run by pytest

Problem:
django.test.TestCase subclasses are not producing AppMaps when their tests are executed using pytest. Although django.test.TestCase is a subclass of unittest.TestCase, it adds specific functionalities for testing Django applications. The current AppMap configuration doesn't recognize the django.test.TestCase subclasses correctly, leading to no AppMaps being generated for these test cases.

Analysis:
The root cause of this issue lies in the way the appmap-python package detects test cases for recording. The existing logic correctly identifies unittest.TestCase subclasses but does not account for django.test.TestCase subclasses. Since django.test.TestCase inherits from unittest.TestCase, it should ideally be detected by the AppMap recorder just as unittest.TestCase is. However, there's either a type-check or an exclusion that omits these Django-specific test cases. By extending the logic to also detect and handle django.test.TestCase, we should be able to resolve this issue and generate AppMaps for these tests when they are run using pytest.

Proposed Changes:

1. Update pytest_runtest_call to include django.test.TestCase subclasses:
   • In the pytest.py file, modify the logic that handles test case checking. Specifically, update the logic to recognize django.test.TestCase in addition to unittest.TestCase.

@pytest.hookimpl
def pytest_runtest_call(item):
    # The presence of a `_testcase` attribute on an item indicates
    # that it was created from a `unittest.TestCase` or `django.test.TestCase`
    if hasattr(item, "_testcase") or (hasattr(item.obj, "__class__") and 
                                       issubclass(item.obj.__class__, django.test.TestCase)):
        setattr(
            item._testcase,  
            "_appmap_pytest_recording",
            True,
        )
        if not noappmap.disables(item.obj, item.cls):
            testing_framework.disable_test_case(item.obj)
            item.obj = recorded_testcase(item)(item.obj)

2. Test the Configuration:
   • Modify the existing tests to include test cases for django.test.TestCase. Extend one of the test classes in test_test_frameworks.py to ensure it uses django.test.TestCase and validate if the AppMaps are generated.

class TestDjangoTestCaseRunner(_TestTestRunner):
    @classmethod
    def setup_class(cls):
        cls._test_type = "pytest"

    def run_tests(self, testdir):
        testdir.test_type = "pytest"
        result = testdir.runpytest("-svv")
        result.assert_outcomes(passed=1)

    def test_enabled(self, testdir):
        self.run_tests(testdir)
        assert len(list(testdir.output().iterdir())) == 1

        verify_expected_appmap(testdir)
        verify_expected_metadata(testdir)

3. Update the pytest.ini Config for Django:
   • Ensure the correct Django settings module is specified in the pytest.ini.

[pytest]
DJANGO_SETTINGS_MODULE = djangoapp.settings

By ensuring that both unittest.TestCase and django.test.TestCase subclasses are detected by appmap-python, this configuration will allow for AppMaps to be generated for Django test cases executed with pytest.

I'm confident that these changes will address the recording issue for django.test.TestCase subclasses. Once you've made these modifications, try running your test suite again to confirm the solution.

[apotterri]

paperless uses pytest to run Django tests. This isn't a usual configuration for a django project. The pytest plugin provided by appmap doesn't currently support this configuration, so no test recordings get created.

[evlawler]

@apotterri correction from Kevin: Please edit Dustin's issue above and it will kick off again and you can see if it is more correctly scoped. This would be a great issue to discuss tomorrow at standup if you could scope the issue and rerun the plan. The issue was not correctly specified by Dustin, since it was listed as a "bug" when it is really an enhancement to support this configuration. So let's improve the issue and see what comes afterward.

[dustinbyrne]

This states that Django uses unittest, so shouldn't that work?
https://docs.djangoproject.com/en/5.0/topics/testing/overview/

Is the issue that the import is django.test.TestCase and not unittest.TestCase?