Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Propagate known pytest failure/skip reasons to pytest summary #865

Merged
merged 2 commits into from
Nov 6, 2024

Conversation

jhkennedy
Copy link
Collaborator

@jhkennedy jhkennedy commented Nov 4, 2024

In #858, there was some confusion on what an xfailed test meant and why the unit test suit was still considered passing. This PR adds the -rxXs flag to PyTest calls (via nox) so that skips and excepted failure reasons (if they were provided) are reported in the test summary:

pytest tests/unit -rxXs
===================================================== test session starts ======================================================
platform linux -- Python 3.13.0, pytest-8.3.3, pluggy-1.5.0
rootdir: /home/jhkennedy/Documents/Code/contributing/earthaccess
configfile: pyproject.toml
plugins: anyio-4.6.2.post1, cov-6.0.0
collected 46 items                                                                                                             

tests/unit/test_api.py ..                                                                                                [  4%]
tests/unit/test_auth.py ...                                                                                              [ 10%]
tests/unit/test_collection_queries.py ...........                                                                        [ 34%]
tests/unit/test_deprecations.py ..                                                                                       [ 39%]
tests/unit/test_formatters.py ..                                                                                         [ 43%]
tests/unit/test_granule_queries.py ............                                                                          [ 69%]
tests/unit/test_results.py ........                                                                                      [ 86%]
tests/unit/test_services.py ..                                                                                           [ 91%]
tests/unit/test_store.py ..x                                                                                             [ 97%]
tests/unit/test_uat.py .                                                                                                 [100%]

=================================================== short test summary info ====================================================
XFAIL tests/unit/test_store.py::test_earthaccess_file_getattr - Expected failure: Reproduces a bug (#610) that has not yet been fixed.
================================================ 45 passed, 1 xfailed in 10.81s ================================================

I've also updated the error message to clarify what's happening in the pytest summary.

I don't think there's much value to a changelog entry here, so I've not added one; happy to add one is anyone things otherwise.

Pull Request (PR) draft checklist - click to expand
  • Please review our
    contributing documentation
    before getting started.
  • Populate a descriptive title. For example, instead of "Updated README.md", use a
    title such as "Add testing details to the contributor section of the README".
    Example PRs: #763
  • Populate the body of the pull request with:
  • Update CHANGELOG.md with details about your change in a section titled
    ## Unreleased.
    If such a section does not exist, please create one. Follow
    Common Changelog for your additions.
    Example PRs: #763
  • Update the documentation and/or the README.md with details of changes to the
    earthaccess interface, if any.
    Consider new environment variables, function names,
    decorators, etc.

Click the "Ready for review" button at the bottom of the "Conversation" tab in GitHub
once these requirements are fulfilled. Don't worry if you see any test failures in
GitHub at this point!

Pull Request (PR) merge checklist - click to expand

Please do your best to complete these requirements! If you need help with any of these
requirements, you can ping the @nsidc/earthaccess-support team in a comment and we
will help you out!

  • Add unit tests for any new features.
  • Apply formatting and linting autofixes. You can add a GitHub comment in this Pull
    Request containing "pre-commit.ci autofix" to automate this.
  • Ensure all automated PR checks (seen at the bottom of the "conversation" tab) pass.
  • Get at least one approving review.

📚 Documentation preview 📚: https://earthaccess--865.org.readthedocs.build/en/865/

Copy link

github-actions bot commented Nov 4, 2024

Binder 👈 Launch a binder notebook on this branch for commit 07e2d85

I will automatically update this comment whenever this PR is modified

Binder 👈 Launch a binder notebook on this branch for commit 1ea22fa

Binder 👈 Launch a binder notebook on this branch for commit 75274a9

Binder 👈 Launch a binder notebook on this branch for commit 97ea6b5

@codecov-commenter
Copy link

codecov-commenter commented Nov 4, 2024

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 58.72%. Comparing base (2f49c08) to head (97ea6b5).
Report is 76 commits behind head on main.

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

❗ There is a different number of reports uploaded between BASE (2f49c08) and HEAD (97ea6b5). Click for more details.

HEAD has 6 uploads less than BASE
Flag BASE (2f49c08) HEAD (97ea6b5)
14 8
Additional details and impacted files
@@             Coverage Diff             @@
##             main     #865       +/-   ##
===========================================
- Coverage   73.88%   58.72%   -15.17%     
===========================================
  Files          31       13       -18     
  Lines        2003     1100      -903     
===========================================
- Hits         1480      646      -834     
+ Misses        523      454       -69     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

mfisher87
mfisher87 previously approved these changes Nov 4, 2024
Copy link
Collaborator

@mfisher87 mfisher87 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nitpick only :)

noxfile.py Outdated Show resolved Hide resolved
@@ -130,7 +130,7 @@ def test_store_can_create_s3_fsspec_session(self):


@pytest.mark.xfail(
reason="This test reproduces a bug (#610) which has not yet been fixed."
reason="Expected failure: Reproduces a bug (#610) that has not yet been fixed."
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure how I feel about including the status in the reason string, but don't feel strongly enough to request anything. If it's helping someone, it's helping someone!

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did this because xfail, xfailed and XFAIL require you to know Pytest well enough to interpret them as "expected failure". As soon as you know that, the x = expected makes sense, but if you don't there's nothing really to help you get there.

I think it's a little more general contributor-friendly this way, but I'm fine leaving this as a "Google it" thing if you don't understand.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a good enough reason for me ;)

mfisher87
mfisher87 previously approved these changes Nov 5, 2024
chuckwondo
chuckwondo previously approved these changes Nov 6, 2024
Co-authored-by: Matt Fisher <3608264+mfisher87@users.noreply.github.com>
@jhkennedy jhkennedy merged commit 50bf0b9 into nsidc:main Nov 6, 2024
13 checks passed
@jhkennedy jhkennedy deleted the pytest-reason branch November 6, 2024 01:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants