Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: add a per test 3min timeout and enable color output #700

Merged
merged 1 commit into from
Feb 6, 2024

Conversation

bringhurst
Copy link
Contributor

@bringhurst bringhurst commented Jan 29, 2023

Why is this needed?

It's difficult to reproduce, but I've seen the github actions test suite take more than 6 hours. Having a timeout set will allow us to get some sort of useful debug information instead of having github actions kill it off before we can analyze anything.

Proposed Changes

  • Do not allow any one unit test to take more than an hour. If it takes longer than an hour, kill it off and print some basic debug info (e.g. a stack trace of all/most frames).
  • Enable color pytest output to make pytest easier to read in general.

Does this PR introduce any breaking change?

No -- this is intended to help debug the test suite.

@bringhurst bringhurst force-pushed the pytest_timeout branch 2 times, most recently from c66bda1 to 0b7f73a Compare January 29, 2023 21:02
@codecov-commenter
Copy link

codecov-commenter commented Jan 29, 2023

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (55f27b2) 96.76% compared to head (5bdd8ee) 96.79%.
Report is 1 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master     #700      +/-   ##
==========================================
+ Coverage   96.76%   96.79%   +0.02%     
==========================================
  Files          27       27              
  Lines        3557     3557              
==========================================
+ Hits         3442     3443       +1     
+ Misses        115      114       -1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@bringhurst
Copy link
Contributor Author

bringhurst commented Jan 29, 2023

I keep seeing this error also (it appears to be a string returned from the codecov api):

Could not determine repo and owner

Setting codecov-actions to verbose doesn't seem to give any more details.

🤔

@ceache
Copy link
Contributor

ceache commented Jan 29, 2023 via email

@a-ungurianu
Copy link
Member

This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

I wonder if this is why you see the codecov things, as it tries leverage the Github App.

@ceache I don't have the rights to add the app. Do you know someone who does?

@bringhurst bringhurst changed the title chore: add a one hour timeout to all tests chore: add a one hour timeout to all tests (debug flaky tests) Jan 29, 2023
@bringhurst bringhurst force-pushed the pytest_timeout branch 2 times, most recently from 66850d3 to 55a242d Compare January 29, 2023 22:12
@bringhurst
Copy link
Contributor Author

bringhurst commented Jan 29, 2023

An example of this showing useful debug output:

https://github.com/python-zk/kazoo/actions/runs/4038897905/jobs/6943223749

It lets the pytest timeout (not github actions timeout) determine that the test failed and prints out a trace of all frames.

@bringhurst bringhurst force-pushed the pytest_timeout branch 3 times, most recently from 33e4705 to 51fa4d5 Compare January 29, 2023 22:53
@bringhurst
Copy link
Contributor Author

bringhurst commented Jan 29, 2023

It looks like this job may time out -- https://github.com/python-zk/kazoo/actions/runs/4039044495/jobs/6943463553

If that works correctly (times out and shows stack traces), I'll open this up for review.

Edit: It looks like this shows a nice stacktrace.

@bringhurst bringhurst changed the title chore: add a one hour timeout to all tests (debug flaky tests) chore: add a one hour timeout to all tests and enable color output Jan 29, 2023
@bringhurst bringhurst marked this pull request as ready for review January 30, 2023 00:12
jeffwidman
jeffwidman previously approved these changes Jan 30, 2023
Copy link
Member

@jeffwidman jeffwidman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Happy to try it, and keep it as long as it's useful!

I do wonder whether this is just actions infra being flaky from time to time, or if we've got some bugs in our code that are hiding... could just be Zookeeper occasionally flaking as it comes up, I know I've seen that from time to time although no idea if that's Zookeeper, or Docker or something else...

tox.ini Outdated Show resolved Hide resolved
@StephenSorriaux StephenSorriaux changed the title chore: add a one hour timeout to all tests and enable color output chore: add a per test 3min timeout and enable color output Feb 5, 2024
@StephenSorriaux
Copy link
Member

I have resurrected this PR and reduced the timeout from 1h to 3min since it is a per-test timeout.

@jeffwidman @a-ungurianu @ceache if you have some time to check it

Copy link
Member

@jeffwidman jeffwidman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great 👍

@StephenSorriaux StephenSorriaux merged commit 242d91e into python-zk:master Feb 6, 2024
30 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants