Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pyln-testing: BitcoinD TailableProc leaking file descriptors #7130

Closed
s373nZ opened this issue Mar 7, 2024 · 3 comments · Fixed by #7669
Closed

pyln-testing: BitcoinD TailableProc leaking file descriptors #7130

s373nZ opened this issue Mar 7, 2024 · 3 comments · Fixed by #7669

Comments

@s373nZ
Copy link
Contributor

s373nZ commented Mar 7, 2024

Issue and Steps to Reproduce

Running pytest locally leaks file descriptors for TailableProc stdout, read/write handles opened in the __init__() constructor. This resulted in sporadic failures and was effectively addressed by raising the ulimit. Current testing strategy is:

  1. make pytest
  2. In a 2nd terminal window get the process idea with ps aux | grep pytest
  3. In a 3rd terminal window, observe the number of opened files grow with watch -n 1 'lsof -p ${PID} | wc
  4. In a 4th terminal window, periodically run lsof -p ${PID} | less +G to see the file descriptors grow for each test.

lsof reports four file descriptors hanging around for each test which are initialized in the TailableProc constructor here.

getinfo output

N/A

@s373nZ s373nZ changed the title pytest: TailableProc leaking file descriptors pyln-testing: TailableProc leaking file descriptors Mar 7, 2024
@cdecker
Copy link
Member

cdecker commented Mar 7, 2024

So the symptom appears to be a missing teardown of some fixtures in case of an error? I think we might be able to make things more flexible by using finalizers from pytest: https://docs.pytest.org/en/6.2.x/fixture.html#adding-finalizers-directly

They are essentially callbacks that are guaranteed to be called during teardown. That could be used as a last-ditch attempt to close the FDs even though the node_factory fixture was not torn down completely due to a cleanup error.

@s373nZ
Copy link
Contributor Author

s373nZ commented Mar 7, 2024

So the symptom appears to be a missing teardown of some fixtures in case of an error?

Actually, the FDs stick around even if the test passed successfully. lsof reports 4 FDs for each test (in a deleted state), but they hang out while the entire test suite runs synchronously. When running in parallel using PYTEST_PAR, they clean themselves up.

I will review the finalizer documentation and try some experiments.

edit: If it was unclear, the subset of local test failures reported in the issue were against my initial changes which closed the files in the TailableProc::stop() function, not the current master branch.

@s373nZ
Copy link
Contributor Author

s373nZ commented Mar 8, 2024

Updated issue description for clarity by removing references to earlier prototype code cross-referenced in #7108 (comment) and describing test results.

@s373nZ s373nZ changed the title pyln-testing: TailableProc leaking file descriptors pyln-testing: BitcoinD TailableProc leaking file descriptors Sep 14, 2024
s373nZ added a commit to s373nZ/lightning that referenced this issue Sep 17, 2024
…sProject#7130])

Changelog-Fixed: pyln-testing: Fix file descriptor leak in bitcoind fixture. ([ElementsProject#7130])
s373nZ added a commit to s373nZ/lightning that referenced this issue Sep 17, 2024
…sProject#7130])

Changelog-Fixed: pyln-testing: Fix file descriptor leak in bitcoind fixture. ([ElementsProject#7130])
ShahanaFarooqui pushed a commit that referenced this issue Oct 4, 2024
Changelog-Fixed: pyln-testing: Fix file descriptor leak in bitcoind fixture. ([#7130])
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants