Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Thread leaks occur during periods of high build count #368

Open
atavakoliyext opened this issue Jan 16, 2025 · 0 comments
Open

Thread leaks occur during periods of high build count #368

atavakoliyext opened this issue Jan 16, 2025 · 0 comments

Comments

@atavakoliyext
Copy link

atavakoliyext commented Jan 16, 2025

Plugin version: 2.0.728

Steps to reproduce the problem:

  1. Run large batches of new builds (no exact size to trigger the leak has been found, but it's unmissable when build queue size reaches 1000)
  2. Wait for builds to complete
  3. Inspect the jvm_threads metric scraped from the TeamCity server's Prometheus endpoint
  4. Generate & inspect a thread dump of the server in the Administration / Diagnostics page

Expected result

  • Thread count stays steady over time.

Actual result

  • Thread count increases sharply while the builds are run, and thread dumps in TeamCity server show thousands of instances of the below stack trace:
"BatchSpanProcessor_WorkerThread-1" daemon group="main" prio=5 tid=935800 nid=935800 waiting 
   java.lang.Thread.State: TIMED_WAITING
    at java.base@17.0.7/jdk.internal.misc.Unsafe.park(Native Method)
    at java.base@17.0.7/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:252)
    at java.base@17.0.7/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1672)
    at java.base@17.0.7/java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:435)
    at io.opentelemetry.sdk.trace.export.BatchSpanProcessor$Worker.run(BatchSpanProcessor.java:266)
    at java.base@17.0.7/java.lang.Thread.run(Thread.java:833)

Examples of the pattern of increased jvm_threads counts is shown below.

Image

Additional information

This was originally reported to JetBrains -- see YouTrack ticket TW-91607 -- who ruled out any usage of OTel SDKs in their code.

Disabling the plugin temporarily while running the reproduction steps also kept jvm_threads from increasing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant