Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Improve jenkins performance, this should address the jenkins 504 timeout errors due to
OutOfMemoryError
Issues Resolved
Part of #346
Analysis-1
Started to Debug with created a 360 java heap dump using an open source yc tool.
Extracted the yc-2023-09-29T04-32-09.zip and used jhat cli debug the heap dump, noted error as: Exception in thread "main" java.lang.OutOfMemoryError: Java heap space.
Also analyzed all the dump files part of the zip and found out multiple threads in
BLOCKED
state each honoring the setting-Xss4m
.Apart form having the threads in
BLOCKED
state. noted multipleI/O
errors with threads interrupted state.Took some measures to fix this.
-Xss
setting to avoid large overloaded thread stack size.Analysis-2
Coming from jenkins GC performance tuning, added the following to the jenkins Jvm.
-XX:+UseG1GC
: Changes to Garbage First Garbage Collector strategy, G1GC is designed to be a low-latency garbage collector. It divides the heap into regions and performs garbage collection on specific regions.-XX:+ExplicitGCInvokesConcurrent
: This flag allows explicit garbage collection calls (System.gc()) to run concurrently with the application threads.-XX:+ParallelRefProcEnabled
: This flag enables parallel reference processing which enables us to use the multiple cores more effectively.-XX:+UseStringDeduplication
: This option enables string deduplication in G1GC. It means that duplicate strings in the heap will be deduplicated, helping to save memory by sharing the same underlying string data.-XX:+UnlockExperimentalVMOptions
: This flag allows the use of experimental JVM options.-XX:+UnlockDiagnosticVMOptions
: This flag unlocks diagnostic VM options, allowing the use of additional diagnostic options for monitoring and troubleshooting the JVM.-XX:G1SummarizeRSetStatsPeriod=1
: This option sets the frequency at which G1GC summarizes.Along with above Jvm options, Manually identified the running threads using the following script and noticed there are some running threads in errored state.
Now Executed
interrupt()
method on each thread and manually killed it. Then after restarting the jenkins the memory usage is stable and did not notice any 504 errors.NOTE: Used
htop
linux tool to monitor the memory in real time.By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.