[BUG] Gradle Check Log missing after a while

[peterzhuamazon]

[BUG] Gradle Check Log missing after a while

Hi,

We recently see the gradle check logs missing over time in Jenkins console:
https://build.ci.opensearch.org/blue/organizations/jenkins/gradle-check/detail/gradle-check/43293/pipeline

java.io.FileNotFoundException: /var/jenkins_home/jobs/gradle-check/builds/43293/log (No such file or directory)

	at java.base/java.io.RandomAccessFile.open0(Native Method)

	at java.base/java.io.RandomAccessFile.open(RandomAccessFile.java:356)

These failures observed on both true failures and success runs.

Thanks.

---

PRs:

• Disable 2.16.0 cron on distribution build workflow opensearch-build#4881
• Make sure Ubuntu is using JDK21 when connecting to Main node #481
• Reduce from 180 to 90 days for gradle check log rotation opensearch-build#4878
• Update Ubuntu AMI to the latest builds #482
• Switch to ELASTIC mode for the EFS #483
• Disable auto log file removal for Jenkins gradle check logs #485

[peterzhuamazon]

Adding latest update from the docs: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Agent-Configuration-File-Details.html

auto_removal – Optional. If this is true, the CloudWatch agent automatically deletes this log file after reading it and it has been rotated. Usually the log files are deleted after their entire contents are uploaded to CloudWatch Logs, but if the agent reaches the EOF (end of file) and also detects another newer log file that matches the same file_path, the agent deletes the OLD file, so you must make sure that you are done writing to the OLD file before creating the NEW file. The [RUST tracing library](https://docs.rs/tracing/latest/tracing/)

has a known incompatibility because it will potentially create a NEW log file and then still attempt to write to the OLD log file.

The agent only removes complete files from logs that create multiple files, such as logs that create separate files for each date. If a log continuously writes to a single file, it is not removed.

If you already have a log file rotation or removal method in place, we recommend that you omit this field or set it to false.

If you omit this field, the default value of false is used.

[peterzhuamazon]

Especially this symptoms matching the case, where one run is waiting for the agent, so that file is EOL and will not getting write until Agent inbound.
As soon as the agent inbound and try to write new content, it was treated as a new file, then agent delete the file after all.

[peterzhuamazon]

Seems good. Thanks.
https://build.ci.opensearch.org/job/gradle-check/43385/console