[8.x](backport #41694) [aws] feat: aws-s3 input registry cleanup for untracked s3 objects

[mergify]

Proposed commit message

This PR partially addresses #39116 by introducing a registry cleanup strategy for aws-s3 input.

The cleanup implemented here removes registry entries if the s3 object is no longer available (aka tracked) when listing inside the polling lookup. The cleanup removes objects that are not tracked from both the local state and internal store (backed by the registry) to reduce the memory usage.

Note that, this only benefits when s3 objects get removed (ex:- using lifecycle policy) and are no longer available. There should be a follow-up for instances where such removal is not done at the bucket. For example, this could be done by,

• Defining a time-based sliding window (ex:- only consider s3 objects of the past 3 days)
• Only accept objects of a known storage class (ex:- S3 Standard ) so that users can archive objects ¹

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

• Build or run filebeat with aws-s3 input
• Add s3 objects and observe memory growth with pprof (regex filter for awss3)
• Remove s3 objects and observe memory reduction

Screenshots

Given below are pprof analyss comparisons for ~4K objects in registry and once they were clean up by removing S3 objects (emptying the bucket)

• With ~4K objects processed

• When S3 objects were removed

Related issues

• Relates #39116
• Depends on [AWS] improve S3 input states copy by only storing filtered entries #41869

Footnotes

1. https://docs.aws.amazon.com/AmazonS3/latest/userguide/lifecycle-transition-general-considerations.html

   ---

   This is an automatic backport of pull request [aws] feat: aws-s3 input registry cleanup for untracked s3 objects #41694 done by Mergify. ↩

[elasticmachine]

Pinging @elastic/obs-ds-hosted-services (Team:obs-ds-hosted-services)