[AWS INPUT]Investigate handling of unexcpected EOF error in aws-s3 input

[gizas]

Current Status

Customer has reported that when they use the aws-s3 polling functionality of aws s3 input witness the unexpected EOF error:

  "message": "failed processing S3 event for object key \"dso-swissre-com/http_request/20241102/20241102T002812Z_20241102T002920Z_b1739eb2.log.gz\" in bucket \"laas-http-logs\": failed reading s3 object (elapsed_time_ns=875637796721): failed to decode json: unexpected EOF",

Due to no special handling of this error, we mark the state of the registry as state.Failed = true, and this means that this s3 object wont be retried to processed again. FYI the only errors that we handle and retry are the [S3 download errors]

beats/x-pack/filebeat/input/awss3/s3_input.go

Line 165 in a3cefc0

if errors.Is(err, errS3DownloadFailed) {

).

Goal

To investigate whether we need to perform special handling and retry the process for error unexpected EOF.

Due to ContentType='application/json' in the gzip files of S3, the input uses the readJson function. This is the function that needs possible update and in mode details the decoder

[elasticmachine]

Pinging @elastic/obs-ds-hosted-services (Team:obs-ds-hosted-services)

[gizas]

@fairouz-es I have run some tests with your provided file.

My Elastic-Agent has no other traffic, so in case your file is uploaded, then it is processed successfully.

This means that the tunning exercise we do on the sdh is needed, as the unexpected EOF does not come from an actual decoding of json error.
Keep investigating

[gizas]

At the moment I suspect the https://cs.opensource.google/go/go/+/refs/tags/go1.23.5:src/encoding/json/stream.go;l=132
This is probably the reason that our Decode function returns an error. So seems nonSpace in dec.buf