Resolving "data could not be added to batch" error

[nishaprabhakar]

Hello @brexhq/substation!

We are running into the following error when parsing a cloudtrail log source.

Context

Input Log Format

Cloudtrail log format: {"Records": [... list of JSON logs]}

Transformation

local twenty_mb = 20000000;

{
  concurrency: 2,
  transforms: [
      helpers.general.expand("Records"),
      sub.tf.obj.insert({obj: {trg: 's3'}, value: true}),
      sub.tf.send.aws.s3(settings={
        bucket_name: "[bucket]",
        file_path: {
          prefix: "s3-data-events-analysis/",
          uuid: true,
          time_format: "2006/01/02/15/04"
        },
        auxiliary_transforms: sub.pattern.tf.fmt.jsonl,
        batch: { size: twenty_mb }
      }),
    ]
}

helpers.libsonnet

{
    general: {
        to_json(key): sub.transform.object.copy(
            settings={object: {target_key: key}}
        ),
        expand(key): sub.tf.agg.from.arr({ object: { source_key: key} }),
    },
}

Error String

transform: send_aws_s3: transform: send_aws_s3: transform: send_stdout: data could not be added to batch

I am trying to understand where this error might be coming from. I have an identical configuration for other cloudtrail log sources which seem to not have issues.

I investigated where the "stdout" part of the error comes from, since send_stdout is not a part of the transform. It seems like the aggregate_to_array transform prints this error, but only the aggregate_from_array transform is used in helpers.

Are there any ideas on how to resolve this? Thanks so much!

[jshlbrd]

@nishaprabhakar Some additional questions that might help us debug:

• Which version of Substation are you using?
• Can you confirm that the compiled JSON configuration matches the Jsonnet configuration?

I checked the main branch and don't see the string send_stdout anywhere, except for the send to stdout transform. That makes me think it could be that your deployment doesn't have the compiled JSON referenced in the Jsonnet config.

[nishaprabhakar]

• We're using v1.1.0 (in our deploy.sh file, we have export SUBSTATION_VERSION=v1.1.0).

The aggregate_to_string transform (transforms/aggregate_to_string.go) has the following code on line 82 (of our version of substation):

if ok := tf.agg.Add(key, msg.Data()); !ok {
    return nil, fmt.Errorf("transform: send_stdout: %v", errSendBatchMisconfigured)
}

[jshlbrd]

@nishaprabhakar I think I see the problem, but you may want to delete that comment (in case you're concerned about the AWS resource listed).

The format to JSONL pattern has the default batch settings coded into it (seen here). That's limiting the total size of the batch to 1 MB, and since you increased the size on the S3 send to 20 MB, you probably have events larger than that; the S3 settings is accepting data larger than 1 MB, but the JSONL conversion isn't.

In this case the best thing to do is rewrite that pattern into your helpers function with a much larger size value (same as the send_s3 config). Alternatively, you could overwrite the default value in your local substation.libsonnet file, but that will be more difficult to maintain when you upgrade.

The reference to send_stdout is definitely a bug, looks like it's been fixed since v1.1.0. We're currently on v1.6.0, so my suggestion is to stay within the last couple releases because we've been regularly adding features and making fixes.

[nishaprabhakar]

Thanks so much for your help! (Also just redacted my comment)