Difference when comparing CloudWatch and OpenSearch query results

[alexzon-tr]

I'm ingesting CloudTrail Logs into OpenSearch using a centralized S3 bucket.
When trying to create an alarm for when I detect the eventName:CreateUser event, I noticed that I couldn't find these events inside OpenSearch.
I've searched for the same events directly from CloudTrail, on CloudWatch Logs, and even inside the S3 Bucket. I was able to find the events in the 3 locations, as expected.

I did the same test but ingesting from CloudWatch Logs to another OpenSearch index. Same results.

Going a little deeper, I've made a query filtering for my user and eventSource:iam.amazonaws.com, exported the results to a spreadsheet, and compared them.
CloudWatch Logs returns much more results (80) than OpenSearch (10), including the CreateUser event I was interested in the first place.
Comparing the eventID field, I could find the 10 events in the bigger list, among others 70 that are not being ingested.

I've looked for Lambda or SQS errors but couldn't find anything.

Any idea why this behavior?

[JoeShi]

looking into it.

[qianyangcassie]

Hi there, could you please expand your time range and filter one specific event id in OpenSearch that you mentioned about others 70 that are not being ingested? Just want to make sure that the logs have not been ingested ever before, because we noticed that when CloudTrail writing the same log into both CloudWatch and S3 bucket, the timestamps are different.

[alexzon-tr]

The time range was pretty broad, 24 hours. There were just a few results because I filtered for my personal user and IAM events.
But sure, I did as you asked: I set the time range to July 25 (entire day) and searched for specific Event IDs.
Queries:

• { $.eventID = "ID HERE" } (CloudWatch Logs)
• eventID:"ID HERE" (OpenSearch)

Same results. I can find some of the events on both CloudWatch and OpenSearch. The majority I can find only on CloudWatch.

[qianyangcassie]

I reproduced the problem according to your steps. We have noticed that when writing CloudTrail logs, OpenSearch will run into the error: Limit of total fields [1000] has been exceeded. Could you please try out the following workaround? We will fix that in later version.

    In OpenSearch Dev Tools console, run command: 
     PUT  <Your Index>/_settings
    {
      "index.mapping.total_fields.limit": 2000
    }

[alexzon-tr]

It worked! Thanks a lot.
I will check how to enable OpenSearch logs to monitor if there are any other errors.

[qianyangcassie]

If there were any errors, we will log that in LogHubLoggingBucket. You may check that. You can find the bucket link in CloudFormation resources.
Besides, in our upcoming new version, we support enabling monitoring on solution console , and you can receive emails when there are errors.