[BUG] "info deleteOldIndices" labeled as ERROR instead of INFO

[cpockrandt]

Describe the bug

The official OpenSearch 2.11.0 docker container logs the following messages as ERROR messages:

{
  "timestamp": "2023-11-15T22:16:49,232Z",
  "level": "ERROR",
  "component": "o.o.a.a.AlertIndices",
  "message": "info deleteOldIndices"
}

To Reproduce

I have a pretty vanilla installation of OpenSearch 2.11.0 running in the official Docker image (deployed with Helm). Logging is configured to output in JSON:

opensearch.config.log4j2.properties: |
      status = error

      appender.console.type = Console
      appender.console.name = console
      appender.console.layout.type = OpenSearchJsonLayout
      appender.console.layout.type_name = server

      rootLogger.level = "{{ default "Info" .Values.global.logLevel }}"
      rootLogger.appenderRef.console.ref = console

Expected behavior

It should be logged as INFO.

[msfroh]

@opensearch-project/admin -- Can we please move this to opensearch-project/alerting?

[StewartWBrown1]

Just to add, it looks like this was resolved in this ticket here, but the backport to 2.x failed here and was never resolved: opensearch-project/security-analytics#534

[StewartWBrown1]

Should be resolved now in opensearch-project/security-analytics#833 as this has been successfully merged and backported

[jowg-amazon]

This issue was merged and backported in these PRs
main: opensearch-project/security-analytics#203
2.x: opensearch-project/security-analytics#833
2.11: opensearch-project/security-analytics#917
2.9: opensearch-project/security-analytics#916

[cpockrandt]

@jowg-amazon Even with OpenSearch 2.13 and 2.14 I get the same behavior. Is it possible that the bug still hasn't been fully fixed?

[StewartWBrown1]

@jowg-amazon Even with OpenSearch 2.13 and 2.14 I get the same behavior. Is it possible that the bug still hasn't been fully fixed?

There were two near-identical issues in different places in OpenSearch. The one at opensearch-project/security-analytics#203 was fixed and had the 'info deleteOldIndices' message printed in logs every 12 hours from 'o.o.s.i.DetectorIndexManagementService'. This is what has been merged and is in currently released OpenSearch versions.

There was another source of the 'info deleteOldIndices' message from "o.o.a.a.AlertIndices". This is the one you reported and are still seeing. It has finally been successfully backported to 2.x as of #1230 so this error message should no longer be reported from OpenSearch 2.15!

[cpockrandt]

Thank you so much for the insight!