Elasticsearch & Opensearch output plugins have unexpected/undocumented behavior

[brian-finisher]

Describe the bug

It took me hours to debug this issue, and it wasn't until digging into the plugin's README files that I figured out what is going on!

Both the Opensearch & Elasticsearch plugins mention the logstash_format parameter impacting the index. What it does not say is that if you set logstash_format = false, it also sets include_timestamp=false. This results in records not being usable in Opensearch Dashboards & Kibana, as there are no time records.

Link to the problematic documentation

https://docs.fluentd.org/output/opensearch#logstash_format-optional
https://docs.fluentd.org/output/elasticsearch#logstash_format-optional

Expected explanation

If true, Fluentd uses the conventional index name format logstash-%Y.%m.%d (default: false). This option supersedes the index_name option.
In addition, include_timestamp (default: false) is set to true, which sends timestamp information which can be used by (Opensearch Dashboards / Kibana).

Additional context

The documentation here: https://github.com/fluent/fluent-plugin-opensearch#include_timestamp does indicate this behavior.

Note that the documentation here is much more thorough and authoritative overall.