Filebeat Inputs/Modules: structured logging

[urso]

Filebeat can run a many inputs and modules. Modules/Inputs can be started statically or dynamically via config reloading, auto discovery, or central config management (which reuses auto-discovery). For errors, warning or progress information it would be helpful to track down the identity of a log message to the original configuration.

For tracking identity in log message Filebeat inputs must make better use of structured logging. For this purpose a logger shall be passed on init. As inputs can fail to setup, the logger should have as much meta-data available as possible + the particular logger must be used for reporting setup errors.

meta data depend on configuration source.

All: UUID for easy correlation, input type
Input in filebeat.yml: tag? (embedded), index (e.g. input number 2)
Input from external file: tag? (external), file name, index
Input via auto discovery: tag? (discovery), provider specific meta data, other?
Input via CM: tag? (cm), beat UUID, config tag, config object ID

Modules should add the module name/set pair + module config file to the input, as modules configure inputs.

[ruflin]

+1. And for the structure of the logs, ECS should be used for the fields where applicable.

[botelastic]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[belimawr]

@cmacknz that looks like a interesting idea for the inputs/shippers, what do you think?

[cmacknz]

This would be on the input side for @kvch to consider I think.

We would probably need to consider how to propagate this information between processes in V2 as well.

[kvch]

Right. The v2 inputs already require loggers during initialization. It's up to the developer of the input to provide as much data as possible in the logger. In the documentation for input developers we could include this guideline as a tip.

[cmacknz]

Perhaps as part of the input testing/compliance suite we could verify that the logs are valid JSON, and contain some minimum set of information with standard keys/tags.

[botelastic]

Hi!
We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!