Tap functionality for viewing live messages

[agup006]

Opening this discussion about how to view live messages from Fluent Bit inputs. Filter's as well as a stretch

[shantanugadgil]

It would be good to be able to monitor/view/inspect incoming packet data, example for the forward input (fluentd)

Example: during setup, the tag field was missing, which was solved using the workaround script:
https://github.com/fluent/fluent-bit/blob/master/scripts/append_tag.lua

It could have helped to see that the input didn't have the tag field at all.

[patrick-stephens]

I massively agree that this would be a huge improvement to figuring things out when stuff is going wrong/missing in the configuration somewhere.

To clarify, my understanding is the following:

• A simple method to see the data at each stage in a particular pipeline, e.g. after input, after filter 1,2,3, etc. up to output plugin.
• No configuration change required to use, i.e. not having to deploy the current tee LUA filters just to see things.
• For operations investigations and not just debug (so not part of a separate image for example).

I think we need to consider the interface for this, what is the best way to access the information?

• We need something that works in K8S as well as natively
• Ideally with minimal change to current deployments
• To plumb into other tooling/automation/etc. as well

Now, a separate CLI tool may be alright natively but I'm not sure that is a great idea in general for containers/K8S.
Exposing an entirely new port for it as well means changing configurations to access it - although possible with ephemeral containers too potentially.

• Should we have it as part of the web server that currently runs? Maybe this is then wrapped in the CLI tool to give a nice output locally, plus allows other integrations with further tools.
• Should we have a kubectl plugin as well?

From a usability perspective it would be good to have this option enabled by default but we need a method to completely disable it from a security view. Limiting it to localhost or similar does not really secure it enough.

[patrick-stephens]

The requirements are broadly the main non-functionals:

• Usability
  • Must be available by default for new users, etc. but can be disabled.
  • No configuration change required.
• Security
  • Unauthorised access to data.
  • Encryption in transit of data.
• Performance
  • There will be an impact, need to ensure it is minimal and ideally non-existent if not used: "pay for what you use".

From a security and performance perspective I see some options:

• We would always want an option to disable it completely - this is the only real secure option but also to prevent performance issues.
• Configurable number of parallel connections (default to 1) for access to limit the impact on performance.
• For K8S we could ensure it is not part of the pod spec by default so people would have to access via a debug container or the host which can then be protected by the RBAC rules of K8S. It could be exposed as a pod port optionally though in the Helm chart.
• Localhost only then on native servers and protected by the access rules of the server, i.e. we do not have to implement and verify a full security stack. It has to be clear that this will expose your data so the only real secure option is to disable it.
• TLS connection only for all access.
• Setting up an authentication stack is likely out of scope, potentially we could integrate with an existing one but which/how on every platform? What do you do if it is not available? I think we could expose a hook potentially for this but not part of the standard offering: we have to indicate the security issue of this and how to prevent it of course.

If we make it a standard format for the data & commands then we can provide a standard CLI tool/image to use but people can also integrate it with other tooling.

[shantanugadgil]

I see the ease of use and extensibility with having a separate standalone tool (or maybe just a subfunction of FluentBit, maybe) with different command line options to the fluentbit binary.

Something analogous to gdb attach is what come to mind :) , where I attach and start monitoring/streaming/etc.

Some way to "attach" to a different/individual "endpoint" would be useful, i.e. "attach after filter1", then "detach" and then "attach after filter2"

[patrick-stephens]

Yeah I tend to agree, having it separate makes sense plus if we have an interface for it then supports different tooling - there could be a default implementation from this OSS project but maybe others want to integrate it into their existing tooling or do it differently.

I think also we should allow us to view configuration - maybe a simple endpoint for this though like Prometheus?
Also being able to view what Fluent Bit itself is doing and any errors/warnings there rather than just the data. However this can be done by looking at FB logs currently.