My Filebeat Not Collecting Logs from Monibuca Streaming Service Container

[clo-github]

hi all,
I've recently been using Filebeat to collect logs and send them to Elasticsearch (without Logstash). I have a VM running five containers:

Nginx
Filebeat
Node-exporter
Cadvisor
Monibuca (a streaming service container)
My Filebeat configuration looks like this:

filebeat.config:
  modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: false
filebeat.autodiscover:
  providers:
    - type: docker
      hints.enabled: true
filebeat.inputs:
  - type: log
    enabled: true
    paths:
      - "/var/lib/docker/containers/34446dd22ca2a115495ade05cad4932fd6536c18704d5757882efb690d1367b3/*-json.log"
      - "/var/lib/docker/containers/3fb19e168a6e0de17439a81dd0a8227beda006a60b9adaf1c13673be4c81dc71/*-json.log"
      - "/var/lib/docker/containers/761a06f18015d53f1fe37faaaa9b1003fb7b1ab633cb23650c55886ade3d8081/*-json.log"
      - "/var/lib/docker/containers/84327d373042e231dba7c3e8a84ff53cfcbca70fc7e18480b33f24422c566895/*-json.log"
      - "/var/lib/docker/containers/d0d089f3fa2d733f6ae9798a1834e4611c28a26edc5de0c1e48a97d76e415169/*-json.log"
processors:
  - add_cloud_metadata: ~
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - decode_json_fields:
      fields: ["message"]
      target: "json"
      overwrite_keys: true
      add_error_key: true
output.elasticsearch:
  hosts: '10.10.1.1:9200'
  username: 'elastic'
  password: '*********'
  index: 'test.log'
  space.id: 'default'
setup.template.name: "test"
setup.template.pattern: "test.log*"

However, when I check the Elasticsearch index, I can only see logs from Nginx and Filebeat itself. The logs from the other three services, including the streaming service, are missing. The primary concern is to view the logs from the streaming service, which have the following format:

06:16:00 DEBUG timeout {"stream": "6", "state": "⌛"} 06:16:00 INFO ⌛->🔴 {"stream": "6", "action": "timeout"} 06:16:02 DEBUG timeout {"stream": "10", "state": "⌛"} 06:16:02 INFO ⌛->🔴 {"stream": "10", "action": "timeout"} 06:16:11 INFO stop {"stream": "9", "type": "testFLV", "ID": "824672616960"} ...
I suspect that the log format might be the issue preventing the logs from the streaming service container from being collected in Elasticsearch.

[botelastic]

This issue doesn't have a Team:<team> label.

[botelastic]

Hi!
We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!