json log using multiline exception

[Iatbzh]

log:
{"message":"","thread":"cccccccf1-d147-ccc-cccccd-0cccccccccccc9","pageSysId":"64","href":"http://tset/test?from=%2test","pageSpDomain":"","@timestamp":"2022-09-03T15:29:54.272Z","time":"2022-09-03 15:29:54.271","level":"INFO","container":{"image":{"name":"test:test"},"id":"cccccccccccccccccccccccccccccccctteateststset"}}

filebeat config:
`filebeat.autodiscover:
providers:

• type: kubernetes
  #scope: cluster
  node: ${NODE_NAME}
  #unique: true
  templates:
• condition:
• contains:
  kubernetes.namespace: "sup"
  config:
• type: container
  paths:
• /var/log/containers/*${data.kubernetes.container.id}.log
  multiline:
  type: pattern
  pattern: '^\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}.\d{3}\s'
  negate: true
  match: after
  max_lines: 20000
  processors:

drop_fields:
fields:

• host.name
• ecs
• log
• agent
• input
• stream
• thread
• kubernetes.node.labels
• kubernetes.node.uid
• kubernetes.node.hostname
• kubernetes.replicaset
• kubernetes.namespace_uid
• kubernetes.labels.pod-template-hash
• container.runtime
  ignore_missing: true
  output.kafka:
  hosts: [test1:9092,test2:9092,test3:9092]
  topic: test
  partition.round_robin:
  reachable_only: false
  required_acks: 1
  compression: gzip
  max_message_bytes: 4194304`
  Result: This log will not be collected

[botelastic]

This issue doesn't have a Team:<team> label.

[mtojek]

Hi @Iatbzh! Please format your issue first using code tags to be clear about what is it.

[Iatbzh]

Sorry, I need to get a new code?

[Iatbzh]

filebeat config: filebeat.autodiscover:
providers:

type: kubernetes
#scope: cluster
node: ${NODE_NAME}
#unique: true
templates:
condition:
contains:
kubernetes.namespace: "sup"
config:
type: container
paths:
/var/log/containers/*${data.kubernetes.container.id}.log
multiline:
type: pattern
pattern: '^\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}.\d{3}\s'
negate: true
match: after
max_lines: 20000
processors:
drop_fields:
fields:

host.name
ecs
log
agent
input
stream
thread
kubernetes.node.labels
kubernetes.node.uid
kubernetes.node.hostname
kubernetes.replicaset
kubernetes.namespace_uid
kubernetes.labels.pod-template-hash
container.runtime
ignore_missing: true
output.kafka:
hosts: [test1:9092,test2:9092,test3:9092]
topic: test
partition.round_robin:
reachable_only: false
required_acks: 1
compression: gzip
max_message_bytes: 4194304``

[Iatbzh]

My problem is: when I use auto discovery to collect business container logs, the json log that takes up two lines cannot be collected

[mtojek]

As I said, please reformat your post using code tags.

[endorama]

Hello @Iatbzh, in order for us to follow up this issue we need it to be clearer than what it is now.

To help us debug this, please format your posts using the appropriate formatting; here is a link to the GitHub docs about formatting code.

You may want to refer at the documentation for handling multiline in filebeat and explain what is the bug here.
Please note that we use this issue tracker only for bugs and don't answer usage question here, if it's not a bug or you are unsure about it please ask this question on our Elastic Discuss forum first.

Thank you!

[botelastic]

Thank you very much for creating this issue. However, we would kindly like to ask you to post all questions and issues on the Discuss forum first. In addition to awesome, knowledgeable community contributors, core Beats developers are on the forums every single day to help you out as well. So, your questions will reach a wider audience there, and if we confirm that there is a bug, then you can reopen this issue with the new information or open a new one.

[botelastic]

Hi!
We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!