Example for body_key and headers_key

[dillonthompson]

I am attempting to use the body_key and the headers_key on my logs but i'm not sure i'm doing it correctly and the docs don't have any examples on how to do it. Can anyone help me out? below is an example of the errors i'm getting and my config

[ warn] [output:http:http.1] failed to extract body/headers using patterns ""$message"" and ""$headers""
[2022/08/08 12:37:45] [ warn] [output:http:http.0] error extracting headers using pattern "$headers"
[2022/08/08 12:37:45] [ warn] [output:http:http.1] failed to extract body/headers using patterns ""$message"" and ""$headers""
[2022/08/08 12:37:45] [ warn] [output:http:http.0] failed to extract body/headers using patterns "$message" and "$headers"
[2022/08/08 12:37:45] [ warn] [output:http:http.1] failed to extract body/headers using patterns ""$message"" and ""$headers""
[2022/08/08 12:37:45] [ warn] [output:http:http.0] error extracting headers using pattern "$headers"
[2022/08/08 12:37:45] [ warn] [output:http:http.1] failed to extract body/headers using patterns ""$message"" and ""$headers""
[2022/08/08 12:37:45] [ warn] [output:http:http.0] failed to extract body/headers using patterns "$message" and "$headers"
[2022/08/08 12:37:45] [error] [output:http:http.1] failed to post requests body key ""$message""
[2022/08/08 12:37:45] [ warn] [output:http:http.0] error extracting headers using pattern "$headers"
[2022/08/08 12:37:45] [ warn] [output:http:http.0] failed to extract body/headers using patterns "$message" and "$headers"
[2022/08/08 12:37:45] [ warn] [output:http:http.0] error extracting headers using pattern "$headers"
[2022/08/08 12:37:45] [ warn] [output:http:http.0] failed to extract body/headers using patterns "$message" and "$headers"
[2022/08/08 12:37:45] [error] [output:http:http.0] failed to post requests body key "$message"

here is my config:

[INPUT]
    NAME   dummy
    Dummy  {"level":"info","message":"{\"hello\":\"world\",\"headers\":\"headers\"}"}
    Tag logs

[OUTPUT]
    Name  http
    Match logs
    Host localhost
    Port 8080
    Uri /api/v1/write
    body_key $message
    headers_key $headers
    Format json
    json_date_key false

[dillonthompson]

I realized that my headers are not supposed to be a string. only the body.
so it should look like this

[INPUT]
    NAME   dummy
    Dummy  {"level":"info","message":"hello world", "headers":{"content-type":"application/json"}}
    Tag logs

[meehanman]

Is there a way to use headers_key and body_key without having to use body_key as text/plain?

For example I want to take the following example data in a single collector::

File1:

{ "level": "fatal", "log": "My Log Message1" }
{ "level": "fatal", "log": "My Log Message2" }

File2:

{ "level": "fatal", "log": "My Log Message3" }
{ "level": "fatal", "log": "My Log Message4" }

then use the Configuration to set the logs in a desired format that included the JSON Log + some header data eg:

pipeline:
  inputs:
  - name: tail
    tag: test:s1
    path: /source1.log
  - name: tail
    tag: test:s2
    path: /source2.log

  filters:
    - name: nest
      match: "*"
      operation: nest
      wildcard: '*'
      nest_under: event
    - name: modify
      match: "*"
      add: 
        - fields_role test-role
        - fields_billing_team engvis
    - name: nest
      match: "*"
      operation: nest
      wildcard: fields_*
      nest_under: fields
      remove_prefix: fields_

  outputs:
    - name: http
      match: "*"
      host: localhost
      port: 8081
      URI: /receiver/v1/http/000000-test-private-key-000000
      format: json_lines
      json_date_key: timestamp
      json_date_format: iso8601
      header_tag: FLUENT-TAG
      header:
        - environment dev
        - foo bar

A single event leaving this configuration would look like:

{
      "event": {
          "level": "fatal",
          "log": "My Log Message1"
      },
      "fields": {
        "role": "test-role",
        "datacenter": "dev"
      }
}

...with the HTTP request looking like:

{
  "method": "POST",
  "url": "...",
  "headers": {
    "Role": "test-role",
    "Fluent-Tag": "test:s1",
    "User-Agent": "Fluent-Bit",
    "Environment": "dev",
    "Foo": "bar",
    "Datacenter": "dev",
    "...": 0
  },
  "data": "{ \"level\": \"fatal\", \"log\": \"My Log Message1\" }\n{...}\n{...}\n"
}