Provided Grok expressions do not match field value

[cdalexndr]

For confirmed bugs, please report:

• Version: filebeat 7.10.1
• Operating System: docker
• Discuss Forum URL:
• Steps to Reproduce:

Using filebeat with nginx module.

Resulting document:

{
  "_index": ".ds-log-entry-000001",
  "_type": "_doc",
  "_id": "1QQ1X3cBSAHugCJe9Dwt",
  "_version": 1,
  "_score": null,
  "_source": {
    "container": {
      "image": {
        "name": "nginx:1.17.3-alpine"
      },
      "name": "docker_entry_1",
      "id": "b7ce66bf2d6d3fd63e450b872ce4e94f1a97d9feb9c6bc622d78c1ae97ac3097"
    },
    "agent": {
      "hostname": "22e378103f12",
      "name": "22e378103f12",
      "id": "9e89fb4f-3303-4822-9b8e-44969a188768",
      "ephemeral_id": "35faa175-e5d6-452d-be0a-8a0cc32646f3",
      "type": "filebeat",
      "version": "7.10.1"
    },
    "log": {
      "file": {
        "path": "/var/lib/docker/containers/b7ce66bf2d6d3fd63e450b872ce4e94f1a97d9feb9c6bc622d78c1ae97ac3097/b7ce66bf2d6d3fd63e450b872ce4e94f1a97d9feb9c6bc622d78c1ae97ac3097-json.log"
      },
      "offset": 2833660
    },
    "message": "172.18.0.1 - boss [01/Feb/2021:20:08:19 +0000] \"POST /api/console/proxy?path=_template&method=GET HTTP/2.0\" 200 67217 \"https://localhost:5601/app/dev_tools\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:85.0) Gecko/20100101 Firefox/85.0\" \"-\"",
    "fileset": {
      "name": "ingress_controller"
    },
    "error": {
      "message": "Provided Grok expressions do not match field value: [172.18.0.1 - boss [01/Feb/2021:20:08:19 +0000] \\\"POST /api/console/proxy?path=_template&method=GET HTTP/2.0\\\" 200 67217 \\\"https://localhost:5601/app/dev_tools\\\" \\\"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:85.0) Gecko/20100101 Firefox/85.0\\\" \\\"-\\\"]"
    },
    "docker": {
      "container": {
        "labels": {
          "co_elastic_logs/module": "nginx",
          "desktop_docker_io/binds/1/SourceKind": "hostFile",
          "app": "entry",
          "com_docker_compose_config-hash": "b44b4e2a184f8787bfcad8adfa168f52dfefe7770d2529a9a614ab88393b391b",
          "desktop_docker_io/binds/2/Target": "/etc/ssl",
          "desktop_docker_io/binds/1/Target": "/etc/nginx/conf",
          "desktop_docker_io/binds/0/Target": "/var/www/pse",
          "com_docker_compose_oneoff": "False",
          "desktop_docker_io/binds/0/Source": "/c/Developer/pse/source/build/docker/config/nginx/pages",
          "com_docker_compose_project": "docker",
          "com_docker_compose_project_config_files": "docker-compose.yml",
          "desktop_docker_io/binds/3/SourceKind": "hostFile",
          "desktop_docker_io/binds/3/Target": "/etc/nginx/conf.d/default.conf",
          "maintainer": "NGINX Docker Maintainers <docker-maint@nginx.com>",
          "desktop_docker_io/binds/0/SourceKind": "hostFile",
          "desktop_docker_io/binds/3/Source": "/c/Developer/pse/source/build/docker/config/nginx/nginx.conf",
          "desktop_docker_io/binds/2/SourceKind": "hostFile",
          "desktop_docker_io/binds/1/Source": "/c/Developer/pse/source/build/docker/config/nginx/conf",
          "desktop_docker_io/binds/2/Source": "/c/Developer/pse/source/build/docker/config/nginx/ssl",
          "com_docker_compose_service": "entry",
          "com_docker_compose_container-number": "1",
          "com_docker_compose_version": "1.27.4",
          "com_docker_compose_project_working_dir": "C:\\Developer\\pse\\source\\build\\docker"
        }
      }
    },
    "input": {
      "type": "container"
    },
    "@timestamp": "2021-02-01T20:08:19.156Z",
    "ecs": {
      "version": "1.5.0"
    },
    "stream": "stdout",
    "service": {
      "type": "nginx"
    },
    "host": {
      "name": "22e378103f12"
    },
    "event": {
      "ingested": "2021-02-01T20:08:27.430898400Z",
      "timezone": "+00:00",
      "module": "nginx",
      "dataset": "nginx.ingress_controller"
    }
  },
  "fields": {
    "@timestamp": [
      "2021-02-01T20:08:19.156Z"
    ]
  },
  "highlight": {
    "docker.container.labels.app": [
      "@kibana-highlighted-field@entry@/kibana-highlighted-field@"
    ]
  },
  "sort": [
    1612210099156
  ]
}

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[whataboutpereira]

Still getting Provided Grok expressions do not match field value in 7.15.2.

[whataboutpereira]

172.18.0.1 - boss [01/Feb/2021:20:08:19 +0000] "POST /api/console/proxy?path=_template&method=GET HTTP/2.0" 200 67217 "https://localhost:5601/app/dev_tools" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:85.0) Gecko/20100101 Firefox/85.0" "-"

I did some testing and I can see it's tripping up on the last "-" which should be http_x_forwarded_for. Filebeat parses the string up until the user agent string once I remove all that follows from the grok pattern.

log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                  '$status $body_bytes_sent "$http_referer" '
                  '"$http_user_agent" "$http_x_forwarded_for"';

[whataboutpereira]

Okay, I've solved the problem with a little bit more digging.

It seems nginx ingress_controller is another kind of format and it seems to default to being enabled if you enable the nginx module and don't explicitly disable it. I noticed I had duplicate log messages - nginx.access which parsed nicely and a duplicate nginx.ingress_controller which was erroring.

I would say this is a false alarm, because ingress_controller should be disabled in filebeat module configuration if one doesn't have these logs.

[gcleaves]

Any idea how to disable nginx ingress_controller when using the Filebeat docker image?

Edit:
It seems that adding the access and error fileset labels succeeds in disabling ingress, docker-compose snippet:

    labels:
      - "co.elastic.logs/module=nginx"
      - "co.elastic.logs/fileset.stdout=access"
      - "co.elastic.logs/fileset.stderr=error"

[botelastic]

Hi!
We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!

[brsolomon-deloitte]

not stale

[botelastic]

Hi!
We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!