elastic-agent system module - filesystem.ignore_types in metricbeat is not being applied

[TheRiffRafi]

• Version: 8.9.0

• Operating System: Windows

• Steps to Reproduce:

1. Have a Elasticsearch - Fleet - Elastic Agent setup.
2. Install the "system" integration to an agent. Go to the integrations settings, the metrics section, filesystem, and add to the List of filesystem types to ignore the unknown and unavailable values in a row each.
3. Save and observe how the Elastic Agent logs still show the error:
   {"log.level":"error","@timestamp":"2023-04-06T08:59:18.495+0200","log.origin":{"[file.name](https://file.name/)":"module/wrapper.go","file.line":256},"message":"Error fetching data for metricset system.filesystem: error getting filesystem usage for [Z:\\](file:///Z://): GetDiskFreeSpaceEx failed: The device is not ready.","[service.name](https://service.name/)":"metricbeat","ecs.version":"1.6.0"}

Something worth pointing out.

I don't know if the way the list is built has something to do with this.
I went looking into how the ignore_types is done in Metricbeat (no Elastic Agent paired) and the values are each part of an array:

https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-metricset-system-filesystem.html#_filtering

When looking at the Elastic Agent diagnostic, I see that the values are passed as a string-value:

 - kind:
                    structvalue:
                      fields:
                        data_stream:
                          kind:
                            structvalue:
                              fields:
                                dataset:
                                  kind:
                                    stringvalue: system.filesystem
                                type:
                                  kind:
                                    stringvalue: metrics
                        filesystem.ignore_types:
                          kind:
                            stringvalue: unknown,unavailable,cdfs

And looking at the yml file from the policy, it comes up as a string:

        filesystem.ignore_types: 'unknown,unavailable,cdfs'

I don't know enough on how the options sent to the Metricbeat process are built so my guess is probably completely wrong, but I thought it was worth pointing out.

I've seen this setting not working twice. Here is another reference: elastic/beats#35187 (comment)

[cmacknz]

@fearful-symmetry any ideas on this one?

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[TheRiffRafi]

This is probably not related to my Array/String suspicion.
We attempted to set the option with a single value, like this: filesystem.ignore_types: 'unknown'
But that also didn't work.

[fearful-symmetry]

Yah, this appears to be a string/array issue, and we're just passing the array like a string: https://github.com/elastic/integrations/blob/main/packages/system/data_stream/filesystem/agent/stream/stream.yml.hbs

[lucabelluccini]

Great investigation @TheRiffRafi

I've checked the manifest for the System integration and we have:

      - name: filesystem.ignore_types
        type: text
        title: List of filesystem types to ignore
        multi: true
        required: false
        show_user: true

In the UI, it is shown as:

There are actually 3 possible states:

1. we need to allow users to NOT define at all filesystem.ignore_types, so it goes to defaults by Filebeat (as per our doc)
2. we need to allow users to define filesystem.ignore_types with a list of values
3. we need to allow users to define filesystem.ignore_types as empty array, so that they really ignore no type

The following snippet could cover (1) and (2), but I do not know how we can express (2) with the manifest we have.

{{#if filesystem.ignore_types}}
filesystem.ignore_types:
{{#each filesystem.ignore_types as |ignored_type i|}}
  - {{ignored_type}}
{{/each}}
{{/if}}

The use case (3) is pretty rare.

Are we planning for a fix for this?
The suggestion above should not be a breaking change, but if we find a solution we need to be careful to not break existing policies (e.g. for users who never defined filesystem.ignore_types, it must behave as now (Filebeat defaults).

[willem-dhaese]

We get a lot of the following errors on Windows Servers:

error getting filesystem usage for D:\: GetDiskFreeSpaceEx failed: The device is not ready.

D:\ is a virtual DVD Drive

Will the resolution of this issue resolve these errors. What filesystem type do we need to exclude?

[TheRiffRafi]

Hi @willem-dhaese, this fix will target a bug where an "array" can't be provided as a list of filesystem types to exclude. So once this fix is addressed, then you can add "unknown" AND "unavailable" as the filesystem types to exclude and that will resolve the issue.

If you are not running an elastic-agent integration, and instead you are running metricbeat as a standalone application, then you just have to add those types in the exclude list as shown in this comment: elastic/beats#35187 (comment)

[StefanSa]

@TheRiffRafi Hi Rafi,
when will there be a solution for this, have the same problems as @willem-dhaese .

[S3l3ct3d]

Any updates on this? I am seeing the same issues.

[lucabelluccini]

I think in order to not generate breaking changes, I wonder if we could change Metricbeat to parse comma separated values as separate items.

[romain-chanu]

Another user reported the same problem (Elastic Agent / version 8.12.x).

[fearful-symmetry]

@pierrehilbert do we want to put this on the board or plan for it? It seems like a simple fix in the integrations code.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)