Skip to content

Commit

Permalink
Merge branch 'main' into poc/es_state_store
Browse files Browse the repository at this point in the history
  • Loading branch information
aleksmaus authored Jan 7, 2025
2 parents 5ec7ed4 + 6474305 commit 904d8e8
Show file tree
Hide file tree
Showing 270 changed files with 37,120 additions and 13,089 deletions.
9 changes: 5 additions & 4 deletions .github/CODEOWNERS
Validating CODEOWNERS rules …
Original file line number Diff line number Diff line change
Expand Up @@ -26,9 +26,9 @@ CHANGELOG*
/.github/CODEOWNERS @elastic/beats-tech-leads
/auditbeat/ @elastic/sec-linux-platform
/deploy/ @elastic/elastic-agent-data-plane
/deploy/kubernetes @elastic/elastic-agent-data-plane @elastic/obs-cloudnative-monitoring
/deploy/kubernetes @elastic/elastic-agent-data-plane @elastic/elastic-agent-control-plane
/dev-tools/ @elastic/elastic-agent-data-plane
/dev-tools/kubernetes @elastic/obs-ds-hosted-services
/dev-tools/kubernetes @elastic/elastic-agent-data-plane @elastic/elastic-agent-control-plane
/docs/ @elastic/elastic-agent-data-plane
/filebeat @elastic/elastic-agent-data-plane
/filebeat/docs/ # Listed without an owner to avoid maintaining doc ownership for each input and module.
Expand Down Expand Up @@ -57,10 +57,11 @@ CHANGELOG*
/heartbeat/ @elastic/obs-ds-hosted-services
/journalbeat @elastic/elastic-agent-data-plane
/libbeat/ @elastic/elastic-agent-data-plane
/libbeat/autodiscover/providers/kubernetes @elastic/elastic-agent-data-plane @elastic/elastic-agent-control-plane
/libbeat/docs/processors-list.asciidoc @elastic/ingest-docs
/libbeat/management @elastic/elastic-agent-control-plane
/libbeat/processors/add_cloud_metadata @elastic/obs-ds-hosted-services
/libbeat/processors/add_kubernetes_metadata @elastic/obs-cloudnative-monitoring
/libbeat/processors/add_kubernetes_metadata @elastic/elastic-agent-data-plane
/libbeat/processors/cache/ @elastic/security-service-integrations
/libbeat/processors/community_id/ @elastic/sec-deployment-and-devices
/libbeat/processors/decode_xml/ @elastic/security-service-integrations
Expand Down Expand Up @@ -105,7 +106,6 @@ CHANGELOG*
/metricbeat/module/system/ @elastic/elastic-agent-data-plane
/metricbeat/module/vsphere @elastic/obs-infraobs-integrations
/metricbeat/module/zookeeper @elastic/obs-infraobs-integrations
/metricbeat/tests @elastic/ingest-eng-prod
/packetbeat/ @elastic/sec-linux-platform
/script/ @elastic/elastic-agent-data-plane
/testing/ @elastic/elastic-agent-data-plane
Expand Down Expand Up @@ -223,6 +223,7 @@ CHANGELOG*
/x-pack/metricbeat/module/iis @elastic/obs-infraobs-integrations
/x-pack/metricbeat/module/istio/ @elastic/obs-cloudnative-monitoring
/x-pack/metricbeat/module/mssql @elastic/obs-infraobs-integrations
/x-pack/metricbeat/module/openai @elastic/obs-infraobs-integrations
/x-pack/metricbeat/module/oracle @elastic/obs-infraobs-integrations
/x-pack/metricbeat/module/panw @elastic/obs-infraobs-integrations
/x-pack/metricbeat/module/prometheus/ @elastic/obs-cloudnative-monitoring
Expand Down
5 changes: 5 additions & 0 deletions .github/workflows/check-default.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,11 @@ jobs:
- uses: actions/setup-go@v5
with:
go-version-file: .go-version
#  when using ubuntu-latest, python 3.10 is not the default version.
- name: Fix Code is not compatible with Python 3.12
uses: actions/setup-python@v4
with:
python-version: '3.10'
- name: Run check-default
run: |
go install github.com/magefile/mage
Expand Down
5 changes: 5 additions & 0 deletions .github/workflows/check-docs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,11 @@ jobs:
run: sudo apt-get install -y libsystemd-dev
- name: Install librpm-dev
run: sudo apt-get install -y librpm-dev
#  when using ubuntu-latest, python 3.10 is not the default version.
- name: Fix Code is not compatible with Python 3.12
uses: actions/setup-python@v4
with:
python-version: '3.10'
- name: Run check
run: |
make check
2 changes: 1 addition & 1 deletion .go-version
Original file line number Diff line number Diff line change
@@ -1 +1 @@
1.22.9
1.22.10
8 changes: 4 additions & 4 deletions .golangci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -152,7 +152,7 @@ linters-settings:

gosimple:
# Select the Go version to target. The default is '1.13'.
go: "1.22.9"
go: "1.22.10"

nakedret:
# make an issue if func has more lines of code than this setting and it has naked returns; default is 30
Expand All @@ -170,19 +170,19 @@ linters-settings:

staticcheck:
# Select the Go version to target. The default is '1.13'.
go: "1.22.9"
go: "1.22.10"
checks: ["all"]

stylecheck:
# Select the Go version to target. The default is '1.13'.
go: "1.22.9"
go: "1.22.10"
# Disabled:
# ST1005: error strings should not be capitalized
checks: ["all", "-ST1005"]

unused:
# Select the Go version to target. The default is '1.13'.
go: "1.22.9"
go: "1.22.10"

gosec:
excludes:
Expand Down
1 change: 1 addition & 0 deletions CHANGELOG-developer.next.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -108,6 +108,7 @@ The list below covers the major changes between 7.0.0-rc2 and main only.
- AWS CloudWatch Metrics record previous endTime to use for next collection period and change log.logger from cloudwatch to aws.cloudwatch. {pull}40870[40870]
- Fix flaky test in cel and httpjson inputs of filebeat. {issue}40503[40503] {pull}41358[41358]
- Fix documentation and implementation of raw message handling in Filebeat http_endpoint by removing it. {pull}41498[41498]
- Fix flaky test in filebeat Okta entity analytics provider. {issue}42059[42059] {pull}42123[42123]

==== Added

Expand Down
85 changes: 85 additions & 0 deletions CHANGELOG.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,91 @@
:issue: https://github.com/elastic/beats/issues/
:pull: https://github.com/elastic/beats/pull/

[[release-notes-8.17.0]]
=== Beats version 8.17.0
https://github.com/elastic/beats/compare/v8.16.1\...v8.17.0[View commits]

==== Known issue

- Standalone Beats docker image will not start if `-e` option is not added {issue}42038[42038].

==== Breaking changes

*Affecting all Beats*

- Drop support for Debian 10 and upgrade statically linked glibc from 2.28 to 2.31. {pull}41402[41402]

==== Bugfixes

*Affecting all Beats*

- Ensure Elasticsearch output can always recover from network errors. {pull}40794[40794]
- Add `translate_ldap_attribute` processor. {pull}41472[41472]
- Remove unnecessary debug logs during idle connection teardown. {issue}40824[40824]
- Remove unnecessary reload for Elastic Agent managed beats when APM tracing config changes from nil to nil. {pull}41794[41794]

*Auditbeat*

- auditd: Use ECS `event.type: end` instead of `stop` for SERVICE_STOP, DAEMON_ABORT, and DAEMON_END messages. {pull}41558[41558]
- auditd: Update syscall names for Linux 6.11. {pull}41558[41558]
- hasher: Geneneral improvements and fixes. {pull}41863[41863]

*Filebeat*

- Fix double encoding of client_secret in the Entity Analytics input's Azure Active Directory provider. {pull}41393[41393]
- Add support for Access Points in the `aws-s3` input. {pull}41495[41495]
- Fix the "No such input type exist: 'salesforce'" error on the Windows/AIX platform. {pull}41664[41664]
- Fix handling of http_endpoint request exceeding memory limits. {issue}41764[41764] {pull}41765[41765]
- Fixes filestream logging the error "filestream input with ID 'ID' already exists, this will lead to data duplication[...]" on Kubernetes when using autodiscover. {pull}41585[41585]

*Metricbeat*

- Log Cisco Meraki `getDevicePerformanceScores` errors without stopping metrics collection. {pull}41622[41622]
- Fix incorrect handling of types in SQL module. {issue}40090[40090] {pull}41607[41607]

*Winlogbeat*

- Fix message handling in the experimental API. {issue}19338[19338] {pull}41730[41730]

==== Added

*Affecting all Beats*

- Add `lowercase` processor. {issue}22254[22254] {pull}41424[41424]
- Add `uppercase` processor. {issue}22254[22254] {pull}41535[41535]
- Replace `compress/gzip` with https://github.com/klauspost/compress/gzip library for gzip compression. {pull}41584[41584]

*Auditbeat*

- Split module/system/process into common and provider bits. {pull}41868[41868]

*Filebeat*

- Improved Azure Blob Storage input documentation. {pull}41252[41252]
- Make ETW input GA. {pull}41389[41389]
- Added input metrics to GCS input. {issue}36640[36640] {pull}41505[41505]
- Add support for Okta entity analytics provider to collect role and factor data for users. {pull}41460[41460]
- Add support for Journald in the System module. {pull}41555[41555]
- Improve S3 polling mode states registry when using list prefix option. {pull}41869[41869]
- AWS S3 input registry cleanup for untracked s3 objects. {pull}41694[41694]
- The environment variable `BEATS_AZURE_EVENTHUB_INPUT_TRACING_ENABLED: true` enables internal logs tracer for the azure-eventhub input. {issue}41931[41931] {pull}41932[41932]

*Libbeat*

- Enrich events with EC2 tags in add_cloud_metadata processor. {pull}41477[41477]

*Metricbeat*

- Add `id` field to all the vSphere metricsets. {pull}41097[41097]
- Bump aerospike-client-go to version v7.7.1 and add support for basic auth in Aerospike module. {pull}41233[41233]
- Add support for region/zone for Vertex AI service in GCP module. {pull}41551[41551]
- Add support for location label as an optional configuration parameter in GCP metrics metricset. {issue}41550[41550] {pull}41626[41626]

*Winlogbeat*

- Add handling for missing `EvtVarType`s in experimental API. {issue}19337[19337] {pull}41418[41418]
- Implement exclusion range support for event_id. {issue}38623[38623] {pull}41639[41639]

[[release-notes-8.16.1]]
=== Beats version 8.16.1
https://github.com/elastic/beats/compare/v8.16.0\...v8.16.1[View commits]
Expand Down
31 changes: 29 additions & 2 deletions CHANGELOG.next.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ https://github.com/elastic/beats/compare/v8.8.1\...main[Check the HEAD diff]
- Drop support for Debian 10 and upgrade statically linked glibc from 2.28 to 2.31 {pull}41402[41402]
- Fix metrics not being ingested, due to "Limit of total fields [10000] has been exceeded while adding new fields [...]". The total fields limit has been increased to 12500. No significant performance impact on Elasticsearch is anticipated. {pull}41640[41640]
- Set default kafka version to 2.1.0 in kafka output and filebeat. {pull}41662[41662]
- Fix templates and docs to use correct `--` version of command line arguments. {issue}42038[42038] {pull}42060[42060]

*Auditbeat*

Expand Down Expand Up @@ -52,6 +53,9 @@ https://github.com/elastic/beats/compare/v8.8.1\...main[Check the HEAD diff]
- Fixes filestream logging the error "filestream input with ID 'ID' already exists, this will lead to data duplication[...]" on Kubernetes when using autodiscover. {pull}41585[41585]
- Add kafka compression support for ZSTD.
- Filebeat fails to start if there is any input with a duplicated ID. It logs the duplicated IDs and the offending inputs configurations. {pull}41731[41731]
- Filestream inputs with duplicated IDs will fail to start. An error is logged showing the ID and the full input configuration. {issue}41938[41938] {pull}41954[41954]
- Filestream inputs can define `allow_deprecated_id_duplication: true` to run keep the previous behaviour of running inputs with duplicated IDs. {issue}41938[41938] {pull}41954[41954]
- The Filestream input only starts to ingest a file when it is >= 1024 bytes in size. This happens because the fingerprint` is the default file identity now. To restore the previous behaviour, set `file_identity.native: ~` and `prospector.scanner.fingerprint.enabled: false` {issue}40197[40197] {pull}41762[41762]

*Heartbeat*

Expand Down Expand Up @@ -127,6 +131,8 @@ https://github.com/elastic/beats/compare/v8.8.1\...main[Check the HEAD diff]
- auditd: Use ECS `event.type: end` instead of `stop` for SERVICE_STOP, DAEMON_ABORT, and DAEMON_END messages. {pull}41558[41558]
- auditd: Update syscall names for Linux 6.11. {pull}41558[41558]
- hasher: Geneneral improvements and fixes. {pull}41863[41863]
- hasher: Add a cached hasher for upcoming backend. {pull}41952[41952]
- Split common tty definitions. {pull}42004[42004]

*Filebeat*

Expand Down Expand Up @@ -192,6 +198,12 @@ https://github.com/elastic/beats/compare/v8.8.1\...main[Check the HEAD diff]
- Fix handling of http_endpoint request exceeding memory limits. {issue}41764[41764] {pull}41765[41765]
- Rate limiting fixes in the Okta provider of the Entity Analytics input. {issue}40106[40106] {pull}41583[41583]
- Redact authorization headers in HTTPJSON debug logs. {pull}41920[41920]
- Further rate limiting fix in the Okta provider of the Entity Analytics input. {issue}40106[40106] {pull}41977[41977]
- Fix streaming input handling of invalid or empty websocket messages. {pull}42036[42036]
- Fix awss3 document ID construction when using the CSV decoder. {pull}42019[42019]
- The `_id` generation process for S3 events has been updated to incorporate the LastModified field. This enhancement ensures that the `_id` is unique. {pull}42078[42078]
- Fix Netflow Template Sharing configuration handling. {pull}42080[42080]
- Updated websocket retry error code list to allow more scenarios to be retried which could have been missed previously. {pull}42218[42218]

*Heartbeat*

Expand Down Expand Up @@ -225,7 +237,9 @@ https://github.com/elastic/beats/compare/v8.8.1\...main[Check the HEAD diff]
- Do not report non-existant 0 values for RSS metrics in docker/memory {pull}41449[41449]
- Log Cisco Meraki `getDevicePerformanceScores` errors without stopping metrics collection. {pull}41622[41622]
- Don't skip first bucket value in GCP metrics metricset for distribution type metrics {pull}41822[41822]

- [K8s Integration] Enhance HTTP authentication in case of token updates for Apiserver, Controllermanager and Scheduler metricsets {issue}41910[41910] {pull}42016[42016]
- Fixed `creation_date` scientific notation output in the `elasticsearch.index` metricset. {pull}42053[42053]
- Fix bug where metricbeat unintentionally triggers Windows ASR. {pull}42177[42177]

*Osquerybeat*

Expand Down Expand Up @@ -256,12 +270,13 @@ https://github.com/elastic/beats/compare/v8.8.1\...main[Check the HEAD diff]
- The environment variable `BEATS_ADD_CLOUD_METADATA_PROVIDERS` overrides configured/default `add_cloud_metadata` providers {pull}38669[38669]
- When running under Elastic-Agent Kafka output allows dynamic topic in `topic` field {pull}40415[40415]
- The script processor has a new configuration option that only uses the cached javascript sessions and prevents the creation of new javascript sessions.
- Update to Go 1.22.7. {pull}41018[41018]
- Update to Go 1.22.10. {pull}42095[42095]
- Replace Ubuntu 20.04 with 24.04 for Docker base images {issue}40743[40743] {pull}40942[40942]
- Reduce memory consumption of k8s autodiscovery and the add_kubernetes_metadata processor when Deployment metadata is enabled
- Add `lowercase` processor. {issue}22254[22254] {pull}41424[41424]
- Add `uppercase` processor. {issue}22254[22254] {pull}41535[41535]
- Replace `compress/gzip` with https://github.com/klauspost/compress/gzip library for gzip compression {pull}41584[41584]
- Add regex pattern matching to add_kubernetes_metadata processor {pull}41903[41903]

*Auditbeat*

Expand Down Expand Up @@ -357,12 +372,18 @@ https://github.com/elastic/beats/compare/v8.8.1\...main[Check the HEAD diff]
- Add ability to remove request trace logs from entityanalytics input. {pull}40004[40004]
- Refactor & cleanup with updates to default values and documentation. {pull}41834[41834]
- Update CEL mito extensions to v1.16.0. {pull}41727[41727]
- Add `unifiedlogs` input for MacOS. {pull}41791[41791]
- Add evaluation state dump debugging option to CEL input. {pull}41335[41335]
- Added support for retry configuration in GCS input. {issue}11580[11580] {pull}41862[41862]
- Improve S3 polling mode states registry when using list prefix option. {pull}41869[41869]
- Add support for SSL and Proxy configurations for websoket type in streaming input. {pull}41934[41934]
- AWS S3 input registry cleanup for untracked s3 objects. {pull}41694[41694]
- The environment variable `BEATS_AZURE_EVENTHUB_INPUT_TRACING_ENABLED: true` enables internal logs tracer for the azure-eventhub input. {issue}41931[41931] {pull}41932[41932]
- The Filestream input now uses the `fingerprint` file identity by default. The state from files are automatically migrated if the previous file identity was `native` (the default) or `path`. If the `file_identity` is explicitly set, there is no change in behaviour. {issue}40197[40197] {pull}41762[41762]
- Rate limiting operability improvements in the Okta provider of the Entity Analytics input. {issue}40106[40106] {pull}41977[41977]
- Added default values in the streaming input for websocket retries and put a cap on retry wait time to be lesser than equal to the maximum defined wait time. {pull}42012[42012]
- Rate limiting fault tolerance improvements in the Okta provider of the Entity Analytics input. {issue}40106[40106] {pull}42094[42094]
- Added infinite & blanket retry options to websockets and improved logging and retry logic. {pull}42225[42225]

*Auditbeat*

Expand All @@ -376,6 +397,7 @@ https://github.com/elastic/beats/compare/v8.8.1\...main[Check the HEAD diff]

- Added status to monitor run log report.
- Upgrade node to latest LTS v18.20.3. {pull}40038[40038]
- Add support for RFC7231 methods to http monitors. {pull}41975[41975]

*Metricbeat*

Expand Down Expand Up @@ -414,6 +436,9 @@ https://github.com/elastic/beats/compare/v8.8.1\...main[Check the HEAD diff]
- Only watch metadata for ReplicaSets in metricbeat k8s module {pull}41289[41289]
- Add support for region/zone for Vertex AI service in GCP module {pull}41551[41551]
- Add support for location label as an optional configuration parameter in GCP metrics metricset. {issue}41550[41550] {pull}41626[41626]
- Collect .NET CLR (IIS) Memory, Exceptions and LocksAndThreads metrics {pull}41929[41929]
- Added `tier_preference`, `creation_date` and `version` fields to the `elasticsearch.index` metricset. {pull}41944[41944]
- Add `use_performance_counters` to collect CPU metrics using performance counters on Windows for `system/cpu` and `system/core` {pull}41965[41965]

*Metricbeat*
- Add benchmark module {pull}41801[41801]
Expand All @@ -436,6 +461,8 @@ https://github.com/elastic/beats/compare/v8.8.1\...main[Check the HEAD diff]
- Format embedded messages in the experimental api {pull}41525[41525]
- Implement exclusion range support for event_id. {issue}38623[38623] {pull}41639[41639]
- Make the experimental API GA and rename it to winlogbeat-raw {issue}39580[39580] {pull}41770[41770]
- Remove 22 clause limitation {issue}35047[35047] {pull}42187[42187]
- Add handling for recoverable publisher disabled errors {issue}35316[35316] {pull}42187[42187]


*Functionbeat*
Expand Down
Loading

0 comments on commit 904d8e8

Please sign in to comment.