Beats - The Lightweight Shippers of the Elastic Stack

The Beats are lightweight data shippers, written in Go, that you install on your servers to capture all sorts of operational data (think of logs, metrics, or network packet data). The Beats send the operational data to Elasticsearch, either directly or via Logstash, so it can be visualized with Kibana.

By "lightweight", we mean that Beats have a small installation footprint, use limited system resources, and have no runtime dependencies.

This repository contains libbeat, our Go framework for creating Beats, and all the officially supported Beats:

Beat	Description
Auditbeat	Collect your Linux audit framework data and monitor the integrity of your files.
Filebeat	Tails and ships log files
Functionbeat	Read and ships events from serverless infrastructure.
Heartbeat	Ping remote services for availability
Journalbeat	Read and ships event from Journald.
Metricbeat	Fetches sets of metrics from the operating system and services
Packetbeat	Monitors the network and applications by sniffing packets
Winlogbeat	Fetches and ships Windows Event logs

In addition to the above Beats, which are officially supported by Elastic, the community has created a set of other Beats that make use of libbeat but live outside of this Github repository. We maintain a list of community Beats here.

Documentation and Getting Started

You can find the documentation and getting started guides for each of the Beats on the elastic.co site:

• Beats platform
• Auditbeat
• Filebeat
• Functionbeat
• Heartbeat
• Journalbeat
• Metricbeat
• Packetbeat
• Winlogbeat

Documentation and Getting Started information for the Elastic Agent

You can find the documentation and getting started guides for the Elastic Agent on the elastic.co site

Getting Help

If you need help or hit an issue, please start by opening a topic on our discuss forums. Please note that we reserve GitHub tickets for confirmed bugs and enhancement requests.

Downloads

You can download pre-compiled Beats binaries, as well as packages for the supported platforms, from this page.

Contributing

We'd love working with you! You can help make the Beats better in many ways: report issues, help us reproduce issues, fix bugs, add functionality, or even create your own Beat.

Please start by reading our CONTRIBUTING file.

If you are creating a new Beat, you don't need to submit the code to this repository. You can simply start working in a new repository and make use of the libbeat packages, by following our developer guide. After you have a working prototype, open a pull request to add your Beat to the list of community Beats.

Building Beats from the Source

See our CONTRIBUTING file for information about setting up your dev environment to build Beats from the source.

Snapshots

For testing purposes, we generate snapshot builds that you can find here. Please be aware that these are built on top of master and are not meant for production.

CI

PR Comments

It is possible to trigger some jobs by putting a comment on a GitHub PR. (This service is only available for users affiliated with Elastic and not for open-source contributors.)

• beats
  • jenkins run the tests please or jenkins run tests or /test will kick off a default build.
  • /test macos will kick off a default build with also the macos stages.
  • /test <beat-name> will kick off the default build for the given PR in addition to the <beat-name> build itself.
  • /test <beat-name> for macos will kick off a default build with also the macos stage for the <beat-name>.
• apm-beats-update
  • /run apm-beats-update
• apm-beats-packaging
  • /package or /packaging will kick of a build to generate the packages for beats.
• apm-beats-tester
  • /beats-tester will kick of a build to validate the generated packages.

PR Labels

It's possible to configure the build on a GitHub PR by labelling the PR with the below labels

• <beat-name> to force the following builds to run the stages for the <beat-name>
• macOS to force the following builds to run the macos stages.

Manual Filebeat Compilation

You need to install mage tool on your local environment. Mage is a make/rake-like build tool using Go. You write plain-old go functions, and Mage automatically uses them as Makefile-like runnable targets. For more information ...

go version # go1.18.4
go install github.com/magefile/mage@latest

You need to run make crosscompile command. The below code block will create a binary named filebeat-linux-amd64 in filebeat/build/bin directory

cd /opt/
git clone https://github.com/picusnext/beats.git
cd beats
make crosscompile
ls /opt/beats/filebeat/build/bin/filebeat-linux-amd64

Filebeat Custom Processor

All our processor in libbeat/processor dir. To create a custom processor, you can take add_docker_metadata as an example.

• Every processor has Run function. It behaves like a middleware. We can kustomize it. Also, we have to add our new fields to the _meta/fields.yml . Otherwise when processor is being loaded, our fields will not be seen in Elasticsearch.

Example:

libbeat/processor/add_docker_metadata/add_docker_metadata.go

func (d *addDockerMetadata) Run(event *beat.Event) (*beat.Event, error) {

  .
  .
  .

  meta := common.MapStr{}
  meta.Put("container.environment.name", "prod")
  event.Fields.DeepUpdate(meta.Clone())

  return event, nil
}

libbeat/processor/add_docker_metadata/_meta/fields.yml

- name: container.environment
  type: object
  object_type: keyword
  description: >
    Image Tags.