migrate libbeat pipeline

[sharbuz]

What is the problem this PR solves?

Jenkins->Buildkite pipelines migration

How does this PR solve the problem?

1. Migrate the libbeat pipeline
   https://buildkite.com/elastic/beats-libbeat/builds?branch=sharbuz%3Amigrate-libbeat-pipeline
2. Update the common conditions/rules for starting the pipelines (excluded the part from the main pipeline)

Examples of the builds:
Metricbeat: https://buildkite.com/elastic/beats-metricbeat/builds/1992
Libbeat: https://buildkite.com/elastic/beats-libbeat/builds/700

Related issues

https://github.com/elastic/ingest-dev/issues/1693

[botelastic]

This pull request doesn't have a Team:<team> label.

[sharbuz]

The pipeline will finish with errors because of:
#37873
#37870
@v1v suggested using different PRs for that.
All other remarks have been fixed.
CC @dliappis

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b migrate-libbeat-pipeline upstream/migrate-libbeat-pipeline
git merge upstream/main
git push upstream migrate-libbeat-pipeline

[sharbuz]

/test

[elasticmachine]

💔 Build Failed

• Buildkite Build
• Commit: b272833

Failed CI Steps

• Package pipeline

History

• 💔 Build #2568 failed 5bfd917
• 💔 Build #2560 failed adf168a
• 💔 Build #2545 failed 5d5ceb8
• 💔 Build #2538 failed 5aaf276
• 💔 Build #2528 failed 6ee43ff
• 💔 Build #2470 failed 3774483

cc @sharbuz

[sharbuz]

/test

[sharbuz]

/test

[jlind23]

Force merging this as the failure is unrelated. @sharbuz will create a follow up issue for @axw and @bturquet to investigate.