Auditbeat pipeline migration to Buildkite #37668

oakrizan · 2024-01-18T14:02:02Z

Proposed commit message

Migration of auditbeat pipeline to Buildkite
Phase according to migration plan: 1
Phase scope:
Migrate separate Jenkinsfiles/Groovy scripts to Buildkite pipelines with following scope:

Execution conditions - only:

changeset
trigger comments

Agents - migrate those available both Jenkinsfile and those that are available in Buildkite
Packaging - as dry-run (without publishing DRA, since it's highly related to execution conditions)

Link to Bildkite branch builds: https://buildkite.com/elastic/auditbeat/builds?branch=oakrizan%3Aauditbeat-pipeline-migration

Metricbeat build fixes are done in PR: #37725

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

[ ]

How to test this PR locally

Related issues

Related to 1693

Use cases

Screenshots

Logs

elasticmachine · 2024-01-18T14:23:04Z

❕ Build Aborted

Expand to view the summary

Build stats

Start Time: 2024-01-18T14:07:49.303+0000
Duration: 15 min 10 sec

Test stats 🧪

Test	Results
Failed	0
Passed	3
Skipped	0
Total	3

Steps errors

Expand to view the steps failures

`Error signal`

Took 0 min 0 sec . View more details here
Description: Error 'org.jenkinsci.plugins.workflow.steps.FlowInterruptedException'

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

elasticmachine · 2024-01-18T14:29:12Z

❕ Build Aborted

Expand to view the summary

Build stats

Duration: 11 min 5 sec

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

elasticmachine · 2024-01-18T14:31:02Z

❕ Build Aborted

Expand to view the summary

Build stats

Duration: 28 min 29 sec

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

elasticmachine · 2024-01-18T14:41:04Z

❕ Build Aborted

Expand to view the summary

Build stats

Duration: 17 min 9 sec

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

elasticmachine · 2024-01-18T15:00:27Z

❕ Build Aborted

Expand to view the summary

Build stats

Duration: 8 min 58 sec

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

elasticmachine · 2024-01-18T15:00:28Z

❕ Build Aborted

Expand to view the summary

Build stats

Duration: 25 min 30 sec

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

elasticmachine · 2024-01-18T15:18:23Z

❕ Build Aborted

Expand to view the summary

Build stats

Duration: 25 min 43 sec

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

elasticmachine · 2024-01-18T15:39:32Z

💔 Build Failed

Expand to view the summary

Build stats

Duration: 7 min 30 sec

Pipeline error

❕ Flaky test report

No test was executed to be analysed.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

elasticmachine · 2024-01-18T15:40:14Z

❕ Build Aborted

Expand to view the summary

Build stats

Duration: 29 min 18 sec

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

elasticmachine · 2024-01-18T17:50:02Z

💚 Build Succeeded

Expand to view the summary

Build stats

Duration: 134 min 49 sec

❕ Flaky test report

No test was executed to be analysed.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

elasticmachine · 2024-01-19T13:58:32Z

❕ Build Aborted

Expand to view the summary

Build stats

Start Time: 2024-01-19T13:43:15.169+0000
Duration: 15 min 11 sec

Test stats 🧪

Test	Results
Failed	1
Passed	3
Skipped	0
Total	4

Test errors

Expand to view the tests failures

`Build&Test / filebeat-unitTest / [empty] – TEST-go-unit.xml`

no error details

Expand to view the stacktrace

 Test report file /var/lib/jenkins/workspace/PR-37668-11-bc7fccea-97e9-4aa0-8c06-19b6a65aa220/src/github.com/elastic/beats/build/filebeat/build/TEST-go-unit.xml was length 0

Steps errors

Expand to view the steps failures

`Error signal`

Took 0 min 0 sec . View more details here
Description: Error 'org.jenkinsci.plugins.workflow.steps.FlowInterruptedException'

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

elasticmachine · 2024-02-06T09:11:51Z

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

dliappis

LGTM

.buildkite/auditbeat/scripts/unit-tests-win.ps1

.buildkite/auditbeat/auditbeat-pipeline.yml

elasticmachine · 2024-02-06T11:39:55Z

💚 Build Succeeded

Buildkite Build
Commit: d86f90a

History

💚 Build #2499 succeeded 72d0ba6
💔 Build #2476 failed 13683ba
💚 Build #2459 succeeded 39e2807
💔 Build #2455 failed 349fc7b
💔 Build #2440 failed f193453
💔 Build #2435 failed b7a311f

cc @oakrizan

* wip: getting GH message * added test scripts * added windows tests * added packaging step * updated packaging execution conditions * win-test failure: updated artifact path * pr fixes * changed group_test.go * moved env setup to separate script * added dynamic step for packaging * added tests and scrosscompile scripts for auditbeat * fixes for crosscompile and macos steps * added env vars for linux packaging * added env vars for linux packaging * enabled packaging step * added docker login * debugging arm packaing * enabled synamic packaging step * updated packaging group steps * fixed conflicts * updated win test script (cherry picked from commit 31819ec) # Conflicts: # .buildkite/env-scripts/env.sh # .buildkite/env-scripts/linux-env.sh # .buildkite/env-scripts/util.sh # .buildkite/env-scripts/win-env.sh # .buildkite/filebeat/filebeat-pipeline.yml # .buildkite/filebeat/scripts/integration-gotests.sh # .buildkite/filebeat/scripts/integration-pytests.sh # .buildkite/filebeat/scripts/package-step.sh # .buildkite/filebeat/scripts/package.sh # .buildkite/filebeat/scripts/unit-tests.sh

* wip: getting GH message * added test scripts * added windows tests * added packaging step * updated packaging execution conditions * win-test failure: updated artifact path * pr fixes * changed group_test.go * moved env setup to separate script * added dynamic step for packaging * added tests and scrosscompile scripts for auditbeat * fixes for crosscompile and macos steps * added env vars for linux packaging * added env vars for linux packaging * enabled packaging step * added docker login * debugging arm packaing * enabled synamic packaging step * updated packaging group steps * fixed conflicts * updated win test script (cherry picked from commit 31819ec)

* wip: getting GH message * added test scripts * added windows tests * added packaging step * updated packaging execution conditions * win-test failure: updated artifact path * pr fixes * changed group_test.go * moved env setup to separate script * added dynamic step for packaging * added tests and scrosscompile scripts for auditbeat * fixes for crosscompile and macos steps * added env vars for linux packaging * added env vars for linux packaging * enabled packaging step * added docker login * debugging arm packaing * enabled synamic packaging step * updated packaging group steps * fixed conflicts * updated win test script (cherry picked from commit 31819ec) # Conflicts: # .buildkite/env-scripts/env.sh # .buildkite/env-scripts/linux-env.sh # .buildkite/env-scripts/util.sh # .buildkite/env-scripts/win-env.sh # .buildkite/filebeat/filebeat-pipeline.yml # .buildkite/filebeat/scripts/integration-gotests.sh # .buildkite/filebeat/scripts/integration-pytests.sh # .buildkite/filebeat/scripts/package-step.sh # .buildkite/filebeat/scripts/package.sh # .buildkite/filebeat/scripts/unit-tests.sh

* wip: getting GH message * added test scripts * added windows tests * added packaging step * updated packaging execution conditions * win-test failure: updated artifact path * pr fixes * changed group_test.go * moved env setup to separate script * added dynamic step for packaging * added tests and scrosscompile scripts for auditbeat * fixes for crosscompile and macos steps * added env vars for linux packaging * added env vars for linux packaging * enabled packaging step * added docker login * debugging arm packaing * enabled synamic packaging step * updated packaging group steps * fixed conflicts * updated win test script (cherry picked from commit 31819ec)

* wip: getting GH message * added test scripts * added windows tests * added packaging step * updated packaging execution conditions * win-test failure: updated artifact path * pr fixes * changed group_test.go * moved env setup to separate script * added dynamic step for packaging * added tests and scrosscompile scripts for auditbeat * fixes for crosscompile and macos steps * added env vars for linux packaging * added env vars for linux packaging * enabled packaging step * added docker login * debugging arm packaing * enabled synamic packaging step * updated packaging group steps * fixed conflicts * updated win test script (cherry picked from commit 31819ec) Co-authored-by: Olga Naydyonock <olga.naidjonoka@elastic.co>

oakrizan added enhancement Team:Elastic-Agent Label for the Agent team backport-7.17 Automated backport to the 7.17 branch with mergify backport-v8.11.0 Automated backport with mergify labels Jan 18, 2024

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Jan 18, 2024

mergify bot assigned oakrizan Jan 18, 2024

oakrizan force-pushed the auditbeat-pipeline-migration branch 2 times, most recently from f95ab5e to 02cc1f7 Compare January 18, 2024 14:17

oakrizan force-pushed the auditbeat-pipeline-migration branch from 02cc1f7 to a1146c9 Compare January 18, 2024 14:23

oakrizan force-pushed the auditbeat-pipeline-migration branch from a1146c9 to 5bf1b41 Compare January 18, 2024 14:35

oakrizan force-pushed the auditbeat-pipeline-migration branch 2 times, most recently from 2d39654 to c19ec72 Compare January 18, 2024 14:52

oakrizan force-pushed the auditbeat-pipeline-migration branch from c19ec72 to d48e754 Compare January 18, 2024 15:10

oakrizan force-pushed the auditbeat-pipeline-migration branch 2 times, most recently from 974eff5 to 10682b2 Compare January 18, 2024 15:35

oakrizan force-pushed the auditbeat-pipeline-migration branch 2 times, most recently from 702a8a6 to c018d6d Compare January 19, 2024 13:52

oakrizan force-pushed the auditbeat-pipeline-migration branch from c018d6d to 2789cf7 Compare January 19, 2024 14:06

oakrizan added 8 commits February 6, 2024 11:07

added env vars for linux packaging

c0c53ae

added env vars for linux packaging

5af6f7e

enabled packaging step

a1f3278

added docker login

8c4cf2c

debugging arm packaing

ec3f080

enabled synamic packaging step

0a78193

updated packaging group steps

dbff6a3

fixed conflicts

72d0ba6

oakrizan force-pushed the auditbeat-pipeline-migration branch from 5249fb1 to 72d0ba6 Compare February 6, 2024 09:09

oakrizan marked this pull request as ready for review February 6, 2024 09:11

dliappis approved these changes Feb 6, 2024

View reviewed changes

.buildkite/auditbeat/scripts/unit-tests-win.ps1 Outdated Show resolved Hide resolved

updated win test script

d86f90a

oakrizan requested review from v1v, rdner and jlind23 February 6, 2024 10:33

v1v reviewed Feb 6, 2024

View reviewed changes

.buildkite/auditbeat/auditbeat-pipeline.yml Show resolved Hide resolved

.buildkite/auditbeat/auditbeat-pipeline.yml Show resolved Hide resolved

.buildkite/auditbeat/auditbeat-pipeline.yml Show resolved Hide resolved

.buildkite/auditbeat/auditbeat-pipeline.yml Show resolved Hide resolved

v1v approved these changes Feb 6, 2024

View reviewed changes

jlind23 approved these changes Feb 6, 2024

View reviewed changes

oakrizan merged commit 31819ec into elastic:main Feb 6, 2024
28 of 29 checks passed

This was referenced Feb 6, 2024

[7.17](backport #37668) Auditbeat pipeline migration to Buildkite #37878

Merged

[8.12](backport #37668) Auditbeat pipeline migration to Buildkite #37879

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Auditbeat pipeline migration to Buildkite #37668

Auditbeat pipeline migration to Buildkite #37668

oakrizan commented Jan 18, 2024 •

edited

Loading

elasticmachine commented Jan 18, 2024

Build stats

Test stats 🧪

`Error signal`

elasticmachine commented Jan 18, 2024

Build stats

elasticmachine commented Jan 18, 2024

Build stats

elasticmachine commented Jan 18, 2024

Build stats

elasticmachine commented Jan 18, 2024

Build stats

elasticmachine commented Jan 18, 2024

Build stats

elasticmachine commented Jan 18, 2024

Build stats

elasticmachine commented Jan 18, 2024

Build stats

elasticmachine commented Jan 18, 2024

Build stats

elasticmachine commented Jan 18, 2024

Build stats

elasticmachine commented Jan 19, 2024

Build stats

Test stats 🧪

`Build&Test / filebeat-unitTest / [empty] – TEST-go-unit.xml`

`Error signal`

elasticmachine commented Feb 6, 2024

dliappis left a comment

elasticmachine commented Feb 6, 2024

Auditbeat pipeline migration to Buildkite #37668

Auditbeat pipeline migration to Buildkite #37668

Conversation

oakrizan commented Jan 18, 2024 • edited Loading

Proposed commit message

Checklist

Author's Checklist

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

elasticmachine commented Jan 18, 2024

❕ Build Aborted

Build stats

Test stats 🧪

Steps errors

Error signal

🤖 GitHub comments

elasticmachine commented Jan 18, 2024

❕ Build Aborted

Build stats

🤖 GitHub comments

elasticmachine commented Jan 18, 2024

❕ Build Aborted

Build stats

🤖 GitHub comments

elasticmachine commented Jan 18, 2024

❕ Build Aborted

Build stats

🤖 GitHub comments

elasticmachine commented Jan 18, 2024

❕ Build Aborted

Build stats

🤖 GitHub comments

elasticmachine commented Jan 18, 2024

❕ Build Aborted

Build stats

🤖 GitHub comments

elasticmachine commented Jan 18, 2024

❕ Build Aborted

Build stats

🤖 GitHub comments

elasticmachine commented Jan 18, 2024

💔 Build Failed

Build stats

Pipeline error

❕ Flaky test report

🤖 GitHub comments

elasticmachine commented Jan 18, 2024

❕ Build Aborted

Build stats

🤖 GitHub comments

elasticmachine commented Jan 18, 2024

💚 Build Succeeded

Build stats

❕ Flaky test report

🤖 GitHub comments

elasticmachine commented Jan 19, 2024

❕ Build Aborted

Build stats

Test stats 🧪

Test errors

Build&Test / filebeat-unitTest / [empty] – TEST-go-unit.xml

Steps errors

Error signal

🤖 GitHub comments

elasticmachine commented Feb 6, 2024

dliappis left a comment

Choose a reason for hiding this comment

elasticmachine commented Feb 6, 2024

💚 Build Succeeded

History

oakrizan commented Jan 18, 2024 •

edited

Loading

`Error signal`

`Build&Test / filebeat-unitTest / [empty] – TEST-go-unit.xml`

`Error signal`