Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fixing flaky tests #199562

Merged
merged 5 commits into from
Nov 13, 2024
Merged

Fixing flaky tests #199562

merged 5 commits into from
Nov 13, 2024

Conversation

ymao1
Copy link
Contributor

@ymao1 ymao1 commented Nov 8, 2024 •
edited by kibanamachine
Loading

Resolves #196226
Resolves #197352
Resolves #198631
Resolves #199722
Resolves #199351
Resolves #199580

Summary

  • Fixes flaky find test that relied on relative dates to use a fixed offset in the find query params. This closes multiple issues because it is a security and spaces test that uses the same test for different user/space scenarios
  • Fixes flaky schedule test that was checking AAD on an ad_hoc_run_param object after backfill was complete and the SO was completed. Added artificial delay to rule execution so the backfill doesn't run as quickly.

@kibanamachine
Copy link
Contributor

Flaky Test Runner Stats

🟠 Some tests failed. - kibana-flaky-test-suite-runner#7364

[❌] x-pack/test/alerting_api_integration/security_and_spaces/group1/config.ts: 174/200 tests passed.

see run history

@ymao1
Copy link
Contributor Author

ymao1 commented Nov 11, 2024

@elasticmachine merge upstream

@kibanamachine
Copy link
Contributor

Flaky Test Runner Stats

🎉 All tests passed! - kibana-flaky-test-suite-runner#7379

[✅] x-pack/test/alerting_api_integration/security_and_spaces/group1/config.ts: 200/200 tests passed.

see run history

@ymao1 ymao1 marked this pull request as ready for review November 12, 2024 17:13
@ymao1 ymao1 requested a review from a team as a code owner November 12, 2024 17:13
@ymao1 ymao1 self-assigned this Nov 12, 2024
@ymao1 ymao1 added Feature:Alerting release_note:skip Skip the PR/issue when compiling release notes Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams) v9.0.0 backport:prev-minor Backport to (8.x) the previous minor version (i.e. one version back from main) v8.17.0 labels Nov 12, 2024
@elasticmachine
Copy link
Contributor

Pinging @elastic/response-ops (Team:ResponseOps)

pmuellr
pmuellr approved these changes Nov 12, 2024
View reviewed changes
Copy link
Member

@pmuellr pmuellr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NIIIIICE! LGTM

@ymao1
Copy link
Contributor Author

ymao1 commented Nov 12, 2024

@elasticmachine merge upstream

@ymao1 ymao1 enabled auto-merge (squash) November 13, 2024 00:00
@ymao1 ymao1 merged commit 51ac51f into elastic:main Nov 13, 2024
22 checks passed
@kibanamachine
Copy link
Contributor

Starting backport for target branches: 8.x

https://github.com/elastic/kibana/actions/runs/11809029574

@elasticmachine
Copy link
Contributor

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

  • [job] [logs] FTR Configs #21 / Observability AI Assistant Functional tests feature_controls/settings_security.spec.ts ai assistant management privileges all privileges allows updating of an advanced setting

Metrics [docs]

✅ unchanged

History

cc @ymao1

kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Nov 13, 2024
@ymao1
Fixing flaky tests (elastic#199562)
1a548d0 
Resolves elastic#196226
Resolves elastic#197352
Resolves elastic#198631
Resolves elastic#199722
Resolves elastic#199351
Resolves elastic#199580

## Summary

* Fixes flaky `find` test that relied on relative dates to use a fixed
offset in the find query params. This closes multiple issues because it
is a security and spaces test that uses the same test for different
user/space scenarios
* Fixes flaky `schedule` test that was checking AAD on an
`ad_hoc_run_param` object after backfill was complete and the SO was
completed. Added artificial delay to rule execution so the backfill
doesn't run as quickly.

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
(cherry picked from commit 51ac51f)
@kibanamachine kibanamachine mentioned this pull request Nov 13, 2024
Merged
@kibanamachine
Copy link
Contributor

💚 All backports created successfully

Status Branch Result
✅ 8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

kibanamachine added a commit that referenced this pull request Nov 13, 2024
@kibanamachine @ymao1
[8.x] Fixing flaky tests (#199562) (#199897)
73376fc 
# Backport

This will backport the following commits from `main` to `8.x`:
- [Fixing flaky tests
(#199562)](#199562)

<!--- Backport version: 9.4.3 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Ying
Mao","email":"ying.mao@elastic.co"},"sourceCommit":{"committedDate":"2024-11-13T01:44:24Z","message":"Fixing
flaky tests (#199562)\n\nResolves
https://github.com/elastic/kibana/issues/196226\r\nResolves
https://github.com/elastic/kibana/issues/197352\r\nResolves
https://github.com/elastic/kibana/issues/198631\r\nResolves
https://github.com/elastic/kibana/issues/199722\r\nResolves
https://github.com/elastic/kibana/issues/199351\r\nResolves
https://github.com/elastic/kibana/issues/199580\r\n\r\n##
Summary\r\n\r\n* Fixes flaky `find` test that relied on relative dates
to use a fixed\r\noffset in the find query params. This closes multiple
issues because it\r\nis a security and spaces test that uses the same
test for different\r\nuser/space scenarios\r\n* Fixes flaky `schedule`
test that was checking AAD on an\r\n`ad_hoc_run_param` object after
backfill was complete and the SO was\r\ncompleted. Added artificial
delay to rule execution so the backfill\r\ndoesn't run as
quickly.\r\n\r\nCo-authored-by: Elastic Machine
<elasticmachine@users.noreply.github.com>","sha":"51ac51fa2b0b982421fbbe70a02f1eeb63154f4a","branchLabelMapping":{"^v9.0.0$":"main","^v8.17.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["Feature:Alerting","release_note:skip","Team:ResponseOps","v9.0.0","backport:prev-minor","v8.17.0"],"title":"Fixing
flaky
tests","number":199562,"url":"https://github.com/elastic/kibana/pull/199562","mergeCommit":{"message":"Fixing
flaky tests (#199562)\n\nResolves
https://github.com/elastic/kibana/issues/196226\r\nResolves
https://github.com/elastic/kibana/issues/197352\r\nResolves
https://github.com/elastic/kibana/issues/198631\r\nResolves
https://github.com/elastic/kibana/issues/199722\r\nResolves
https://github.com/elastic/kibana/issues/199351\r\nResolves
https://github.com/elastic/kibana/issues/199580\r\n\r\n##
Summary\r\n\r\n* Fixes flaky `find` test that relied on relative dates
to use a fixed\r\noffset in the find query params. This closes multiple
issues because it\r\nis a security and spaces test that uses the same
test for different\r\nuser/space scenarios\r\n* Fixes flaky `schedule`
test that was checking AAD on an\r\n`ad_hoc_run_param` object after
backfill was complete and the SO was\r\ncompleted. Added artificial
delay to rule execution so the backfill\r\ndoesn't run as
quickly.\r\n\r\nCo-authored-by: Elastic Machine
<elasticmachine@users.noreply.github.com>","sha":"51ac51fa2b0b982421fbbe70a02f1eeb63154f4a"}},"sourceBranch":"main","suggestedTargetBranches":["8.x"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/199562","number":199562,"mergeCommit":{"message":"Fixing
flaky tests (#199562)\n\nResolves
https://github.com/elastic/kibana/issues/196226\r\nResolves
https://github.com/elastic/kibana/issues/197352\r\nResolves
https://github.com/elastic/kibana/issues/198631\r\nResolves
https://github.com/elastic/kibana/issues/199722\r\nResolves
https://github.com/elastic/kibana/issues/199351\r\nResolves
https://github.com/elastic/kibana/issues/199580\r\n\r\n##
Summary\r\n\r\n* Fixes flaky `find` test that relied on relative dates
to use a fixed\r\noffset in the find query params. This closes multiple
issues because it\r\nis a security and spaces test that uses the same
test for different\r\nuser/space scenarios\r\n* Fixes flaky `schedule`
test that was checking AAD on an\r\n`ad_hoc_run_param` object after
backfill was complete and the SO was\r\ncompleted. Added artificial
delay to rule execution so the backfill\r\ndoesn't run as
quickly.\r\n\r\nCo-authored-by: Elastic Machine
<elasticmachine@users.noreply.github.com>","sha":"51ac51fa2b0b982421fbbe70a02f1eeb63154f4a"}},{"branch":"8.x","label":"v8.17.0","branchLabelMappingKey":"^v8.17.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->

Co-authored-by: Ying Mao <ying.mao@elastic.co>
@ymao1 ymao1 deleted the flaky-test-196226 branch November 13, 2024 12:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pmuellr pmuellr pmuellr approved these changes

@adcoelho adcoelho Awaiting requested review from adcoelho

Assignees

@ymao1 ymao1

Labels
backport:prev-minor Backport to (8.x) the previous minor version (i.e. one version back from main) Feature:Alerting release_note:skip Skip the PR/issue when compiling release notes Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams) v8.17.0 v9.0.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Failing test: X-Pack Alerting API Integration Tests.x-pack/test/alerting_api_integration/security_and_spaces/group1/tests/alerting/backfill/schedule·ts - alerting api integration security and spaces enabled Alerts - Group 1 alerts backfill rule runs schedule backfill superuser at space1 should handle scheduling multiple backfill job requests for a single rule appropriately Failing test: X-Pack Alerting API Integration Tests.x-pack/test/alerting_api_integration/security_and_spaces/group1/tests/alerting/backfill/schedule·ts - alerting api integration security and spaces enabled Alerts - Group 1 alerts backfill rule runs schedule backfill space_1_all_with_restricted_fixture at space1 should handle schedule request where some requests succeed and some requests fail appropriately Failing test: X-Pack Alerting API Integration Tests.x-pack/test/alerting_api_integration/security_and_spaces/group1/tests/alerting/backfill/find·ts - alerting api integration security and spaces enabled Alerts - Group 1 alerts backfill rule runs find backfill space_1_all_with_restricted_fixture at space1 should handle finding backfill requests with query string appropriately Failing test: X-Pack Alerting API Integration Tests.x-pack/test/alerting_api_integration/security_and_spaces/group1/tests/alerting/backfill/find·ts - alerting api integration security and spaces enabled Alerts - Group 1 alerts backfill rule runs find backfill global_read at space1 should handle finding backfill requests with query string appropriately Failing test: X-Pack Alerting API Integration Tests.x-pack/test/alerting_api_integration/security_and_spaces/group1/tests/alerting/backfill/find·ts - alerting api integration security and spaces enabled Alerts - Group 1 alerts backfill rule runs find backfill superuser at space1 should handle finding backfill requests with query string appropriately
Failing test: X-Pack Alerting API Integration Tests.x-pack/test/alerting_api_integration/security_and_spaces/group1/tests/alerting/backfill/find·ts - alerting api integration security and spaces enabled Alerts - Group 1 alerts backfill rule runs find backfill space_1_all_alerts_none_actions at space1 should handle finding backfill requests with query string appropriately
4 participants
@ymao1 @kibanamachine @elasticmachine @pmuellr