[Obs AI Assistant] Add retry statements as an attempt to resolve flaky tests #200022
+56
−48
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
viduni94
added
release_note:enhancement
Team:Obs AI Assistant
backport:version
|
Pinging @elastic/obs-ai-assistant (Team:Obs AI Assistant) |
botelastic
bot
added
the
ci:project-deploy-observability
🤖 GitHub commentsExpand to view the GitHub comments
Just comment with:
|
viduni94
force-pushed
the
retry-flaky-test-for-knowledge-base-instructions
branch
from
7323649
to
60f4376
Compare
💚 Build Succeeded
Metrics [docs]
Historycc @viduni94 |
sorenlouv
reviewed
Nov 13, 2024
|
|
||
| const instructions = res.body.userInstructions; | ||
| const instructions = res.body.userInstructions; | ||
| expect(instructions).to.have.length(3); |
There was a problem hiding this comment.
Does expect actually throw and is able to be caught and retried? If so, good!
There was a problem hiding this comment.
Yes, based on the below example, where I made expect(instructions).to.have.length(2); - which should throw an error.
sorenlouv
approved these changes
Nov 14, 2024
|
Starting backport for target branches: 8.x |
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Nov 14, 2024
…y tests (elastic#200022) Closes elastic#192222 ## Summary ### Problem The test appears to be flaky, potentially because the entries are not available at the time of retrieval. This cannot be reproduced locally or via the flaky test runner. (more details [here](elastic#196026 (comment))) ### Solution Add a retry when fetching the instructions and check whether the number of instructions returned by the API endpoint is the same number of instructions expected. ### Checklist - [x] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed (cherry picked from commit 53c05a3)
💚 All backports created successfully
Note: Successful backport PRs will be merged automatically after passing CI. Questions ?Please refer to the Backport tool documentation |
kibanamachine
added a commit
that referenced
this pull request
Nov 14, 2024
…e flaky tests (#200022) (#200213) # Backport This will backport the following commits from `main` to `8.x`: - [[Obs AI Assistant] Add retry statements as an attempt to resolve flaky tests (#200022)](#200022) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Viduni Wickramarachchi","email":"viduni.wickramarachchi@elastic.co"},"sourceCommit":{"committedDate":"2024-11-14T16:09:42Z","message":"[Obs AI Assistant] Add retry statements as an attempt to resolve flaky tests (#200022)\n\nCloses https://github.com/elastic/kibana/issues/192222\r\n\r\n## Summary\r\n\r\n### Problem\r\nThe test appears to be flaky, potentially because the entries are not\r\navailable at the time of retrieval. This cannot be reproduced locally or\r\nvia the flaky test runner. (more details\r\n[here](https://github.com/elastic/kibana/pull/196026#issuecomment-2409056856))\r\n\r\n### Solution\r\nAdd a retry when fetching the instructions and check whether the number\r\nof instructions returned by the API endpoint is the same number of\r\ninstructions expected.\r\n\r\n### Checklist\r\n\r\n- [x] [Flaky Test\r\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\r\nused on any tests changed","sha":"53c05a33e7fe917b7d64b7d3ca664aaf3f9cf3ca","branchLabelMapping":{"^v9.0.0$":"main","^v8.17.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:enhancement","v9.0.0","Team:Obs AI Assistant","ci:project-deploy-observability","backport:version","v8.17.0"],"title":"[Obs AI Assistant] Add retry statements as an attempt to resolve flaky tests","number":200022,"url":"https://github.com/elastic/kibana/pull/200022","mergeCommit":{"message":"[Obs AI Assistant] Add retry statements as an attempt to resolve flaky tests (#200022)\n\nCloses https://github.com/elastic/kibana/issues/192222\r\n\r\n## Summary\r\n\r\n### Problem\r\nThe test appears to be flaky, potentially because the entries are not\r\navailable at the time of retrieval. This cannot be reproduced locally or\r\nvia the flaky test runner. (more details\r\n[here](https://github.com/elastic/kibana/pull/196026#issuecomment-2409056856))\r\n\r\n### Solution\r\nAdd a retry when fetching the instructions and check whether the number\r\nof instructions returned by the API endpoint is the same number of\r\ninstructions expected.\r\n\r\n### Checklist\r\n\r\n- [x] [Flaky Test\r\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\r\nused on any tests changed","sha":"53c05a33e7fe917b7d64b7d3ca664aaf3f9cf3ca"}},"sourceBranch":"main","suggestedTargetBranches":["8.x"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/200022","number":200022,"mergeCommit":{"message":"[Obs AI Assistant] Add retry statements as an attempt to resolve flaky tests (#200022)\n\nCloses https://github.com/elastic/kibana/issues/192222\r\n\r\n## Summary\r\n\r\n### Problem\r\nThe test appears to be flaky, potentially because the entries are not\r\navailable at the time of retrieval. This cannot be reproduced locally or\r\nvia the flaky test runner. (more details\r\n[here](https://github.com/elastic/kibana/pull/196026#issuecomment-2409056856))\r\n\r\n### Solution\r\nAdd a retry when fetching the instructions and check whether the number\r\nof instructions returned by the API endpoint is the same number of\r\ninstructions expected.\r\n\r\n### Checklist\r\n\r\n- [x] [Flaky Test\r\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\r\nused on any tests changed","sha":"53c05a33e7fe917b7d64b7d3ca664aaf3f9cf3ca"}},{"branch":"8.x","label":"v8.17.0","branchLabelMappingKey":"^v8.17.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> Co-authored-by: Viduni Wickramarachchi <viduni.wickramarachchi@elastic.co>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #192222
Summary
Problem
The test appears to be flaky, potentially because the entries are not available at the time of retrieval. This cannot be reproduced locally or via the flaky test runner. (more details here)
Solution
Add a retry when fetching the instructions and check whether the number of instructions returned by the API endpoint is the same number of instructions expected.
Checklist