[BUG] Dashboards-Observability flaky tests

[ps48]

What is the bug?

Integration test for dashboards observability plugin is failing occasionally and should be made more robust. Currently below failing tests are marked as excluded in 2.10 from PR: #896

How can one reproduce the bug?

• Panels - https://ci.opensearch.org/ci/dbc/integ-test-opensearch-dashboards/2.10.0/6610/linux/arm64/tar/test-results/4173/integ-test/observabilityDashboards/with-security/stdout.txt
• Notebooks - https://ci.opensearch.org/ci/dbc/integ-test-opensearch-dashboards/2.10.0/6610/linux/x64/tar/test-results/4176/integ-test/observabilityDashboards/with-security/stdout.txt

What is the expected behavior?

The above two tests must pass without the exclusion.

What is your host/environment?

• OSD 2.10

Do you have any screenshots?

If applicable, add screenshots to help explain your problem.

Do you have any additional context?

Add any other context about the problem.

[wanglam]

Hi @ps48, do you have any plan to fix this issue in the coming release?

[wanglam]

Hi @ps48 , just find there are two PRs to related to this issue. The #1001 has been merged, and #1006 has been approved. Shall we close this issue after all the PRs merged?

[ps48]

Yes let's merge the PR with backports and then we should be good to close here. It would be better to verify the CI for observability is green for panels and notebooks after all PRs are merged in.

[wanglam]

Hi @ps48 , It seems the observability tests workflow still failed in the 2.x branch. Could you help me take a look?

[ps48]

@wanglam we are good to close the issue as observability tests are green and passing. We just need some reviews for backports to main: PR1 & PR2

[wanglam]

These backport PRs has been merged. Close this issue.