Optimize integration test failure issue creation

[gaiksaya]

Is your feature request related to a problem? Please describe

The integration test failures at distribution level created GH issues in respective component repository.
https://github.com/issues?q=is%3Aopen+is%3Aissue+archived%3Afalse+user%3Aopensearch-project+label%3Aautocut+%22%5BAUTOCUT%5D+Integration+Test+%22+

However, creating a new issue for each different is creating more noise. Hence proposing to optimize the GH issue creation process.

Describe the solution you'd like

Create only issues for test failure just like build failure issues and add additional comments for other distributions/architecture/multiple runs.

On the high level this is what the issue will look like:

**Title**: Integration test failed for searchRelevanceDashboards: 3.0.0
**Label**: integtest-failure, v3.0.0
**Body**: 
The integration test failed at distribution level for component searchRelevanceDashboards
Version: 3.0.0
Distribution: tar
Architecture: x64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test-opensearch-dashboards/5109/display/redirect

* Test-report manifest:*
- https://ci.opensearch.org/ci/dbc/integ-test-opensearch-dashboards/2.12.0/7118/linux/x64/tar/test-results/5109/integ-test/test-report.yml

Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found [here](https://github.com/opensearch-project/opensearch-build/tree/main/src/report_workflow#guide-on-test-report-manifest-from-ci).

Describe alternatives you've considered

Keep what it is but do not create GH issues for all distributions.

Additional context

Further enhancements would be to play with labels for ease of understanding with what is the most recent distribution failures platform.

Acceptance Criterias

• Create only 1 issue per version
• Keep updating or commenting same issue on same platform or architecture

[gaiksaya]

Problem Statement:

We run integration test for all available components from the test manifest at the distribution level. This includes running tests for all distributions and architectures such as tarball, RPM, Debian, Windows and architectures such as x64 and arm64. We also create GH issues for all failing components with all details. Example: opensearch-project/dashboards-maps#577

However, the current approach followed is creating new issues for each version, distribution, architecture. This worked well for one distribution (tarball) and 2 architectures (x64, arm64). With increase in the supporting distributions this has lead to creation of many issues per distribution. Also there is a bug with current set up where architecture is not taken into consideration while closing the issue.
For example:

• zip: [AUTOCUT] Integration Test failed for custom-codecs: 2.12.0 zip distribution custom-codecs#103
• rpm: [AUTOCUT] Integration Test failed for custom-codecs: 2.12.0 rpm distribution custom-codecs#101
• debian: [AUTOCUT] Integration Test failed for custom-codecs: 2.12.0 deb distribution custom-codecs#100
• tar: [AUTOCUT] Integration Test failed for custom-codecs: 2.12.0 tar distribution custom-codecs#91

Approaches:

Approach 1: [Recommended]

We can resolve this by maintaining one issue per version similar to build failure issues. The build failure issue can handle various distribution/architecture failures in one workflow as all distributions are build in one go.
However, that is not true for integration tests. Each run represents a different distribution and architecture. https://build.ci.opensearch.org/view/Test/job/integ-test/

Hence, proposing to make use of with GH labels. Labels are unique and easy to manage. Creating or commenting on the GH issue when the tests fail is no brainer. However, closing the issue is a challenge as we might not know when to close the issue. It may pass for one distribution and fail for others.

Scenario 1: Windows integration test failed for customImportMapDashboards

1. Open Issue: If the issue with title [AUTOCUT] Integration Test failed for customImportMapDashboards: 3.0.0 does not exist, open a new issue

2. Comment on Issue: If the issue with title [AUTOCUT] Integration Test failed for customImportMapDashboards: 3.0.0 already exists, Check for label zip . In case it is missing add a label else proceed with adding a comment.

Scenario 2: Windows integration test passed for customImportMapDashboards

In case of integration test passing, the issue should only be close if:

1. Only the label of zip is present.
2. No labels are present

If there are additional labels such as tar, rpm present on the same issue with title AUTOCUT] Integration Test failed for customImportMapDashboards: 3.0.0 just remove the label

Pros:

1. One stop shop for all things related to integ-test failures at the distribution level
2. Easy to track and compare between distributions
3. Distribution labels would be efficient to know which all distribution is causing issue for a given component

Cons:

1. Too much noise with comments on the same issue
2. GitHub hides a bunch of comments for easy visibility that can lead to missing few comments. But latest should be good to track.

Approach 2:

Continue creating issues per distribution.

Pros:

1. Each issue related to a specific distribution can be tracked separately
2. No label parsing and manipulation required

Cons:

1. Creates too much noise with the number of issues
2. Might lead to component teams ignoring the issues related to specific distribution as cause of failure for all distribution might be same
3. Too many metrics to manage and debug