[Star tree] Star tree file formats refactoring and fixing offset bug

[bharath-techie]

Description

This PR removes the exposure of DocIdSetIterator in the file formats flow.

So, in order to prevent exposing DISI directly as part of star tree file format readers , this change contains a simple wrapper iterator that abstracts the DISI , and exposes relevant methods tied to star tree to retrieve the values.

Apart from this also fixing a bug where file size greater than Integer.MaxValue is not handled correctly.

Related Issues

#15974 - issue for the bug found with offsets

Check List

• Functionality includes testing.
• API changes companion pull request created, if applicable.
• Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

❌ Gradle check result for f6d3ca9: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 67e8e7e: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 14a7575: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for b4f38ea: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❕ Gradle check result for b4f38ea: UNSTABLE

• TEST FAILURES:

      1 org.opensearch.remotestore.RemoteStoreStatsIT.testDownloadStatsCorrectnessSinglePrimaryMultipleReplicaShards

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov]

Codecov Report

Attention: Patch coverage is 77.04918% with 14 lines in your changes missing coverage. Please review.

Project coverage is 71.92%. Comparing base (3937ccb) to head (b4f38ea).
Report is 110 commits behind head on main.

Files with missing lines	Patch %	Lines
...be/startree/utils/SequentialDocValuesIterator.java	68.75%	2 Missing and 3 partials ⚠️
...cube/startree/builder/StarTreeDocsFileManager.java	50.00%	1 Missing and 2 partials ⚠️
...eindex/datacube/startree/index/StarTreeValues.java	76.92%	1 Missing and 2 partials ⚠️
...tartree/utils/iterator/StarTreeValuesIterator.java	57.14%	3 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #15975      +/-   ##
============================================
+ Coverage     71.90%   71.92%   +0.02%     
+ Complexity    64392    64351      -41     
============================================
  Files          5278     5280       +2     
  Lines        300877   300870       -7     
  Branches      43478    43476       -2     
============================================
+ Hits         216351   216412      +61     
+ Misses        66747    66655      -92     
- Partials      17779    17803      +24     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[opensearch-trigger-bot]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-15975-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 80ff07eb0c5485832e6fcbc46a9d100435ed3051
# Push it to GitHub
git push --set-upstream origin backport/backport-15975-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-15975-to-2.x.

[opensearch-trigger-bot]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-15975-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 80ff07eb0c5485832e6fcbc46a9d100435ed3051
# Push it to GitHub
git push --set-upstream origin backport/backport-15975-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-15975-to-2.x.