Compatibility with segment replication

[dreamer-89]

Coming from #407 (comment) as a follow up and with core now supporting real time reads for segment replication enabled indices (opensearch-project/OpenSearch#8536), we can revert #417 which enforces DOCUMENT replication strategy for lock indices.

CC @dbwiddis @owaiskazi19 @joshpalis

Request owners to label this issue with 2.10.0 release.

[bowenlan-amzn]

@joshpalis do we want to change this preference to against primary shard?

job-scheduler/src/main/java/org/opensearch/jobscheduler/sweeper/JobSweeper.java

Line 392 in 6abaa3c

.preference("_shards:" + shardId.id() + "|_only_local")

cc: @dreamer-89

[dreamer-89]

@joshpalis do we want to change this preference to against primary shard?

job-scheduler/src/main/java/org/opensearch/jobscheduler/sweeper/JobSweeper.java

Line 392 in 6abaa3c

.preference("_shards:" + shardId.id() + "|_only_local")

cc: @dreamer-89

@joshpalis @dbwiddis @owaiskazi19 : Can you please confirm @bowenlan-amzn question above ?

Caution: Please do verify that for strong reads (if any), your plugin relies only on get/mget APIs. I am asking as it is still possible to have strong reads guarantees via write paths by using IMMEDIATE/ WAIT_UNTIL refresh policy which ensures replica shard copies are refreshed with the indexing request, ensuring any follow up data retrieval request receives latest data. With SEGMENT both IMMEDIATE/WAIT_UNTIL does not guarantee replica shard refreshes, so for strong reads, any follow up data retrieval query (other than get/mget APIs), client would need to provide either _primary - hits primary shard only but provides strong consistency or _primary_first - hits primary first (as name suggest), if primary not available request is routed to replica copies resulting in better availability (though data when hitting replica could be stale).

[joshpalis]

I've raised a PR to set the search preference of the JobSweeper from _only_local to _primary