Segment Replication support: How to handle segment replication with the Security Index

[stephen-crawford]

This issue is intended to be a discussion forum for figuring out how we want to handle segment replication. As part of the changes introduced with segment replication features (opensearch-project/OpenSearch#8182, opensearch-project/OpenSearch#8158) replication of the security index has been brought up.

A description of the situation can be found here: opensearch-project/OpenSearch#8182 (comment) and is provided by one of the Segment Replication contributors. Copying from their description, the major point of concern is:

However there are perf implications here that need to be called out. SegRep will on average increase the amount of time that replica shards are inconsistent from their primaries due to the time required to copy out segments (time dependent on cluster config/hardware etc). System indices are generally small and should copy out quickly, so I wouldn't expect this to be of major concern. Are there are cases where stronger read/write consistency is desired on the system index? If this is the case plugins would need to opensearch-project/OpenSearch#7375 for searches.

This means that there are potentially issues with replicated Security Indices are not consistent with the original and could lead to hypothetical areas of concern.

We should use this issue to determine the extent to which this would impact the security posture and be able to provide guidance to the Seg. Rep. developers on what we will need to uphold the highest security standards possible.

[cwperks]

@scrawfor99 I think the security codebase needs to update the ConfigurationLoader to set a preference to use the primary shard when reloading the security index in the cache on every node.

In this section of code the security plugin can also set preference to _primary to instruct the mget request to run the query against the primary shard which will have the freshest data.

[davidlago]

not taking any action at this point