[BUG] 2.10 Secure Integ Tests for Geospatial Plugin are failing due to migration of .opendistro-job-scheduler-lock index to System Index

[navneet1v]

Hi,
As part of 2.10 release of OpenSearch, the Security enabled Integ Tests for Geospatial Plugin are failing due to migration of .opendistro-job-scheduler-lock index to System Index.

Change which have caused the issue PR: #478

Error log:

org.opensearch.client.ResponseException: method [PUT], host [https://localhost:9200/], URI [_plugins/geospatial/ip2geo/datasource/updatedatasourceitnz], status line [HTTP/1.1 403 Forbidden]
    Warnings: [this request accesses system indices: [.opendistro-job-scheduler-lock], but in a future major version, direct access to system indices will be prevented by default]
    {"error":{"root_cause":[{"type":"security_exception","reason":"no permissions for [] and User [name=admin, backend_roles=[admin], requestedTenant=null]"}],"type":"security_exception","reason":"no permissions for [] and User [name=admin, backend_roles=[admin], requestedTenant=null]"},"status":403}
        at __randomizedtesting.SeedInfo.seed([2B29E33EB32391D4:5305C83A89F8B2E]:0)
        at app//org.opensearch.client.RestClient.convertResponse(RestClient.java:376)
        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:346)
        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:321)
        at app//org.opensearch.geospatial.GeospatialRestTestCase.createDatasource(GeospatialRestTestCase.java:122)
        at app//org.opensearch.geospatial.ip2geo.action.UpdateDatasourceIT.testUpdateDatasource_whenIncompatibleFields_thenFails(UpdateDatasourceIT.java:77)

{"error":{"root_cause":[{"type":"security_exception","reason":"no permissions for [] and User [name=admin, backend_roles=[admin], requestedTenant=null]"}],"type":"security_exception","reason":"no permissions for [] and User [name=admin, backend_roles=[admin], requestedTenant=null]"},"status":403}
        at __randomizedtesting.SeedInfo.seed([2B29E33EB32391D4:53E53D6A9674E6]:0)
        at app//org.opensearch.client.RestClient.convertResponse(RestClient.java:376)
        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:346)
        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:321)

The Geospatial plugin is not accessing this index of its own and reusing the Job Scheduler interfaces. Can you please look into this?

[sbcd90]

ism also face same issue cc: @bowenlan-amzn

[navneet1v]

cc: @CEHENKLE , @bbarani

[cwperks]

@navneet1v @sbcd90 Are you receiving this while trying to meddle (most likely delete) with the index in some way?

This index was added to the security plugin's list of system indices to give it system index protection which means even admin cannot delete the index. To delete the index you would need to connect with the admin certificate.

It was added in opensearch-project/security#3237

[navneet1v]

@navneet1v @sbcd90 Are you receiving this while trying to meddle (most likely delete) with the index in some way?

No, plugin is not deleting the index

@cwperks

[sbcd90]

we get this error while indexing & searching on this index.

[navneet1v]

we get this error while indexing & searching on this index.

+1

[cwperks]

@joshpalis ^

I know you were facing issues with stashing the thread context in JobScheduler. Could this be related? ThreadContext stashing may be required in JS

[stephen-crawford]

Security needs signoff from Job Scheduler maintainers that reverting is the preferred path forward. The alternative is to correct the behavior to unblock Navneet.

This is the PR to revert but we need a clear decision from job scheduler that is what you want to do: opensearch-project/security#3351 (review)

[joshpalis]

I know you were facing issues with stashing the thread context in JobScheduler. Could this be related? ThreadContext stashing may be required in JS

Since this issue is on the path of indexing/searching it may be necessary to stash the thread context in JobScheduler for every CRUD operation on the lock index. This would require modifications to the integration tests that I observed were consistently failing when the thread context was stashed. I concur with @scrawfor99, we'll need to revert this security PR to unblock the 2.10 release in a timely manner and handle stashing the thread context in the next release

CC : @cwperks @vibrantvarun

[joshpalis]

I have raised a PR to revert the system index change for 2.10 : #494

CC : @scrawfor99

[joshpalis]

@navneet1v 2.10 RC generation was successful. I've raise this issue to track the next steps to resolve the system index issues