[BUG] IndexStateManagementSecurityBehaviorIT test fails when run twice against a remote cluster

[DarshitChanpura]

What is the bug?
While debugging the fix for #1100, I ran into:

REPRODUCE WITH: ./gradlew ':integTest' --tests "org.opensearch.indexmanagement.IndexStateManagementSecurityBehaviorIT.test add policy" -Dtests.seed=5DCE79AE35A726F7 -Dtests.security.manager=false -Dtests.locale=de-CH -Dtests.timezone=Africa/Djibouti -Druntime.java=17

org.opensearch.indexmanagement.IndexStateManagementSecurityBehaviorIT > test add policy FAILED
    java.lang.AssertionError: expected:<201> but was:<200>
        at __randomizedtesting.SeedInfo.seed([5DCE79AE35A726F7:6DEFFE55BAD2DB76]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.failNotEquals(Assert.java:835)
        at org.junit.Assert.assertEquals(Assert.java:120)
        at org.junit.Assert.assertEquals(Assert.java:146)
        at org.opensearch.indexmanagement.SecurityRestTestCase.executeRequest(SecurityRestTestCase.kt:381)
        at org.opensearch.indexmanagement.SecurityRestTestCase.createUser(SecurityRestTestCase.kt:398)
        at org.opensearch.indexmanagement.IndexStateManagementSecurityBehaviorIT.setupUsersAndRoles(IndexStateManagementSecurityBehaviorIT.kt:82)

Upon further investigation, it was discovered that IndexStateManagementSecurityBehaviorIT is failing because of https://github.com/opensearch-project/index-management/blob/main/src/test/kotlin/org/opensearch/indexmanagement/SecurityRestTestCase.kt#L399 check inside SecurityRestTestCase.kt. If internal user doesn’t exist in the system then 201 CREATED is returned which is expected by the test. But, if internal user already exists then 200 OK is returned which is unexpected.

This issue only occurs when tests are run more than once against an existing cluster. If the cluster is tore-down and re-started this issue will not occur. This is because the data is wiped with teardown.

How can one reproduce the bug?
Steps to reproduce the behavior:

1. Start a full distribution opensearch cluster
2. Run this twice: ./gradlew integTest -Dtests.rest.cluster=localhost:9200 -Dtests.cluster=localhost:9200 -Dtests.clustername="opensearch-cluster" -Dhttps=true -Duser=admin -Dpassword=<your-admin-password>
3. See error

What is the expected behavior?
Maybe this isn't a bug as the test are not expected to be run against remote cluster twice without tearing the cluster down. But if it is expected to be run more than once, then the test should not fail.

• OpenSearch 2.12 with security plugin installed.

[DarshitChanpura]

When running it twice with this command:

./gradlew integTest -Dtests.rest.cluster=localhost:9200 -Dtests.cluster=localhost:9200 -Dtests.clustername="opensearch-cluster" -Dhttps=true -Duser=admin -Dpassword=<your-admin-password> --tests "org.opensearch.indexmanagement.IndexStateManagementSecurityBehaviorIT.test add policy" 

I couldn't reproduce it again:

➜  index-management git:(2.12) ✗ ./gradlew integTest -Dtests.rest.cluster=localhost:9200 -Dtests.cluster=localhost:9200 -Dtests.clustername="opensearch-cluster" -Dhttps=true -Duser=admin -Dpassword=myStrongPassword123! --tests "org.opensearch.indexmanagement.IndexStateManagementSecurityBehaviorIT.test add policy"
=======================================
OpenSearch Build Hamster says Hello!
  Gradle Version        : 8.5
  OS Info               : Linux 5.10.205-172.807.amzn2int.x86_64 (amd64)
  JDK Version           : 17 (Amazon Corretto JRE)
  JAVA_HOME             : /usr/lib/jvm/java-17-amazon-corretto.x86_64
  Random Testing Seed   : F18D86075F680416
  In FIPS 140 mode      : false
=======================================

Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

For more on this, please refer to https://docs.gradle.org/8.5/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.

BUILD SUCCESSFUL in 34s
13 actionable tasks: 2 executed, 11 up-to-date
➜  index-management git:(2.12) ✗ ./gradlew integTest -Dtests.rest.cluster=localhost:9200 -Dtests.cluster=localhost:9200 -Dtests.clustername="opensearch-cluster" -Dhttps=true -Duser=admin -Dpassword=myStrongPassword123! --tests "org.opensearch.indexmanagement.IndexStateManagementSecurityBehaviorIT.test add policy" 
=======================================
OpenSearch Build Hamster says Hello!
  Gradle Version        : 8.5
  OS Info               : Linux 5.10.205-172.807.amzn2int.x86_64 (amd64)
  JDK Version           : 17 (Amazon Corretto JRE)
  JAVA_HOME             : /usr/lib/jvm/java-17-amazon-corretto.x86_64
  Random Testing Seed   : A18E873E4F7FCED1
  In FIPS 140 mode      : false
=======================================

Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

For more on this, please refer to https://docs.gradle.org/8.5/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.

BUILD SUCCESSFUL in 33s
13 actionable tasks: 2 executed, 11 up-to-date
➜  index-management git:(2.12) ✗ ./gradlew integTest -Dtests.rest.cluster=localhost:9200 -Dtests.cluster=localhost:9200 -Dtests.clustername="opensearch-cluster" -Dhttps=true -Duser=admin -Dpassword=myStrongPassword123! --tests "org.opensearch.indexmanagement.IndexStateManagementSecurityBehaviorIT.test add policy"
=======================================
OpenSearch Build Hamster says Hello!
  Gradle Version        : 8.5
  OS Info               : Linux 5.10.205-172.807.amzn2int.x86_64 (amd64)
  JDK Version           : 17 (Amazon Corretto JRE)
  JAVA_HOME             : /usr/lib/jvm/java-17-amazon-corretto.x86_64
  Random Testing Seed   : 355E72463C6067AB
  In FIPS 140 mode      : false
=======================================

Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

For more on this, please refer to https://docs.gradle.org/8.5/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.

BUILD SUCCESSFUL in 34s
13 actionable tasks: 2 executed, 11 up-to-date
➜  index-management git:(2.12) ✗ 

Maybe this is not a bug?

[dblock]

Catch All Triage - 1 2 3 4 5