Can't use more than one TestParameter annotation with Android Instrumentation tests.

[ultraon]

When I use only one property with TestParameter annotation tests works well.
As soon as I add a 2nd property with TestParameter annotation then all tests fail.

After some investigation I've found that androidx.test.runner.AndroidJUnitRunner.runnerArgs.tests contains incorrect list of test classes:

• my.test.MyTestClass#testCheckSmth[boolParam1=false
• boolParam2=false]

And as soon as TestLoader.doCreateRunner() tries to create test class instance from string boolParam2=false] using reflection it fails later.

It is possible that input parameters incorrectly parsed from test args bundle splitting my.test.MyTestClass#testCheckSmth[boolParam1=false,boolParam2=false] by coma.

Test class sample:

// This is instrumentation test, should run on device or emulator
@RunWith(TestParameterInjector::class)
class MyTestClass {

	@TestParameter
	var boolParam1 = false

	@TestParameter
	var boolParam2 = false

	@Test
	fun testCheckSmth() {
	  // test body...
	}
}

When I use the same sample with standard JUnit tests it works well.

Test output:

Starting 4 tests on Android_TV_720p_API_Q(AVD) - Q

boolParam2=false] > [Android_TV_720p_API_Q(AVD) - Q] FAILED 
        java.lang.ClassNotFoundException: Invalid name: boolParam2=false]
        at java.lang.Class.classForName(Native Method)

boolParam2=true] > [Android_TV_720p_API_Q(AVD) - Q] FAILED 
        java.lang.ClassNotFoundException: Invalid name: boolParam2=true]
        at java.lang.Class.classForName(Native Method)

boolParam2=false] > [Android_TV_720p_API_Q(AVD) - Q] FAILED 
        java.lang.ClassNotFoundException: Invalid name: boolParam2=false]
        at java.lang.Class.classForName(Native Method)

boolParam2=true] > [Android_TV_720p_API_Q(AVD) - Q] FAILED 
        java.lang.ClassNotFoundException: Invalid name: boolParam2=true]
        at java.lang.Class.classForName(Native Method)

> Task :app:connectedDebugAndroidTest FAILED

[ultraon]

Just in case, I tried with https://github.com/square/burst - have the same issue.

[nymanjens]

Looks like this library doesn't work well with AndroidJUnitRunner.

Can you give an example of how I can reproduce this locally? How do I combine AndroidJUnitRunner and TestParameterInjector?

[TWiStErRob]

Spaces are not supported: android/android-test#956 might be the same issue with commas as that's the separator for listing explicit executions.

[nymanjens]

Right, so this could be fixed by filtering spaces and commas from the test name for Android tests.

I guess TestParameterInjector could look at the classpath and see if AndroidJUnitRunner is loaded in there.

[TWiStErRob]

Careful, Robolectric might also have it on classpath.
See https://robolectric.org/blog/2021/10/06/sharedTest/

[nymanjens]

Hmm, so replacing spaces/commas by underscores is really easy, but I wouldn't want to do this for more use cases than necessary. As a non-expert on Android, can anyone recommend a good way of distinguishing this AndroidJUnitRunner-like use case from others that are working fine (if they exist)?

[TWiStErRob]

I would recommend checking Espresso source code to see how they detect if the OS is Android. I'm sure okhttp/okio also has one too.

[alexvanyo]

Just tried this today with a fairly up-to-date Android project and tools, and this seems to work fine now? At least, it was running with my usage with more than one TestParameter.