Add ability to disable native automation at the fixture level

[Makavelic]

What is your Scenario?

There are still several bugs that prevent me from using native automation in every test (iframe support is poor at the moment). I would like a way to disable native automation at the fixture level

What are you suggesting?

https://github.com/DevExpress/testcafe/pull/8021/files

If we can follow the same pattern as the above PR to disable native automation that would be perfect.

What alternatives have you considered?

I currently have my own custom testcafe test runner where I can parse the meta tag and dynamically turn off native automation. This works great when I use the runner but does not work if I just use the CLI (testcafe test runner in vscode). It would be great if I could simply do something like .fixture.disableNativeAutomation. Otherwise people on my team need to manually change the disable flag depending on what test file they're running.

Additional context

No response

[aleks-pro]

Hello @Makavelic ,

Thank you for the feedback.

I currently have my own custom testcafe test runner where I can parse the meta tag and dynamically turn off native automation

Please share your TestCafe test runner code. We will examine it. This will help us better understand your usage scenario.

The Native Automation mode support is our priority. So, it would be very helpful if you share a project sample that demonstrates iframe support issues you are facing with NA mode.

[Makavelic]

My test runner code is pretty hacky, but basically we have a powershell script that can figure out which test files match a suite meta tag and run each file sequentially and generate a single report for the group of files. This was done because testcafe will start running a different test file before the previous file is complete as the browsers free up when concurrency is turned on. This created data conflicts for us because a new test file will run before the cleanup in .after / .afterEach occurs. It's possible testcafe doesn't behave this way anymore but since it works for us, we still use our runner this way. As part of the powershell script, we also scan whether the meta tag includes disableNativeAutomation. If it does, our runner will set disableNativeAutomation to true. We only use the testcafe test runner as part of CI though, when people run tests locally they use the testcafe test runner vs code extension which just uses the CLI. This means they need to manually change the value set in .testcafeRc.json depending on whether the test file has disableNativeAutomation set. It would be a lot better if something like https://github.com/DevExpress/testcafe/pull/8021/files was done instead so tests run the same in every environment.

My team has logged all the native automation bugs we could isolate but they haven't been resolved yet. Additionally, we do notice a lot more instability where our test suites will just freeze intermittently but we haven't been able to isolate the problem so we can't log a bug yet. A lot of our issues might be because of the things you flag under limitations:

TestCafe hangs when it accesses iframes in local HTML files.
TestCafe cannot interact with iframes that the document.write method injects into the page.

[aleks-pro]

Hello @Makavelic ,

Thank you for the explanation.

We researched the possibility of disabling the Native Automation mode at the level of a test or a fixture and it's turned out that it's a very complex task that requires changing TestCafe's runner architecture. We will take your feedback into account. But for now we have other priorities e.g. Native Automation mode improvements.
As a workaround, we suggest that you split your tests into two different tasks: one where the NA mode is enabled and the other where the NA mode is disabled.