Uncaught TypeError When Using Roles and Concurrency

[GuacoIV]

What is your Scenario?

When I use concurrency and roles, I get an uncaught JavaScript error in hammerhead.js. It seems to only occur if I start moving the browser windows around to various positions as they launch. Interestingly, the message and call stack are empty in TestCafe's error message, yet they are visible in the browser console if you --debug-on-fail and look. This is bad because I can't even ignore the error using a specific message filter.

What is the Current behavior?

• Tests intermittently fail in the fixture's beforeEach which sets the Role
• The error message and stack trace are blank in the output of the shell I launched it from

Note that this does not happen every single time. See Steps to Reproduce.

What is the Expected behavior?

• All tests pass
• Uncaught exceptions have a message that can be filtered on, if they occur

What is the public URL of the test page? (attach your complete example)

https://practicetestautomation.com/practice-test-login/

What is your TestCafe test code?

import { Role } from 'testcafe';

var role = new Role("https://practicetestautomation.com/practice-test-login/", async t => {
    await t
        .typeText('#username', 'student')
        .typeText('#password', 'Password123')
        .click('#submit');
}, { preserveUrl: true });

fixture('Getting Started')
    .beforeEach(t => t.useRole(role))
    .page('https://practicetestautomation.com/practice-test-login/');

test('My first test', async t => action(t));
test('My second test', async t => action(t));
test('My third test', async t => action(t));
test('My fourth test', async t => action(t));
test('My fifth test', async t => action(t));
test('My sixth test', async t => action(t));

async function action(t) {
    for (var i; i < 20; i++) {
        await t.click('#menu-item-20')
            .wait(50);
    }
}

Your complete configuration file

{
"concurrency" : 4
}

Your complete test report

Running tests in:

• Chrome 131.0.0.0 / Windows 11
• Chrome 131.0.0.0 / Windows 11
• Chrome 131.0.0.0 / Windows 11
• Chrome 131.0.0.0 / Windows 11

Getting Started
× My first test

1. • Error while restoring configuration after Role switch -
     A JavaScript error occurred on "https://practicetestautomation.com/practice-test-login/".
     Repeat test actions in the browser and check the console for errors.
     Enable the “skipJsErrors” option to ignore JavaScript errors during test execution. Learn more: "https://testcafe.io/documentation/404038/recipes/debugging/skip-javascript-errors"
     If the website only throws this error when you test it with TestCafe, please create a new issue at:
     "https://github.com/DevExpress/testcafe/issues/new?template=bug-report.md".

   JavaScript error details:
   Script error.
   No stack trace available

   Browser: Chrome 131.0.0.0 / Windows 11

   6 |        .typeText('#password', 'Password123')
   7 |        .click('#submit');
   8 |}, { preserveUrl: true });
   9 |
   

   10 |fixture('Getting Started')

   11 | .beforeEach(t => t.useRole(role))
   12 | .page('https://practicetestautomation.com/practice-test-login/');
   13 |
   14 |test('My first test', async t => action(t));
   15 |
   16 |test('My second test', async t => action(t));

   at (C:\Users<USER>\getting-started\getting-started.js:11:24)

[TestRail] Test missing the TestRail Case ID in test metadata: My first test
√ My second test
[TestRail] Test missing the TestRail Case ID in test metadata: My second test
× My third test

1. • Error while restoring configuration after Role switch -
     A JavaScript error occurred on "https://practicetestautomation.com/practice-test-login/".
     Repeat test actions in the browser and check the console for errors.
     Enable the “skipJsErrors” option to ignore JavaScript errors during test execution. Learn more: "https://testcafe.io/documentation/404038/recipes/debugging/skip-javascript-errors"
     If the website only throws this error when you test it with TestCafe, please create a new issue at:
     "https://github.com/DevExpress/testcafe/issues/new?template=bug-report.md".

   JavaScript error details:
   Script error.
   No stack trace available

   Browser: Chrome 131.0.0.0 / Windows 11

   6 |        .typeText('#password', 'Password123')
   7 |        .click('#submit');
   8 |}, { preserveUrl: true });
   9 |
   

   10 |fixture('Getting Started')

   11 | .beforeEach(t => t.useRole(role))
   12 | .page('https://practicetestautomation.com/practice-test-login/');
   13 |
   14 |test('My first test', async t => action(t));
   15 |
   16 |test('My second test', async t => action(t));

   at (C:\Users<USER>\getting-started\getting-started.js:11:24)

[TestRail] Test missing the TestRail Case ID in test metadata: My third test
√ My fourth test
[TestRail] Test missing the TestRail Case ID in test metadata: My fourth test
√ My fifth test
[TestRail] Test missing the TestRail Case ID in test metadata: My fifth test
√ My sixth test
[TestRail] Test missing the TestRail Case ID in test metadata: My sixth test

2/6 failed (26s)

Screenshots

Steps to Reproduce

1. Run the command to start TestCafe (note that there is concurrency setting of 4 in the config, or otherwise add parameter)
2. Browser windows start opening. Begin to move them around as if you want to view all of them while they run.

The issue does not occur every time so if it does not, try focusing on a particular window to move little bits at a time, possibly the first or last one (not certain if it matters though)

TestCafe version

3.7.0 (also occurs in 3.7.1)

Node.js version

v22.11.0

Command-line arguments

testcafe chrome getting-started.js

Browser name(s) and version(s)

Chrome 131

Platform(s) and version(s)

Windows 11

Other

In case it matters, note that I am using the preserveUrl option in the constructor of the Role

[Bayheck]

Hello,

Thank you for reaching out to us.

I did not reproduce this issue. This is a rare usage scenario, and TestCafe specializes in fully automated functional testing. So, we believe manual testing (manually managing browsers) is far beyond its capabilities.

Please clarify why you need to manually interact with your tests.

[GuacoIV]

Hi Baychek,

The reason for moving the browser window is simply to be able to see and watch them work and monitor progress and failures. The windows start all stacked on top of each other and so it does not strike me as something out of the ordinary or "far beyond the capability" for the user to simply move the windows to monitor them, but rather a natural thing to do.

Though you may have not replicated, or you may need to do some extra movements in this stripped down simple working example, in our real world tests this happens almost every time. I would like to request that you please don't allow my ability to produce a 100% hit rate example to affect the analysis of this being an issue.

If it is documented somewhere that I should be using some configuration to position them, or that doing so manually can break them please let me know.

[Bayheck]

Hello, could you please elaborate on

in our real world tests this happens almost every time

Do you also manually move the browser windows in your real world test runs?

I am asking since I thought that the issue occurs only when you move the windows in local tests, based on:

It seems to only occur if I start moving the browser windows around to various positions as they launch

As for,

The reason for moving the browser window is simply to be able to see and watch them work and monitor progress and failures.

You can check the progress and test run results using our reporters. (https://testcafe.io/documentation/402825/guides/intermediate-guides/reporters)

NOTE: TestCafe emulates mouse and keyboard events (such as dragging, hovering, moving, keypressing, etc.) Interacting with the system via the cursor or keyboard can affect test execution and we have not tested this use case; therefore, it is not recommended.

[GuacoIV]

Hi - sure thing.

Regarding "in our real world tests this happens almost every time": By this I mean when running not the simple example I constructed but our actual tests against our website and not the practice test automation site. We do run the tests locally and since we are starting to use concurrency, would like to to move the windows either out of the way from where they spawn and/or arrange them so they can be monitored.

That's right. If we don't move the windows around then the issue does not occur. But, the windows may spawn in an inconvenient location.

I can't deny that one can technically monitor progress with the reporters. But I think it's also reasonable to say that it is not ideal for 3/4 of what's going on to be obscured from view.

Regarding this note:

NOTE: TestCafe emulates mouse and keyboard events (such as dragging, hovering, moving, keypressing, etc.) Interacting with the system via the cursor or keyboard can affect test execution and we have not tested this use case; therefore, it is not recommended.

Could you please point me to where this is documented so that I understand what the documented limitations are? The click/drag in question occurs on the window's title bar and not within the browser viewport itself.

I've realized that there is a separate repo for testcafe-hammerhead. Since hammerhead is in the call stack, I'd like to propose if we can't shift this issue over to that repo?

[aleks-pro]

Hello, @GuacoIV

Thank you for the clarification.

Could you please point me to where this is documented so that I understand what the documented limitations are?

It looks like we have no notes regarding those limitations yet but we definitely should add them. I will add corresponding task to our backlog.

I've realized that there is a separate repo for testcafe-hammerhead. Since hammerhead is in the call stack, I'd like to propose if we can't shift this issue over to that repo?

Of course we could do this, but as maintainers we are trying to persist all the requests regarding testcafe and its' dependent projects in a single repository. It helps us to process user request more efficiently.

@GuacoIV, I'll be honest with you—your usage scenario is quite rare. Additionally, we don’t have a straightforward way to reproduce the issue, which complicates debugging and implementation. As a result, the chances that we find resources to address it are very low. However, if we receive similar requests, we can reconsider the priority of this issue.

In general, concurrency is used when you have multiple tests that significantly slow down the overall task execution. In such cases, you can run all your tests in concurrency mode, identify the failed ones, and then rerun those tests without concurrency. This approach should help resolve the issue for you.

[GuacoIV]

@aleks-pro Thanks for your feedback. Understood about keeping the issues in one place. I find that in the documentation that concurrency is encouraged as a best practice. Is it uncommon to be running the tests locally and that is what is common about my scenario?

As another note, I did find my example is missing await keywords when calling action. Also I apparently did not initialize the variable i to zero in the loop. I don't know if that go mistakenly cleaned up when I was getting ready to submit this or what. Oddly, I'm not able to replicate the issue in my simple example today (whether I make those alterations or not), whereas I had about a 50% hit rate on the day that I found the issue.

[aleks-pro]

Is it uncommon to be running the tests locally and that is what is common about my scenario?

Usually, you run tests locally to create and debug them and use some kind of CI server to automate test task execution. The uncommon part in your usage scenario is the interaction with the browser window while you execute tests. So, executing tests locally is fine.

I did find my example is missing await keywords when calling action.

We didn't notice this in your example. It's very possible that this was the cause of the issue. Please check if you can still reproduce the problematic behavior. If not, we will close the issue.