Playwright test timing out - flaky

[tijlleenders]

Describe the bug
Playwright test is timing out. This is seen as a PASS in CI.

To Reproduce
See GH Action:

[vinaybadgujar102]

@tijlleenders the test shows passed because as mentioned in the line 39, one test is flaky means it failed initially but passed on retry. you can refer to https://playwright.dev/docs/test-retries.

Tests sometimes fail because the elements related to sharing/collaboration feature doesn't always work as expected. For ex. when user shares a goal, the reciever should see it after refreshing the page, but this doesn't always happen. This unpredictable behavior is probably why the test passes when run again

[tijlleenders]

main branch is failing after soft delete PR. This is not a good situation. Please find a fix.

[vinaybadgujar102]

@tijlleenders the test fails when it tries to accept the changes made to the goal during collaboration. I don't know why this is happening. This works fine when I do it myself (manually), but the test isn't passing. I need to look into this thoroughly, it might take some time.

[tijlleenders]

You mean running the same test in an automated way - so not manually - works on your machine?

[vinaybadgujar102]

No. What I mean is that if I, as a user, perform the steps mentioned in the test manually, it works correctly. However, it does not work in the test environment.
It is not working on my machine either.

[tijlleenders]

Ok, well then the first step is to get them working in an automated fashion on your local machine.

• If that works, we look at the difference in run environment vs CI.
• If not, it's the automated test that is the issue, not the environment/CI. Probably you need to add a refresh (otherwise you won't trigger fetch updates) - or add a longer wait.

[vinaybadgujar102]

I tried everything I could to make the test run correctly (added refreshes, waits, page changes, playwright version updates, etc) but it doesn't seem to work, don't know why.

[tijlleenders]

Ok, well then the first step is to get them working in an automated fashion on your local machine.

* If that works, we look at the difference in run environment vs CI.

* If not, it's the automated test that is the issue, not the environment/CI. Probably you need to add a refresh (otherwise you won't trigger fetch updates) - or add a longer wait.

Does it work automated locally?

[vinaybadgujar102]

No it is not working locally also. It passes once in a while but mostly fails.

[tijlleenders]

Ok, well at least then you have a good setup to iterate possible solutions. What's the flop-rate? Does it succeed 10% or 90% of the time? Just retrying the test twice can be a workaround for now.

[vinaybadgujar102]

Does it succeed 10% or 90% of the time?

10%.
I will have to go through the documentation deeply to find the solution.

[tijlleenders]

@vinaybadgujar102 Can we increase the time the test is allowed to run to two minutes to see if that fixes the flaky tests?

[vinaybadgujar102]

@tijlleenders, Will give this a shot.

[tijlleenders]

Closed by #2049