[chore]: fix go routine leaks in tests

[odubajDT]

Description:

• removing re-running parameter from gotestsum
• fixing tests/code with go routine leaks
• resolving race conditions (mostly caused by parallel tests)
• placing goleak ignorers for go routine leaks from external libraries

Link to tracking Issue: #34495

[mx-psi]

Why is this option removed?

[odubajDT]

enabling the option led to re-running the failed tests (which mostly failed to to go routine leaks and race conditions), which always led to a passed test. This behavior was from some reason blocking the actual problems which were shown after it was removed

[mx-psi]

I added this in #31253 because we have a problem with flaky tests and not enough resources to deal with them. I feel like this is still the case and that if we remove this we will get a more flaky CI which is detrimental to contributors. Is there any way we can detect leaks while still re-running other unit tests?

[odubajDT]

I am not aware of any possibility here. In fact it seems very strange to me that all of the failing tests become passed after re-running them for the 2. time.

We can leave the re-run option in place until we find the root cause why the re-executed tests have very deterministic behavior in terms of go-leaks and data-races, but we should at least consider merging the fixed tests as part of this PR.

Maybe it can be a caching issue in the gotestsum wrapper, not sure here...

[mx-psi]

Can you report this upstream on the gotestsum repository then to see if it's actually an issue with them?

[odubajDT]

reported here

But there seems to be a bigger problem with the --rerun-fails option, as I see multiple issues created on the upstream related to it https://github.com/gotestyourself/gotestsum/issues

[odubajDT]

cc @mx-psi

[mx-psi]

I subscribed to gotestyourself/gotestsum/issues/442 and I would like to wait a bit more to see if upstream has anything to say here. I see some issues related to --rerun-fails, but I don't see any specifically related to this.

I am happy to merge the rest of the changes and only keep --rerun-fails to a separate PR, would that work for you?

[odubajDT]

ok, reverted the --rerun-fails and will keep an eye on the issue opened on gotestsum upstream.

When it will be resolved, will open a new PR