pytest discovery fails over ssh if there are too many tests

[tapple]

Type: Bug

Behaviour

Expected vs. Actual

Pytest unit tests are not showing up in the sidebar, when using a remote ssh. In the python output window, I see the pytest --collect-only command ran, and produced the expected output. However, the unit test sidebar still says "no tests have been found in this workspace"

Steps to reproduce:

Windows, VSCode 1.84 (October 2023)
Ubuntu vm
Python extension v2023.20.0
SSH extension v0.106.5

If I downgrade VSCode, the ssh extension, and the python extension to the September 2023 version, unit tests work again

Diagnostic data

• Python version (& distribution if applicable, e.g. Anaconda): 3.9.15
• Type of virtual environment used (e.g. conda, venv, virtualenv, etc.): Conda
• Value of the python.languageServer setting: Default

Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)

XXX

User Settings

languageServer: "Pylance"

testing
• pytestArgs: "<placeholder>"
• pytestEnabled: true

Extension version: 2023.20.0
VS Code version: Code 1.83.1 (f1b07bd25dfad64b0167beb15359ae573aecd2cc, 2023-10-10T23:48:05.904Z)
OS version: Windows_NT x64 10.0.22621
Modes:
Remote OS version: Linux x64 5.15.0-88-generic

System Info

Item	Value
CPUs	11th Gen Intel(R) Core(TM) i7-1185G7 @ 3.00GHz (8 x 2995)
GPU Status	2d_canvas: enabled canvas_oop_rasterization: enabled_on direct_rendering_display_compositor: disabled_off_ok gpu_compositing: enabled multiple_raster_threads: enabled_on opengl: enabled_on rasterization: enabled raw_draw: disabled_off_ok video_decode: enabled video_encode: enabled vulkan: disabled_off webgl: enabled webgl2: enabled webgpu: enabled
Load (avg)	undefined
Memory (System)	15.71GB (2.51GB free)
Process Argv	--crash-reporter-id 985499f3-ac88-4212-8591-40a2d769d4cd
Screen Reader	no
VM	0%

Item	Value
Remote	SSH: vm
OS	Linux x64 5.15.0-88-generic
CPUs	11th Gen Intel(R) Core(TM) i7-1185G7 @ 3.00GHz (3 x 2995)
Memory (System)	4.22GB (0.59GB free)
VM	100%

A/B Experiments

vsliv368cf:30146710
vsreu685:30147344
python383:30185418
vspor879:30202332
vspor708:30202333
vspor363:30204092
vslsvsres303:30308271
vserr242:30382549
pythontb:30283811
vsjup518:30340749
pythonptprofiler:30281270
vshan820:30294714
vstes263:30335439
vscoreces:30445986
vscod805:30301674
binariesv615:30325510
bridge0708:30335490
bridge0723:30353136
vsaa593cf:30376535
pythonvs932:30410667
py29gd2263:30880072
vscaat:30438848
vsclangdc:30486549
c4g48928:30535728
dsvsc012cf:30540253
pynewext54:30695312
azure-dev_surveyone:30548225
2e4cg342:30602488
f6dab269:30613381
showlangstatbar:30737416
pythonfmttext:30731395
fixshowwlkth:30771522
showindicator:30805244
pythongtdpath:30769146
i26e3531:30792625
pythonnosmt12:30797651
pythonidxpt:30866567
pythonnoceb:30805159
asynctok:30869155
dsvsc013:30795093
dsvsc014:30804076
dsvsc015:30845448
pythontestfixt:30871694
pythonregdiag2:30871582
pythonmypyd1:30879173
pythoncet0:30874138
pythontbext0:30879054
dsvsc017cf:30880893

[eleanorjboyd]

Are you on the new testing rewrite? You can check for it by looking at Experiment 'pythonTestAdapter' is active in your python logs (use the command palette to run the command Developer: Set Log Level to trace to see it in your logs). If not can you please turn it on and let me know if the behavior changes? You can do so adding this setting to your users settings.json "python.experiments.optInto": ["pythonTestAdapter"]. Thanks!

[tapple]

I do see in my Python output window, near the top:

2023-11-06 14:14:52.272 [info] Experiment 'pythonTestAdapter' is active

I never turned it on manually, and it says info, not trace

I see it in both vscode versions: 1.83 and 1.84

[eleanorjboyd]

could you include your full logs from the Python output channel? Thanks

[tapple]

• Python.3.log
• Python.2.log
• Python.1.log
• Python.log

The problem seems to be the quantity of the unit tests:

================== 83040 tests collected in 265.78s (0:04:25) ==================

If I ignore the folder with 80k unit tests, it works

The size of the log files and the log rotation seems to desync the python output window as well. It frequently doesn't switch to the latest log file

[tapple]

oh. I waited 10 minutes, and the tests did show up in the tests sidebar. I guess it does work, it's just slow. Also, I get a lot of "window not responding" popups whenever I load the sidebar. maybe unit test refresh should run in a non-blocking way. doubt that's possible without a rework

[tapple]

here's a file that reproduces all the problems:

import pytest
import time

def generate_tests_slow():
    time.sleep(240)
    return range(80000)

@pytest.mark.parametrize("value", generate_tests_slow())
def test_lag(value):
    pass

[eleanorjboyd]

Hello! I have just returned from vacation and will take a look at this shortly, thank you for your patience!

[eleanorjboyd]

@roblourens any thoughts here? Seems like a slow running process run by an extension is causing a repeat warning message "window not responding" as it is working. Is this something I can change on my end to demonstrate a process is under way and not get this pop up?

[roblourens]

Either the test sidebar UI is loading tests in an inefficient way, or the extension is sending them in an inefficient way

cc @connor4312 can you load 80k tests?

[connor4312]

In my testing, loading the 75k tests from the lsprotocol repo was reasonably speedy, with less than a second on the renderer, though the extension host took longer (#22414). I observed the same with the same test case. A renderer profile would be useful here https://github.com/microsoft/vscode/wiki/Performance-Issues#profiling-the-renderer-process

[github-actions]

Because we have not heard back with the information we requested, we are closing this issue for now. If you are able to provide the info later on, then we will be happy to re-open this issue to pick up where we left off.

Happy Coding!