Docker API concurrency Issue when launching too many containers simultaneously

[sk2]

Hi,
I'm experimenting with larger topologies and trying to start a topology of 9 alpine hosts connected via 20 FRR routers.

I intermittently get errors such as

ERRO[0033] failed to create node h2: Post "http://%2Fvar%2Frun%2Fdocker.sock/v1.41/containers/6a1636a58b632e76bc2267373128e276e434171980e7dc51284fee7607f241a8/start": context deadline exceeded

and then subsequently

ERRO[0035] failed to Statfs "": no such file or directory

and

ERRO[0037] failed to run postdeploy task for node h2: failed to Statfs "": no such file or directory

If I then manually start these nodes eg docker start

docker start clab-demo-h2

they start up fine.
The nodes and number of nodes which fail vary between launches, which further suggests something with the timing.

This occurs with FRR/Alpine but I suspect it would also be similar for larger router-container based labs too (which is something I'm looking to try next)

The CPU load of the system isn't too high during this process, so I suspect it may be a bottleneck with the Docker API.
Something like sleep/retry would likely fix it.

Happy to provide some example labs if that would help.

[hellt]

Hi @sk2
can you first try to limit the number of concurrent workers to, say, 10:

containerlab dep --max-workers 10

sometimes with large labs the concurrent API calls to docker API caused issues with the deployment

[sk2]

Thanks! This looks like it did the trick - first run using this and it worked successfully :)

[davidgwatkins]

I ran into this error as well:

-bash: start_pipeline: pgrp pipe: Too many open files in system
-bash: /usr/bin/sudo: Too many open files in system
-bash: cannot make pipe for command substitution: Too many open files in system
(main) expert@expert-cws:~/clab/coin-test$ logout
-bash: /home/expert/.bash_logout: Too many open files in system
-bash: /etc/bash.bash_logout: Too many open files in system

Tried going as low as qty 2 for max-workers, but still the same behavior. Increased system limits.... not hitting CPU/mem utilization on the VM.. it's really strange.