Simultaneous boot of many appliances

[tacerus]

Hi,

Thanks for the great software, I managed vrnetlab containers manually before and found this to make network simulation way easier! :-)

I can start ~5 devices at once with no issues, however if I try the same with about ~25 vQFX and vSRX devices the boot of each appliance will stall at about this point:

Hit [Enter] to boot immediately, or space bar for command prompt.
2023-08-13 02:52:10,132: launch     TRACE    OUTPUT VCP:
Booting [/kernel]...
/
Simulating VIRTUAL ELIT!!

vQFX_serial_number: VM5F3D5FF6E7
Serial Number: VM5F3D5FF6E7
Product Name: vQFX-TVP PC (i440FX + PIIX, 1996)
Version: 1.0i440fx-7.1
Board Version: 1GDB: debug ports: sio
GDB: current port: sio
KDB: debugger backends: ddb gdb kdm
KDB: current backend: ddb

I don't know why this happens, as the host machine has plenty resources:

$ free -h
                    total        used        free      shared  buff/cache   available
Mem:          174Gi        36Gi       129Gi        24Mi        11Gi       138Gi
Swap:          16Gi        71Mi        16Gi
$ nproc
60

This memory usage is at the point of the lockup and there are no OOM messages in the kernel log.

I found that starting the individual vrnetlab containers with a delay in between helps the lab to come up.

Is such a "one by one" container startup possible with Containerlab?
Using BOOT_DELAY does not seem to help, as the boot is delayed in the individual containers and not between them.

Of course, any ideas as to why this lockup happens would be appreciated as well!

The machine is using nested virtualization using KVM, but the host machine has plenty free resources as well.

[LimeHat]

Is such a "one by one" container startup possible with Containerlab?

Possible, but not automatically. You can configure startup-delay for individual nodes.

[hellt]

Hi @tacerus
Exactly as @LimeHat said, startup delay feature was created to fulfill a need to start nodes in batches that (imperatively) boot up fine on a given hw.

[tacerus]

Thank you both, I missed that option!