Issues with SR-OS custom variants (distributed)

[danmel42]

Hi folks,

I'm trying to move my lab from GNS3 to containerlabs. I use SR-OS, SR Linux, vMX and Linux devices mostly. That works very well in general.

However I have some trouble to bring distributed custom SR-OS devices into life.

I've for instance an IXR-X3 (22.10.R2).

topology:
  nodes:
    edge01:
      kind: vr-sros
      image: vrnetlab/vr-sros:22.10.R2
      type: >-
        cp: cpu=6 ram=6 slot=A chassis=ixr-x3 card=cpm-ixr-x/imm36-qsfpdd ___
        lc: cpu=8 ram=8 max_nics=36 slot=1 chassis=ixr-x3 card=cpm-ixr-x/imm36-qsfpdd
      license: license-sros22.txt

Device boots up, but the boot of IMM 1 in slot 1 fails and the card is going to reboot. This behavior repeats forever.

A:admin@edge01# show card

===============================================================================
Card Summary
===============================================================================
Slot      Provisioned Type                         Admin Operational   Comments
              Equipped Type (if different)         State State         
-------------------------------------------------------------------------------
1         imm36-qsfpdd                             up    booting       
A         cpm-ixr-x                                up    up/active     
===============================================================================

[/]
A:admin@edge01# show card

===============================================================================
Card Summary
===============================================================================
Slot      Provisioned Type                         Admin Operational   Comments
              Equipped Type (if different)         State State         
-------------------------------------------------------------------------------
1         imm36-qsfpdd                             up    provisioned   
              (not equipped)                       
A         cpm-ixr-x                                up    up/active     
===============================================================================

[/]
A:admin@edge01#

The same is true for a setup with an IXR-6 where all the IMMs are constantly rebooting

Then I've a setup with a SR-1se (also 22.10.R2)

topology:
  nodes:
    edge01:
      kind: vr-sros
      image: vrnetlab/vr-sros:22.10.R2
      type: >-
        cp: cpu=4 ram=6 slot=A chassis=sr-1se card=cpm-1se/imm36-800g-qsfpdd ___
        lc: cpu=4 ram=6 max_nics=36 slot=1 chassis=sr-1se card=cpm-1se/imm36-800g-qsfpdd
      license: license-sros22.txt

This setup doesn't even boot completly. It starts, but I can't ssh to the console.

All setups mentioned above are running fine with GNS3 on the same physical host as containerlab.
For instance the SR-1se:

A:admin@vSIM# show chassis 
===============================================================================
System Information
===============================================================================
  Name                              : vSIM
  Type                              : 7750 SR-1se
  Chassis Topology                  : Standalone
  Location                          : (Not Specified)
  Coordinates                       : (Not Specified)
  CLLI code                         :           
  Number of slots                   : 2
  Oper number of slots              : 2
  Num of faceplate ports/connectors : 36
  Num of physical ports             : 0
  Critical LED state                : Off
  Major LED state                   : Off
  Minor LED state                   : Off
  Over Temperature state            : OK
  Base MAC address                  : 62:98:ff:00:00:00
  FP Generations                    : FP5
  System Profile                    : profile-a

===============================================================================
Chassis Summary
===============================================================================
Chassis   Role                Status
-------------------------------------------------------------------------------
1         Standalone          up
===============================================================================

[/]
A:admin@vSIM# show card 

===============================================================================
Card Summary
===============================================================================
Slot      Provisioned Type                         Admin Operational   Comments
              Equipped Type (if different)         State State         
-------------------------------------------------------------------------------
1         imm36-800g-qsfpdd:he19200g               up    up            
A         cpm-1se                                  up    up/active     
===============================================================================

[/]
A:admin@vSIM# show mda

===============================================================================
MDA Summary
===============================================================================
Slot  Mda   Provisioned Type                            Admin     Operational
                Equipped Type (if different)            State     State
-------------------------------------------------------------------------------
1     1     ms36-800g-qsfpdd                            up        up
===============================================================================

[/]
A:admin@vSIM# 

So do you know any restrictions regarding the distributed setups or regarding SR-OS version 22.10.R2?
Or did I mess up the type definition somehow?

Thanks for help and best regards,
Daniel

[hellt]

Hi @danmel42
you can check what happens with the node by looking at its logs
docker logs -f <container name>

I hope you will see some errors there once you redeploy the lab

[hellt]

it seems that you used an older version of hellt/vrnetlab to build the container
can you check that and potentially rebuild with the latest version?

[danmel42]

@hellt Many thanks for the hint.
It seems to me from the logs that there are not enough vCPUs and not enough memory allocated for the VM(s).

2023-01-27 15:10:18,491: launch INFO SR OS Variant: cp: cpu=4 ram=6 slot=A chassis=sr-1se card=cpm-1se/imm36-800g-qsfpdd ___ lc: cpu=4 ram=6 max_nics=36 slot=1 chassis=sr-1se card=cpm-1se/imm36-800g-qsfpdd
So from the configuration the cp should get 4 vCPUs and 6 GB of memory, the lc should get 4 vCPUs and 6GB of memory.

But then qemu starts the 2 VMs with 2 vCPUs each and 4GB of memory for each VM.

2023-01-27 15:10:18,545: vrnetlab INFO Starting SROS_cp 2023-01-27 15:10:18,545: vrnetlab DEBUG ['qemu-system-x86_64', '-enable-kvm', '-display', 'none', '-machine', 'pc', '-monitor', 'tcp:0.0.0.0:4000,server,nowait', '-m', '4096', '-serial', 'telnet:0.0.0.0:5000,server,nowait', '-drive', 'if=ide,file=/sros-overlay-cp.qcow2', '-cpu', 'host', '-smp', '2', '-uuid', '31a14163-e87a-41d4-a1c0-1b002ab8d158', '-smbios', '"type=1,product=TIMOS:address=172.31.255.30/30@active license-file=tftp://172.31.255.29/license.txt primary-config=tftp://172.31.255.29/config.txt system-base-mac=52:54:00:67:ab:00 slot=A chassis=sr-1se card=cpm-1se/imm36-800g-qsfpdd"', '-device', 'virtio-net-pci,netdev=br-mgmt,mac=52:54:00:eb:14:00', '-netdev', 'bridge,br=br-mgmt,id=br-mgmt', '-device', 'virtio-net-pci,netdev=vcp-int,mac=52:54:00:fc:62:01', '-netdev', 'tap,ifname=vcp-int,id=vcp-int,script=no,downscript=no']

2023-01-27 15:10:24,566: vrnetlab INFO Starting SROS_lc 2023-01-27 15:10:29,572: vrnetlab DEBUG ['qemu-system-x86_64', '-enable-kvm', '-display', 'none', '-machine', 'pc', '-monitor', 'tcp:0.0.0.0:4001,server,nowait', '-m', '4096', '-serial', 'telnet:0.0.0.0:5001,server,nowait', '-drive', 'if=ide,file=/sros-overlay-lc.qcow2', '-cpu', 'host', '-smp', '2', '-uuid', '31a14163-e87a-41d4-a1c0-1b002ab8d158', '-smbios', '"type=1,product=TIMOS:slot=1 chassis=sr-1se card=cpm-1se/imm36-800g-qsfpdd"', '-device', 'pci-bridge,chassis_nr=1,id=pci.1', '-device', 'pci-bridge,chassis_nr=2,id=pci.2', '-device', 'virtio-net-pci,netdev=mgmt,mac=52:54:00:cb:ed:00', '-netdev', 'user,id=mgmt,net=10.0.0.0/24', '-device', 'virtio-net-pci,netdev=vfpc-int,mac=52:54:00:8f:1b:00', '-netdev', 'tap,ifname=vfpc1-int,id=vfpc-int,script=no,downscript=no']

Current Time: FRI JAN 27 15:11:10 UTC 2023
Loading license file for 'TiMOS-C-22.10.R2' from: tftp://172.31.255.29/license.txt ... OK
Executing TiMOS image at 0x150000

KVM based vcpu
Running in a KVM/QEMU virtual machine
ACPI: found 2 cores, 2 enabled

2023-01-27 15:11:30,658: launch     TRACE    OUTPUT: 
Total Memory: 4GB  Chassis Type: 0x2d  Core Freq: 2394.230 MHz
TiMOS-C-22.10.R2 cpm/x86_64 Nokia 7750 SR Copyright (c) 2000-2022 Nokia.
All rights reserved. All use subject to applicable license agreements.
Built on Wed Dec 21 17:45:12 PST 2022 by builder in /builds/c/2210B/R2/panos/main/sros

That could be the reason for system crashing during boot (SR-1se) or linecards constantly rebooting (IXR-6 / IXR-X3).

Daniel

[danmel42]

it seems that you used an older version of hellt/vrnetlab to build the container
can you check that and potentially rebuild with the latest version?

That exactly was the trick. Thank you. That's what happens when you just copy and don't think about it. 🤪