Skip to content

Latest commit

 

History

History
229 lines (175 loc) · 8.78 KB

README.md

File metadata and controls

229 lines (175 loc) · 8.78 KB
Raw

Stratum on a Broadcom based switch

The following guide details how to build Stratum for Broadcom based switches (i.e. Tomahawk) using Broadcom SDKs.

For additional information related to optical transponders like the Cassini, see the TAI README.

ONLPv2 operating system on the switch

Stratum requires an ONLPv2 operating system on the switch. ONF maintains a fork with additional platforms. Follow the ONL instructions to setup your device. Here is what your switch should look like:

# uname -a
Linux as7712 4.14.49-OpenNetworkLinux #4 SMP Tue May 14 20:43:21 UTC 2019 x86_64 GNU/Linux
# cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
NAME="Debian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
VERSION_CODENAME=stretch
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
# cat /etc/onl/SWI
images:ONL-onf-ONLPv2_ONL-OS_<some-date>_AMD64.swi

Note the ONLPv2!

# cat /etc/onl/platform
x86-64-<vendor-name>-<box-name>-32x-r0

Pre-built Docker image

Stratum for Broadcom switches can be run inside Docker on the switch itself. Follow their instructions on how to setup Docker. As part of CI, we publish Stratum with a pre-compiled binary and a set of default configuration files as a Docker container. There are two versions, one for SDKLT (:sdklt) and one for OpenNSA (:openNSA).

docker pull stratumproject/stratum-bcm:sdklt  # or :opennsa, to update the image
cd stratum/hal/bin/bcm/standalone/docker
DOCKER_IMAGE_TAG=sdklt ./start-stratum-container.sh  # or =opennsa

Compile from source

Sometimes you have to build Stratum from source, e.g. because you develop some private feature or want to try a fix not yet pushed to GitHub.

Build dependencies

Stratum comes with a development Docker container for build purposes. This is the preferred and supported way of building Stratum, as it has all dependencies installed.

If you for some reason want to build natively, here are some pointers to an environment that worked for us:

  • clang-6.0 or newer

  • Linux 4.4.0-161-generic

  • Ubuntu 16.04.6 LTS

Building the stratum_bcm Debian package

You can build the same package that we publish manually with the following steps:

git clone https://github.com/stratum/stratum.git
cd stratum
./setup_dev_env.sh  # You're now inside the docker container
bazel build //stratum/hal/bin/bcm/standalone:stratum_bcm_sdklt_deb  # or stratum_bcm_opennsa_deb
scp ./bazel-bin/stratum/hal/bin/bcm/standalone/stratum_bcm_sdklt_deb.deb root@<your_switch_ip>:
# On the switch
apt-get install -f ./stratum_bcm_sdklt_deb.deb

If you're not building inside the docker container, skip the ./setup_dev_env.sh step.

Starting Stratum

To start Stratum, use the convenience script we install with the package:

start-stratum.sh

You should see the ports coming up and have a SDKLT shell prompt:

I0628 18:29:10.806623  7930 bcm_chassis_manager.cc:1738] State of SingletonPort (node_id: 1, port_id: 34, slot: 1, port: 3, unit: 0, logical_port: 34, speed: 40G): UP
BCMLT.0>

Stratum can be customized with several configuration files, passed as CLI flags. We provide sensible defaults for most platforms and the script automatically uses the correct ones for the platform.

  • base_bcm_chassis_map_file: Protobuf defining chip capabilities and all possible port configurations of a chassis. Example found under: /stratum/hal/config/**platform name**/base_bcm_chassis_map.pb.txt
  • chassis_config_file: Protobuf setting the config of a specific node. Selects a subset of the available port configurations from the chassis map. Determines which ports will be available. Example found under: /stratum/hal/config/**platform name**/chassis_config.pb.txt
  • bcm_sdk_config_file: Autogenerated SDK configuration file.
  • bcm_hardware_specs_file: ACL and UDF properties of chips. Found under: /stratum/hal/config/bcm_hardware_specs.pb.txt
  • bcm_serdes_db_proto_file: Contains SerDes configuration. Not implemented yet, can be an empty file.

If you followed the build instructions, these should be on the switch under /etc/stratum/stratum_configs. Depending on your actual cabling, setup or hardware, you'll have to adjust the config files.

Pushing a Pipeline

Before P4Runtime requests can be sent to program the data plane, a pipeline has to be pushed to the switch. We provide a reference pipeline for Broadcom based switches called main.p4. Refer to its README for further instructions.

Manual SDK setup on the switch

ONLY needed when not using Docker or the start script!

The start-stratum.sh script takes care of setting up the SDKs for you. Should something go wrong, these steps help you troubleshoot:

SDKLT

SDKLT requires two Kernel modules to be installed for Packet IO and interfacing with the ASIC. We provide prebuilt binaries for Kernel 4.14.49 in the Debian package and the SDKLT tarball. Install them before running stratum:

cd /usr/lib/stratum/
# or
wget https://github.com/opennetworkinglab/SDKLT/releases/...
tar xf sdklt-4.14.49.tgz
insmod linux_ngbde.ko && insmod linux_ngknet.ko

Check for correct install:

# lsmod
Module                  Size  Used by
linux_ngknet          352256  0
linux_ngbde            32768  1 linux_ngknet
# dmesg -H
[Jan10 10:53] linux-kernel-bde (6960): MSI not used
[  +2.611898] Broadcom NGBDE loaded successfully

OpenNSA

From the Stratum package, install the Kernel modules:

insmod linux-kernel-bde.ko && insmod linux-user-bde.ko && insmod linux-bcm-knet.ko

Check for correct install:

# lsmod
Module                  Size  Used by
linux_bcm_knet         77824  0
linux_user_bde         24576  0
linux_kernel_bde       61440  2 linux_bcm_knet,linux_user_bde
# dmesg -H
[Apr14 21:31] linux-kernel-bde (11885): MSI not used

Troubleshooting

insmod: ERROR: could not insert module linux_ngbde.ko: Invalid module format

You are trying to insert Kernel modules build for a different Kernel version. Make sure your switch looks exactly like described under Runtime dependencies.

No traffic on ports with partial ChassisConfig [OpenNSA]

When using a Chassis config that does not contain all possible ports of the switch, traffic might not be received or sent on a subset or all of the ports. This is because of a quirk in the OpenNSA SDK and its config file, which we generate from the chassis config. It seems that only initializing some ports of a port quad (i.e. 3 and 4, but not 1 and 2, using logical port numbers) leads to this behaviour. Easiest fix is to use the full file and set unwanted ports administratively to down state.

OpenNSA config file error

ERROR: Assertion failed: (rm) at /projects/ntsw_sw18_scratch/sdkrel/openbcm/sdk/src/sal/core/unix/sync.c:556
*** Aborted at 1586901146 (unix time) try "date -d @1586901146" if you are using GNU date ***
PC: @                0x0 (unknown)
*** SIGABRT (@0x31e5) received by PID 12773 (TID 0x7f9c73602840) from PID 12773; stack trace: ***
    @     0x7f9c7241b890 (unknown)
    @     0x7f9c71650067 gsignal
    @     0x7f9c71651448 abort
    @           0x5ebfff _default_assert
    @           0x5eba08 sal_mutex_take
    @           0xce1b40 sal_vprintf
    @           0xce1c2d sal_printf
    @           0xce28df sal_config_file_process
    @           0xce2ad5 sal_config_refresh
    @           0x52b98c stratum::hal::bcm::BcmSdkWrapper::InitializeSdk()
    @          0x2126983 stratum::hal::bcm::BcmChassisManager::InitializeBcmChips()
    @          0x211bc1c stratum::hal::bcm::BcmChassisManager::PushChassisConfig()
    @          0x2108042 stratum::hal::bcm::BcmSwitch::PushChassisConfig()
    @          0x25f643a stratum::hal::ConfigMonitoringService::PushChassisConfig()
    @          0x25f5be2 stratum::hal::ConfigMonitoringService::PushSavedChassisConfig()
    @          0x25f57fc stratum::hal::ConfigMonitoringService::Setup()
    @          0x2477e72 stratum::hal::Hal::Setup()
    @           0x51f1e6 stratum::hal::bcm::Main()
    @     0x7f9c7163cb45 __libc_start_main
    @           0x447c9a _start
    @                0x0 (unknown)

Your base chassis map contains values that OpenNSA does not understand. You can check which config was generated in --bcm_sdk_config_file.