Snabb 2022.01 “Octarina”

This release contains the a new driver for Mellanox Connect-X series NICs, an improved Intel AVF driver with support for RSS, and an experimental XDP driver. As usual this release also brings many smaller improvements and bug fixes.

What's Changed

• Mellanox ConnectX-4 Driver (WIP) by @capr in #918
• Completed Mellanox initialization sequence by @lukego in #943
• Upstream lwAFTR 2022.01.x by @eugeneia in #1465
• Fixes for regressions in max-next headed for Octarina release by @eugeneia in #1466
• Collected changes for Jan 2022 release: Octarina by @eugeneia in #1463
• Snabb 2022.01 “Octarina” by @eugeneia in #1467
• XDP app: driver for Linux AF_XDP sockets by @eugeneia in #1450
• intel_avf: fix integration breakage and free resources on shutdown (stop()) by @eugeneia in #1451
• Various YANG fixes related to choice statements and the consistency checker by @eugeneia in #1452
• Add filter option to XDP app for punting packets to Linux by @eugeneia in #1454
• lib.numa: gracefully handle failure to read CPU performance governor by @eugeneia in #1455
• lib.ptree: cleanup obsolete aggregated stats by @eugeneia in #1456
• arch/checksum: fix bug in carry propagation by @eugeneia in #1457
• lib.ctable: fix hugepages allocation size by @alexandergall in #1459
• core.packet: fix account_free physical capacity formula by @eugeneia in #1460
• Driver for Mellanox Connect-X 4/5/6 cards by @eugeneia in #1461
• Updated Intel AVF driver with VLAN+RSS support by @eugeneia in #1462
• Tracing JIT stability improvements from lwAFTR by @eugeneia in #1464

Full Changelog: v2019.11...v2022.01

Contributors

Alexander Gall (24):

• 48f6080: Cherry-picked apps.melanox from lukego/debug100g
• 4474000: Uncommited changes from lukego
• 580cf9b: connectx4.lua: minor changes
• 10ceea4: connectx4.lua: fix alignment of HCA command queue
• 1b9cf2b: connectx4.lua: fix RSS
• 8332671: Register ConnectX5 device in lib.hardware.pci
• 263919a: connectx4.lua: identify driver for device_info
• d57c67a: connectx4.lua: add statistics counters
• 22af847: connectx4.lua: use async commands to read stats counters
• a949245: connectx4.lua: use a factory to create HCAs
• b4d9252: connectx4.lua: fix bug with async commands when debug_hexdump is enabled
• e2eecff: connectx4.lua: fix page sizes in CREATE_{C,R}Q
• 76c0c80: apps.vhost: suppress -Wstringop-truncation
• 56884a7: apps.mellanox.connectx4: add MT28800 as supported device
• a0e8585: apps.mellanox.connectx4: add per-queue drop counters for ConnectX5
• fc37be4: apps.mellanox.connectx4: unbind device in IO:new()
• adc7d3d: apps.mellanox.connectx4: fix mcast counters
• 2219748: Fix transitions in CXQ state machine
• 6cdcb40: apps.mellanox: free CXQ during process shutdown
• c579524: apps.mellanox: use correct WQE stride in RX queues
• 556e427: mellanox: complete renaming ConnectX4 -> ConnectX
• a348899: lib.ctable: fix hugepages allocation size
• a86c61e: connectex.lua: extend RSS hashing to non-TCP/UDP IP packets
• 5f10b3b: connectx.lua: convert tabs to spaces

Andy Wingo (5):

• 9a787b5: Fix snabb top interface view for multiple queues
• 7a0c6c8: Disable vmprofile on lwAFTR by default
• 648c67a: Add 2019.06.02 changelog entry
• 6524dec: fix typo
• 828fa4f: lwAFTR version 2019.06.02

Cosmin Apreutesei (6):

• 45d0696: stub module and readme
• 8720a60: pci init
• cd3a103: vendor/dev id
• dfbd705: firmware revision register
• 34929f1: print firmware revision maj and min
• fe8065f: init procedure until QUERY_HCA_CAP

Luke Gorrie (24):

• a6973da: lib.hardware.pci: Add reset_device()
• ccdf38e: mellanox: Added text/hexdump trace of Linux driver
• 276f4fa: connectx4: Progress on initialization routine
• 6a426bd: connectx4: Translate virtual to physical on demand
• f349c41: core.memory: Add 'align' argument to dma_alloc()
• 8c55f66: connectx4: Request alignment on memory.dma_alloc()
• 2c1a92d: connectx4: Add multi-mailbox command support
• 6abbd96: connectx4: Successful QUERY_HCA_CAP query
• 92ec410: connectx4: Command mailbox chaining fixes
• 13e6550: connectx4: Set debug = false
• cd8e30e: connectx4: Remove stray print statements
• a285999: connectx4: Move checkstatus() into post()
• 4575dc7: connectx4: Device initialization fixes
• 8c0be9d: connectx4: Comment on ommitted SET_DRIVER_VERSION
• f196ac7: connectx4: Added CREATE_EQ (event queue) command
• c1cee02: connectx4: Added initial VPORT commands
• 51d4d45: connectx4: Give 'lock' argument to map_pci_memory
• 7659eb6: connectx4: Clean driver initialization
• 21d0dc3: connectx4: Refactored for multiprocess operation
• f08da3c: connectx4: IO app now has only 1xSQ and 1xRQ
• ffe4a22: connectx4: Add 'mtu' paramter (default 9500)
• 06c43f0: connectx4: Cleaner (simplistic) selftest method
• e367b6b: connectx4: Fix recv completion queue wrap-around
• 42c2196: mellanox: Rename ConnectX4 -> ConnectX (because it works with 5+ too)

Max Rottenkolber (150):

• 77e548c: Squashed 'lib/ljsyscall/' changes from d78235c..e587f8c55
• 09b29d9: ljsyscall: add some XDP constants
• 3f82010: ljsyscall: fix a bug in getsockopt when querying for structs
• 1e176cb: ljsyscall: lookup strflags in linux bpf(2) wrappers
• 573a29b: ljsyscall: add BPF_MAP type XSKMAP
• 63f0cfe: apps.test.synth: add packets option
• f535525: apps.test.synth: emit all given packets
• 6174563: core.packet: expose constants for packet alignment and headroom
• 984d4a4: apps.xdp: driver for AF_XDP sockets
• e056fa1: Remove src/xdp.snabb (accidentally checked in xdp proof of concept)
• ee4e42d: core.packet: expose free_internal
• b677292: apps.xdp: fix free stats accounting, cleanup fill/completion mngmnt
• 1b3593f: apps.xdp: add duplex test case, fix random_v4_packets
• 22d754c: apps.xdp: cleanup bpf_ld_imm64/LD_MAP_FD construction
• 4e31856: apps.xdp: generate more descriptive errors on bind(2) failure
• e53f314: apps.xdp: fixup comment about engine breath stimulation
• d225e75: apps.xdp: make num_chunks configurable in snabb_enable_xdp(opt)
• d44fb1d: apps.xdp: do not leak packets in XDP:stop()
• f8cb80c: apps.xdp: munmap(2) rings in XDP:stop() (do not leak rings mappings)
• e06c82f: apps.intel_avf: use new PCI acccess API (fix integration breakage)
• 7722f0d: apps.intel_avf: free resources on shutdown (fix leak)
• 6c330ec: lj_syscall: add BPF_OBJ_* syscall wrappers, extend bpf_attr union
• ea955a8: apps.xdp: allow sharing interfaces between Snabb processes
• 01f11e9: pflua: do not cdef struct bpf_ins (collides with ljsyscall)
• ae35674: core.app: add clearvmprofiles() function
• 1124b80: lib.timers.ingress_drop_monitor: clear vmprofiles on flush
• 8f2a216: lib.timers.ingress_drop_monitor: reset last_value on drop-free period
• 63968f6: raptorjit: clear all hotcounts every second
• dfadac8: raptorjit: amend prev. commit, also clear trace exit hotcounts
• b7c50cb: raptorjit: amend prev. commit, do not clear SNAPCOUNT_DONE
• bd84224: apps.xdp: expose driver reference
• 1a9eba5: lib.scheduling: add enable_xdp option
• 18cc365: apps.xdp: kick tx queue if not empty
• 19072b4: lib.yang.data: reorder translated members before printing
• ae4f734: snabb config: use consistency checker in command parsing
• 05b495e: lib.yang.path_data: expand choices in consistency checker
• e96ae1a: lib.yang.data: rework 19072b4 (remove order parameter) add tests
• 95d3a74: lib.yang: remove leftover debug print
• 3fdf926: pflua: fix typo bug in native codegen (xor/or emission)
• 07acce2: pflua: make regalloc architecture independent
• cd4d773: apps.xdp.bpf: add basic disassembler, endiannes opcodes
• 48ff71c: apps.xdp.pf_ebpf_codegen: add eBPF backend for pflua
• c158ac8: apps.xdp: add filter option (offload non-matching packets to kernel)
• 6b5851a: lib.numa: gracefully handle failure to read CPU performance governor
• 16c99ed: lib.ptree: cleanup obsolete aggregated stats
• 0068df6: arch/checksum: fix bug in carry propagation
• 3461a09: apps.xdp: only run rxtx_match selftest if SNABB_XDP_NQUEUES=1
• 00cac36: apps.xdp: debug stop()
• 8d4ee13: apps.xdp: mark stop() NYI / not supported
• 2bcc644: lwaftr --xdp sketch
• dca3004: snabb-softwire-v2: add default for leaf error-rate-limiting/packets
• 8c2000b: apps/ipv4,6/reassemble,fragment: handle padded packets
• 561b52a: lwaftr/setup: fix interfaces for bump-in-the-wire Interfaces used in bump-in-the-wire configurations were swapped, possibly during a prior refactor. Undo this.
• 1cdc635: snabb-softwire-v2.yang: documentation edits
• 6bcce09: lib.numa: gracefully handle failure to read CPU performance governor
• 191581a: apps.lwaftr.ndp: send NS via solicited node multicast
• 4817ae8: lwaftr: support vlans in XDP mode
• 004587b: lwaftr: automate linux interface setup for XDP
• bce8579: snabb-softwire-v2: allow more than two queues
• 909dca7: lib.protocol.ethernet: add ptoi used by connectx driver
• 6bca98b: apps.mellanox.connectx: fixup lib.hardware.pci usage
• 832ea9d: apps.mellanox.connectx_test: traceprof is now part of RaptorJIT
• d58ce74: apps.mellanox.connectx_test: get PCI addresses from environment
• 6036de6: lib.protocol.etherenet: do endian conversion in ptoi
• bf84158: apps.connectx: selftest fixups
• fc7dfd2: apps.mellanox.connectx: support RSS + MAC/VLAN
• 55ea547: Revert "apps.mellanox.connectx: support RSS + MAC/VLAN"
• 680355e: apps.mellanox.connext: fix data segment address translation in sq:transmit
• 740b63c: Revert "Revert "apps.mellanox.connectx: support RSS + MAC/VLAN""
• 22d2619: apps.mellanox.connectx_test: test MACVLAN+RSS
• 4b0487d: apps.mellanox.connectx: invert MAC/VLAN hierarchy in macvlan_rqlist
• 704126c: apps.mellanox.connectx: switch multicast packets when using MAC+VLAN
• 6d18bea: apps.mellanox.connectx_test: exercise L3-only RSS
• d4a1083: lwaftr: keep full config in worker setup
• 907fb77: lwaftr: add mellanox.connectx driver support
• 4c84dd0: apps.intel_avf: add multiqueue/rss support
• 8c151ea: lwaftr: add support for intel_avf driver
• e3dd48f: apps.intel_avf: add VLAN filter/stripping/insertion support
• dbc4f34: lwaftr/lwutil/setup: refactor, add num_queues, is_lowest_queue
• 448d626: lib.hardware.pci: add Intel X710 device info
• d54e29d: lwaftr: support auto-config of AVF-compatble NICs
• 80557cf: intel_avf: move bits() constructor out of fast-path
• be7dc08: program.lwaftr.setup: remove bogus stale require
• 263c4e2: lwaftr.setup: fix error reporting in cmd util
• 6df64ca: core.lib: fix error detection in writefile
• 720235f: lwaftr.setup: print more useful error message... when unable to write sriov_numvfs
• fa553a6: apps.intel_avf: move device stats to pci/
• 85428ea: snabb top: handle devices that do not specify speed
• 89c48fc: lib.ptree: start manager before setting initial configuration
• 4379d1c: snabb-softwire-v2.yang: fix descriptions of device leaves
• 98e97b4: Revert "snabb-softwire-v2.yang: fix descriptions of device leaves"
• 75a3373: Revert "lwaftr: keep full config in worker setup"
• bff9d0f: lib.numa: gracefully handle unset PATH in assert_irqbalanced_disabled
• b97ccc2: Revert "snabb-softwire-v2: allow more than two queues"
• 21013be: Revert "snabb-softwire-v2.yang: documentation edits"
• f903e0f: Revert "snabb-softwire-v2: add default for leaf"
• 0288de4: lib.yang.data: parse defaults of nested leaves
• eea42d2: lib.numa: do not migrate pages from preferred node
• 1588452: lib.protocol.ethernet: fix truncation bug in ptoi
• a6734ae: lwaftr: qualify shared_next_mac_key with vlan
• c1c4854: core.packet: fix account_free physical capacity formula
• 8c44e61: apps.intel_avf: fix IRQ setup for additional queues
• 18fc9f5: lwaftr: fix bugs introduced in 75a3373
• 3c669a9: lwaftr: create snabb-softwire-v3
• faeb6ff: snabb-softwire-v3: fix mistakes in leaf descr.
• 2dddff6: snabb-softwire-v3: Add default value for error-rate-limiting/packets
• eff1a9a: snabb-softwire-v3: allow more than two queues
• 1f8ee81: snabb-softwire-v3: external-interface/device -> external-device
• d75f079: lwaftr: use snabb-softwire-v3 schema
• e06c198: apps.ipv4.arp: add request/reply counters
• e3b400a: apps.lwaftr.ndp: add NS/NA counters
• 6f5e905: apps.ipv{4,6}.echo: add echo I/O counters
• bd914c1: lwaftr/snabb-softwire-v3: state additions and fixes
• 37cfb36: apps.intel_avf: add option to add additional macs
• 2bfba71: apps.intel_avf: update README.md
• b112c94: lwaftr: configure internal-interface multicast MAC address for intel_avf
• d3a30c6: lib.protocol.ethernet: remove ptoi apps.connectx: use lib.macaddress instead of ptoi
• ee37d77: apps.intel_avf: add multiqueue/rss support
• 451aeec: apps.intel_avf: add VLAN filter/stripping/insertion support
• ce533cf: lib.hardware.pci: add Intel X710 device info
• 3a39748: intel_avf: move bits() constructor out of fast-path
• b2ac1c3: apps.intel_avf: move device stats to pci/
• 7566bcf: snabb top: handle devices that do not specify speed
• dae1061: apps.intel_avf: fix IRQ setup for additional queues
• 5ccf479: apps.intel_avf: add option to add additional macs
• d22af4f: apps.intel_avf: update README.md
• dc40426: apps.test.synth: add packets option
• 355051b: intel_avf: truncate CRC from received packets
• d3fd3e7: intel_avf: more robust HENA configuration
• 991acd9: intel_avf: fix tests, separate RSS tests, add VLAN, multicast tests
• 6845091: intel_avf: add explicit VLAN test case
• 9c83c25: intel_avf: fix rx/tx packet lengths
• c1220be: apps.mellanox: add ConnectX 6 type
• 5b68b01: apps.mellanox.connectx_test: add multicast and checks
• cc59d64: apps.mellanox.connectx: implement stop()
• ad4c941: apps.mellanox.connectx_test: exercise stop()...
• d36174e: apps.mellanox.connectx_test: add match test
• bf1322e: apps.mellanox: add README
• 22f950c: lib.hardware.pci: add mellanox MT2892 Family [ConnectX-6 Dx] model
• ffb0ba9: apps.xdp: only run rxtx_match selftest if SNABB_XDP_NQUEUES=1
• 1a5f7c4: apps.xdp: mark stop() NYI / not supported
• 7b0d35b: lwAFTR 2022.01.13
• 1531f37: apps.ipv4.arp: fix selftest
• 3f3c40c: lwaftr: update end-to-end test data
• 247ef95: core.packet: revert globalization of newly exported variables and functions
• 5f045e7: core.app: setvmprofile unconditionally
• 52048a9: Fixup previous commit: remove unused profile option in lib.ptree.worker
• 862af2c: Fix regression in config-migrations
• 992a845: lwaftr migrate-configuration: support snabb-softwire-v2, fix bugs
• 6bd9e7a: snabbnfv: do not load ingress_drop_monitor