Skip to content

Snabb 2022.07 “Parmigiano”
Compare
Choose a tag to compare
@eugeneia eugeneia released this 04 Jul 10:29
· 393 commits to master since this release
v2022.07
11154da

This release upstreams the multi-core scalable IPFIX probe developed by @alexandergall at SWITCH, “Snabbflow”. Additionally, we have extended the core.app interface to support the new tick and per-link push methods. Other notable changes include improved inter-process link scaling, and a new benchmark for our Connect-X driver to be used in CI. As usual this release also comes with various minor improvements and bug fixes.

What's Changed

  • Upstream ipfix / snabbflow into max-next by @eugeneia in #1479
  • Use MPMC bounded queue for group freelist by @eugeneia in #1472
  • core.app 3.0: tick method, advanced app linking features by @eugeneia in #1475
  • map_pci_memory: workaround for kernel race condition by @eugeneia in #1476
  • lwaftr bench: build pcaps in generate-configuration, print summary by @eugeneia in #1477
  • Collected changes for July 2022 release: Parmigiano by @eugeneia in #1478
  • apps.mellanox.connectx: benchmark matrix by @eugeneia in #1469
  • ConnectX: various driver enhancements by @eugeneia in #1473
  • default.nix: remove obsolete brute-force shebang patching (breaks wit… by @eugeneia in #1474
  • Fixes for lwAFTR in many-worker configurations by @eugeneia in #1480

Full Changelog: v2022.01.1...v2022.07

Contributors

Alexander Gall (105):

  • 0f5068d: lj_trace.c: unpatch blacklisted bytecodes when flushing traces
  • 5df67e8: lj_trace.c: Add penalty to trace abort message
  • b19d442: intel_mp: fix range of array registers for i210
  • 51214ea: intel_mp: add sanity checks for the number of rx/tx descriptors
  • d4a9871: intel_mp: add queue enable counters, fix rxdrop counter
  • fe709ed: lib.protocol.ipv4: avoid allocation for pseudo-header
  • d15cbff: socket.raw: don't use select()
  • 54780c2: ctable: fix power-of-2 issue
  • 3002614: Add optional callback to the CTable:resize() method
  • 0cda3d1: lib: disable tail-call optimization for endian conversions
  • 62d0e72: Avoid factor-of-2 headroom in ctable
  • 4c15c4e: Check bound of values in lib/lpm/lpm4_248.lua
  • d4f8914: lib.lpm4_248: implement search in pure Lua
  • 30b1e6f: Flush JIT traces after ctable resize
  • d59aaff: Export flow before resetting fields
  • 5ec2a64: Properly count records for IPFIX and NetFlow v9
  • f4e93cd: Send full-sized export packets
  • 556bc58: Get flow octet counter from IP header
  • 246655c: Add tcpControlBits to flow sets, factorize transport processing
  • 132f5b9: Allow multiple files as input for make_ipfix_element_map()
  • e2587ae: Delegate exporting packets to FlowSets
  • f40c1b1: Add counters
  • 605dac0: Add sub-program to print stats for running ipfix processes
  • a25d353: Add take_all() method to token bucket
  • e0d93bc: Make table scan time configurable
  • 30b7c4f: Support VLAN-tagged traffic, IPv6 extension headers
  • b36e45b: Support PEN, strings and octetArray in templates
  • d72d42d: Add optional state variables to flow cache value
  • 46263a7: Add optional template-specific stats counters
  • 57b71a0: Refactor template definitions
  • 045f48e: Add templates for HTTP
  • 2575d94: Add templates for DNS
  • 9e6d48b: Add options to ipfix probe program, support routed tap interface
  • 463022d: Add extended template
  • 7f6c241: Switch to packet-metadata module from RSS
  • 24a5063: Allow multiple IPFIX instances in the same process
  • 8178577: Add probe_rss ipfix sub-program
  • 84b486a: Disable map logging in extended template by default
  • 7a9e3c8: Add SNMP Interface MIB for PCI devices
  • 5b9e56e: program.ipfix: redesign RSS framework
  • a15f4f3: program.ipfix: support hardware RSS in probe
  • 2c05b4c: program.ipfix: fix stats sub-program for redesign
  • 95ff0ca: program.ipfix: integrate mellanox driver
  • 5151c96: program.ipfix: include PCI stats from control app
  • 560e5b1: program.ipfix: increase receive buffers for mellanox driver
  • a9e9f7f: program.ipfix: adapt to interlink semantics of v2018.06
  • 28cca93: apps.ipfix: tune log messages
  • 84cbf0d: apps.ipfix: fix selftests
  • a2097f3: core.app: skip array indices when traversing input links
  • 395a4f8: core.app: assert that links are connected only once
  • 2c7929c: core.app: support link-specific push methods
  • 565d7e6: apps.basic: apply link push methods to basic apps
  • ea076d8: VlanMux: optimize tag pop/push
  • bc36ddb: VlanMux: apply specialized push methods
  • 1cfadb3: core.app: only call push() if the link is not empty
  • 5fb9816: Add housekeeping() method to apps
  • 233c894: Adapt to new versions of logger and token bucket
  • be1f65c: apps.vhost: suppress -Wstringop-truncation
  • 33eefac: lib.hardware.pci: add new device to supported Mellanox cards
  • b655643: apps.ipfix, apps.rss: adapt to specialized push link semantics
  • 40373df: program.iptix.probe_rss: add JIT configuration for RSS workers
  • 14e32eb: apps.rss: mitigate unbiased branches in metadata handling
  • c113670: apps.rss: explicitely unroll the loop over packet queues
  • 744d6d5: apps.rss.rss: Preserve order in classes_active, fix explicite loop unrolling
  • 3647611: program.ipfix.probe_rss: make receive queue size configurable for mellanox
  • 72f5482: program.ipfix.stats: use global counters when per-queue stats are not available
  • c24d55c: lib.interlink: increase queue size to allow bursts for ipfix
  • b279790: Make be1f65c conditional for GCC >= 8
  • 9bfdaae: program.ipfix: add IF MIB for routed tap interfaces
  • 670c411: apps.rss.rss: use specialized function to copy addresses for hashing
  • 54409f8: core.packet: increase maximum number of allocated packets
  • a1780b3: Fix transitions in CXQ state machine
  • 90c5762: map_pci_memory: wait for kernel driver to release the device
  • 87c1a0e: apps.mellanox.connectx4: unbind device in IO:new()
  • 08f25ca: apps.intel_mp.intel_mp: fix device unbinding for Intel driver
  • b47b38d: apps.intel_mp.intel_mp: remove disable bus master cleanup
  • 8f1a480: apps.ipfix.ipfix: move record expiration to housekeeping
  • da76c64: apps.mellanox.connectx4: fix mcast counters
  • 5f05dab: lib.hardware.pci.map_pci_memory: only wait if first call failed
  • 15ef82b: program.ipfix.probe_rss: use pools to assign collector address and port
  • afbe2bd: apps.rss.rss: Add VLAN match capability
  • 6d5bafa: program.ipfix: increase logging rate
  • eb659ba: program.ipfix.probe_rss: add observation domain to log output
  • 58ce70e: program.ipfix: disable date in log messages by default, fix READMEs
  • 432f38a: Add scan-protection feature
  • 541c0d8: interlink: fix shmem names in receiver/transmitter shutdown
  • b9c499e: program.ipfix.probe_rss: restart dead workers
  • 429fa21: program.ipfix: make maximum load-factors of hash tables configurable
  • 276a26d: ipfix: restrict flow-rate tracker to signature flows
  • aafd550: program.ipfix.probe_rss: scale scan-protection rates with number of processes
  • 3620245: program.ipfix.probe_rss: add CPU pool and NUMA binding
  • bc9211f: Untabify
  • 5a47fde: program.ipfix.probe_rss: make CPU-pinning configurable
  • 1f082b8: program.ipfix.probe_rss: increase logging rate limit for master process
  • 4af5345: program.ipfix: rename ConnectX4 -> ConnectX
  • 4296bfa: Don't export empty flows
  • fc407c2: Use proper flow start time after active timeout
  • caa11e1: lib.ctable: fix hugepages allocation size
  • 64a62f2: apps.mellanox: Make RX/TX flow control configurable, default to off/off
  • 4c1571f: apps.mellanox: Add per-queue drop stats
  • 69a15b9: apps.mellanox: Use lib.parse in ConnectX:new() and IO:new()
  • 3a5b5ae: apps.mellanox: Use consumer counter in cxq to simplify ownership handling
  • be7a343: connectx.lua: Add basic event handling
  • 7c269fa: map_pci_memory: wait for kernel driver to release the device
  • 1c7497c: lib.hardware.pci.map_pci_memory: only wait if first call failed

Andy Wingo (2):

  • 907ce81: Use binary search for all lookups
  • b457b6e: Add a type cache, in case that's important

Luke Gorrie (5):

  • b29666f: lj_trace.c: Clear penalty slot after successful trace
  • b4060ef: lj_jit.h: Increase HOTCOUNT_MAX and expand HotPenalty.val
  • 0a90764: fixup! lj_jit.h: Increase HOTCOUNT_MAX and expand HotPenalty.val
  • c685d8b: lj_trace.c: Reset hotcount table after flush
  • 5443940: lj_record.c: Relax heuristic for root trace meeting JIT loop

Max Rottenkolber (76):

  • bd5d4ec: wip
  • 2f6b983: wip 2
  • e301b53: mellanox benchmark numa
  • aaea69b: add Source software benchmark to rule it out as a bottleneck
  • 7634573: measure rx at sink
  • 7f967e4: apps.mellanox: packetblaster
  • 9adad52: mellanox benchmark: use packetblaster
  • 350aea9: apps.mellanox.connectx: cleanup and optimization
  • 3d86feb: mellanox benchmark: use stats counters instead of querying HCA
  • ae5121b: mellanox benchmark: cleanup output
  • b1cf848: mellanox benchmarks example scripts
  • 87207cf: apps.connectx: remove dead code from packetblaster
  • 0ab555f: mellanox benchmark: add --opt option to supply driver config options
  • 312b740: mellanox benchmark: add forwarding mode
  • 28f249a: mellanox benchmark: add --nmacs, --nvlans paremetrs, remove source-sofftest
  • 4fa1d01: mellanox benchmark: fix up previous commit
  • 6c69c04: mellanox benchmark: add sink mode, sink halts
  • 7195667: mellanox benchmark: IMIX source
  • f799af2: apps.interlink: add instrumented latency benchmark
  • fbc13ab: core.packet: factor packet reclaimation from group_fl into reclaim_step
  • a9c8436: apps.interlink: better instrumentation
  • 3b32a2f: core.packet: lock-free group freelist
  • e1a70b0: core.group_freelist: cache head_add/tail_remove, document
  • c0ab1b1: core.packet: better group freelist design
  • cdec562: apps.interlink: fix bugs in wait_test.snabb
  • d01df58: core.group_freelist: add missing load barrier
  • 99e2a6c: core.group_freelist: shrink queue
  • 2ec3a89: apps.interlink: fremove obsolete line wait_test.snabb
  • cab614d: apps.interlink: sort cores in wait_test.snabb
  • 328ba09: apps.mellanox.connectx_test: make sure links are up in basic_match
  • 3cd72c2: apps.mellanox: packetblaster
  • 60ce041: apps.mellanox: cleanup and optimization
  • 66ab5b1: apps.mellanox: remove dead code from packetblaster
  • 67850d1: Mellanox benchmark: let source linger for when TX > RX
  • ee2a703: apps.connectx: hide event prints behind debug_info flag
  • 84cb61c: core.packet: ensure freelist has enough space for group fl rebalancing
  • 616e6e2: core.group_freelist: enlarge group_freelist to hold up to 2 million packets
  • 7086845: default.nix: remove obsolete brute-force shebang patching (breaks with newer nix)
  • c45ae86: lib/luajit: add build artifacts to gitignore
  • a06db22: core.app: remove use_restart facilities
  • 840d07b: core.app: make breathe_pull/push_order local
  • ec7e080: core.app: add tick() app method support
  • add9e77: snabbmark: add micro-benchmark for tick() methods
  • e372868: core.app: link() 3.0
  • 0ee2090: snabbmark: add microbenchmark for push_link methods
  • b05b08d: src/README.md: document core.app 3.0 features
  • e9e977f: core.app: refactor tick() throttle
  • bc970a0: apps.test.match: fix selftest
  • 2895647: apps.lwaftr.V4V6: fix selftest (test_join)
  • c676980: apps.pcap.tap: fix selftest
  • 26bcea8: apps.rate_limiter: fix selftest
  • b6ea727: apps.packet_filter: fix selftest
  • 30f8546: core.app: fix bug in compute_breathe_order...
  • df9e77f: snabbvmx: fix lwaftr_app_check test setup
  • ed5c947: core.app: copy app class push_link to fresh table in instance
  • 6dd5418: lib.hardware.pci: cleanup previous commits
  • b9f5125: src/README.md: formatting
  • addfc10: apps.ipfix: fix selftest
  • 1021d2c: lib.ctable: replace MAX_MAX_DISPLACEMENT with configurable max_displacement_limit
  • fc3e377: Revert some changes from 0c4205b because they break tested behavior
  • d6a641c: lib.protocol.ipv4: add ipv4:pton_cidr
  • c23857c: lib.protocol.ipv4/6: remove bogus methods
  • 1e94811: apps.ipfix: use lib.poptrie
  • 45d1b07: apps.ipfix: fix previous poptrie usage for 32-bit ASNs
  • 657c884: lib.protocol: add ipv6:pton_cidr(), fix documentation of ipv4:pton_cidr()
  • c154ab1: apps.ipfix: support IPv6 prefix to AS mapping
  • 2447d17: lib.poptrie: support 32-bit leaves
  • c89289d: apps.ipfix: fix up previous commits
  • 9ad014b: lib.protocol: add dot1q header
  • a064124: apps.ipfix: exercise extended templates in selftest
  • 04227d7: program.ipfix: pfx_to_as is now split into pfx4/6_to_as
  • 798495b: lwaftr: softbench
  • 3d8ddcc: lib.numa: do NUMA migrations only for >1 node systems
  • cf73e1e: apps.mellanox: make per-queue counter creation configurable
  • d67f63d: lwaftr: updated ConnectX config defaults
  • 056d434: ljsyscall: fix bug in S.sysctl where we wrote at position ~= 0

Contributors

eugeneia and alexandergall
Assets 2
Loading