-
Notifications
You must be signed in to change notification settings - Fork 0
/
setup.txt
158 lines (126 loc) · 6.45 KB
/
setup.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
####### PTP setup #######
Guides:
https://docs.mellanox.com/display/ONYXv381174/Precision+Time+Protocol
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/system_administrators_guide/ch-configuring_ptp_using_ptp4l
Mellanox PTP: https://community.mellanox.com/s/article/Running-Linux-PTP-with-ConnectX-4-ConnectX-5-ConnectX-6
git clone git://git.code.sf.net/p/linuxptp/code linuxptp
make
sudo make install
Run:
sudo ptp4l -m -P -i enp59s0 (for master)
sudo ptp4l -m -P -i enp59s0 -s (for slave)
Run as a service on startup:
Edit /lib/systemd/system/ptp4l.service and set proper interface (-i enp59s0)
sudo systemctl daemon-reload
sudo systemctl start ptp4l
sudo systemctl status ptp4l
sudo tail -f /var/log/syslog (logs)
####### On the switch ########
Need to enable multicast on the switch so that PTP master/slaves can find each other.
Help: https://docs.mellanox.com/pages/viewpage.action?pageId=34250299
ssh sw100 (RivVeylsEg4)
Ports Eth1/17 and Eth1/18 for yeti-00 and 01 CX-6s
start: "enable", "config terminal"
setup vlan:
vlan 3
name ptp-ayelam
add ports to vlan:
interface ethernet 1/17 switchport access vlan 3
interface ethernet 1/18 switchport access vlan 3
interface vlan 3 ip address 10.0.5.0 255.0.0.0
enable multicast on vlan:
ip routing
ip multicast-routing
protocol pim
interface vlan 3 ip pim sparse-mode
interface vlan 3 ip igmp version 3
ip igmp snooping
vlan 3 ip igmp snooping
show ip igmp interface (should show vlan 3)
show ip igmp interface (will show the ptp groups once ptp4l is started on the hosts)
####### DPDK setup ######
sudo -H pip3 install meson
sudo apt-get install -y python3-pyelftools python-pyelftool
git clone https://github.com/DPDK/dpdk
cd dpdk
meson -Dexamples=all build
ninja -C build
####### Pktgen Setup #####
Install dpdk to system
chmod -R a+wx dpdk/ (for sudo access to your nfs folder)
sudo ninja -C build install
sudo ldconfig
export PKG_CONFIG_PATH=/usr/local/lib/x86_64-linux-gnu/pkgconfig
Build Pktgen
sudo apt-get install libpcap-dev libnuma-dev
sudo apt install lua5.3
git clone https://github.com/pktgen/Pktgen-DPDK
make
or, make buildtdma
or, make buildlua
sudo ./Builddir/app/pktgen -l 0,2,4,8 -n 4 -a 0000:3b:00.0,tx_pp=500,txq_inline=256,txqs_min_inline=4 -- -m "[2:4].0" -T
# sudo ./build/app/dpdk-testpmd -l 0,2,4,8 -n 4 -a 0000:3b:00.0,tx_pp=500 -- -i --nb-cores=2 --txq=2 --rxq=2
####### Packet Pacing Setup #########
https://doc.dpdk.org/guides/nics/mlx5.html
https://github.com/DPDK/dpdk/blob/main/drivers/net/mlx5/mlx5.c
Patches that introduced and discussed this feature:
https://patches.dpdk.org/project/dpdk/patch/1594298216-3746-1-git-send-email-viacheslavo@mellanox.com/
https://inbox.dpdk.org/dev/7707630.AmxAOCipGc@thomas/T/
All patches: https://patches.dpdk.org/project/dpdk/list/?series=&submitter=&state=*&q=packet+pacing&archive=both&delegate=
Patch descriptions:
This patch creates the special completion queue providing
reference completions to schedule packet send from
other transmitting queues.
The dedicated Rearm Queue is needed to fire the work requests to
the Clock Queue in realtime. The Clock Queue should never stop,
otherwise the clock synchronization mignt be broken and packet
send scheduling would fail. The Rearm Queue uses cross channel
SEND_EN/WAIT operations to provides the requests to the
CLock Queue in robust way.
This patch allocates the Packet Pacing context from the kernel,
configures one according to requested pace send scheduling
granularuty and assigns to Clock Queue.
rate = NS_PER_S / sh->txpp.tick;
MLX5_SET(set_pp_rate_limit_context, &pp, rate_mode, sh->txpp.test ? MLX5_DATA_RATE : MLX5_WQE_RATE); // bps or pps
tx_pp <== send scheduling granularity. (so 1 packet every 500ns, can't go more.. can send 6.25kB at 100Gbps)
This patch creates the special completion queue providing
reference completions to schedule packet send from
other transmitting queues.
The dedicated Rearm Queue is needed to fire the work requests to
the Clock Queue in realtime. The Clock Queue should never stop,
otherwise the clock synchronization mignt be broken and packet
send scheduling would fail. The Rearm Queue uses cross channel
SEND_EN/WAIT operations to provides the requests to the
CLock Queue in robust way.
(all this stuff for precise clock synchronization)
Enabling tx_pp flag: sudo ./build/app/dpdk-testpmd -l 0,2,4,8 -n 4 -a 0000:3b:00.0,tx_pp=500 -- -i --nb-cores=2 --txq=2 --rxq=2
But I hit this error first: "mlx5_net: WQE rate mode is required for packet pacing"
With some guidance from https://www.mail-archive.com/users@dpdk.org/msg05378.html, was able to figure out the issue by some serious guesswork:
sudo mlxconfig -d 0000:3b:00.0 set REAL_TIME_CLOCK_ENABLE=1 (and reboot)
http://patches.dpdk.org/project/dpdk/patch/20210307100251.22538-2-viacheslavo@nvidia.com/#128900
Patch that explains various timestamp formats and how to configure them on queues
####### Latency experiments #########
Pktgen:
make buildtdma
sudo ./Builddir/app/pktgen -l 0,2,4,8 -n 4 -a 0000:3b:00.0,tx_pp=500 -- -m "[2:4].0" -T -f ../lua/yeti-00-init.lua -G
sudo ./Builddir/app/pktgen -l 0,2,4,8 -n 4 -a 0000:3b:00.0,tx_pp=500 -- -m "[2:4].0" -T -f ../lua/yeti-01-init.lua -G
bash latexp.sh
Adjusted latencies:
cat latencies.dat | \
awk '{ if (NR == 1) { printf "%s,%s\n",$1,"Interval"; } else { printf "%lu,%lu\n",$1,$1-last; last=$1; } }' \
> data/int_txpp_1mus_burst_1_simple.dat
Latency plots:
python plot.py -z cdf -yc "Latency" -xl "Latency (micro-sec)" \
-d data/lat_notxpp_burst_1_poisson.dat -l "poisson" \
-d data/lat_notxpp_burst_1_simple.dat -l "simple" \
-nm --xmax 5000 -of png -o plots/latencies_no_txpp.png
-d data/lat_notxpp_burst_32_poisson.dat -l "32,poisson" \
-d data/lat_notxpp_burst_32_simple.dat -l "32,simple" \
python plot.py -z cdf -xl "Delay (micro sec)" \
-dyc data/lat_txpp_1mus_burst_1_simple_head_250.dat "Latency" -l "1 mus pacing" \
-dyc data/lat_txpp_10mus_burst_1_simple_head_250.dat "Latency" -l "10 mus pacing" \
-nm --xmax 4000 -of png -o plots/latencies_txpp.png
python plot.py -z cdf -xl "Interval (micro sec)" \
-dyc data/int_txpp_1mus_burst_1_simple_head_250.dat "Interval" -l "1 mus pacing" \
-dyc data/int_txpp_10mus_burst_1_simple_head_250.dat "Interval" -l "10 mus pacing" \
-nm -of png -o plots/interval_txpp.png