PFC pause is the mechanish used by PFC to perform flow control during a congestion. When a switch determines there is a congestion happening in the downstream network it starts sending a PFC pause packet to let its upstream neighbor know about the congestion and the high priority queue which is configured to be lossless to slow down or pause the transmit. This test plan will validate performance of a lossless traffic flow under congestion.
PFC pause frames can prevent a switch queue from sending any data in a time interval. The PFC frame has a 8-bit field called 'class-enable vector' or 'priority enable ventor' where the bit N tells which priority N it pauses. if this bit is set to 1, it means pause. The PFC frame has 8 pause durations fields, one for each priority. For a priority to pause, the PFC frame also needs to specify its pause duration. The pause duration for each priority is a 2-byte value that expresses time as a number of quanta, where each quanta represents the time needed to transmit 512 bits at the current network speed. For example, the value of 65535 at a 40 Gbps link means 65535 * 512bits / 40Gbps = 838.84 microseconds. Therefore, to fully block a switch queue, we need to generate PFC pause frames fast enough. For example, to block 40G link, we need to generate a PFC pause frame with 65535 quantas in every 838.84 microseconds. A pause duration of zero quanta has the special meaning of unpausing a priority.
In this test plan we will work with two PFC lossless priorities: 3 and 4. Note that only the lossless priorities can react to or generate PFC frames. In other words, PFC frames should not have any impact on traffic on lossy priorities. Packets with Differentiated Services Code Point (DSCP) 3 and 4 are mapped to priority 3 and 4, respectively.
After a device sends a PFC pause frame, it will take some time for the pause frame to arrive at the upstream device and take effect. To absorb packets arriving during this period of time, we need to reserve some buffer space. This reserved buffer space is called "PFC headroom". The PFC Headroom value depends on several factors, including the response time of the sender to react to PFC pause frames upon receiving one - known as the PFC response time.
The IXIA chassis allows PFC response times to be configured manually i.e. create an additional delay between the time an IXIA port receives a PFC pause frame with a "pause quanta", and the IXIA port correspondingly responds to the PFC pause frame. With the “PFC pause delay” feature enabled, the IXIA port will wait for N quanta before stopping traffic. The pause time by itself is unchanged. The “PFC pause delay” effectively increases the PFC pause response time. This can be configured by setting the pfc.pfc_delay parameter (corresponding to N in our example).
Put another way, if the IXIA port is configured with a “PFC Delay” of N and it receives a PFC pause frame with Y quanta, the port will wait for N quanta before stopping traffic for Y quanta.
For a "PFC pause delay" value that is sufficiently large, the Tx IXIA port takes some additional time to respond to PFC pause frames generated by the ingress switch port, and slow down the traffic flow. In the mean time, the Tx IXIA port continues to send traffic at line rate to a route that is already congested. As a result, there is a lot of in-flight packets to hold in the switch's headroom buffer which it fails to hold (due to limited headroom buffer space) and overflows, causing a loss in traffic to the destination port.
The testbed consists of two IXIA ports and a SONiC device under test (DUT) as follows. Both IXIA ports should have the same bandwidth capacity. To reduce the configuration complexity, we recommond configuring the switch as a Top of Rack (ToR) / Tier 0 (T0) switch and binding two switch interfaces to the Vlan.
_________
| |
IXIA tx port ------ | DUT |------ IXIA rx port
|_________|
In this experiment, we need to create three traffic items:
- Test data traffic: Data packets sent from the IXIA tx port to the IXIA rx port at the lossless priorities (e.g., e.g., 3, 4 or both 3 and 4) that we want to test. Note that the packets should be marked with the correct DSCP value (e.g., DSCP 3 for priority 3). The traffic demand should be 50% of the line rate.
- Background data traffic: Data packets sent from the IXIA tx port to the IXIA rx port at the other priorities that we want to test (e.g., 0-2 and 5-7). The traffic demand should be 50% of the line rate.
- PFC pause storm: Persistent PFC pause frames from the IXIA rx port to the IXIA tx port. The priorities of PFC pause frames should be same as those of test data traffic. And the inter-frame transmission interval should be smaller than per-frame pause duration. This experiment needs the following five steps:
- Start the PFC pause storm to fully block the test priorities at the switch.
- After a fixed duration (e.g., 1 second), start test data traffic and background data traffic simultaneously.
- After a fixed duration (e.g., 5 seconds), stop test data traffic and background data traffic.
- Check if the IXIA rx port receives (1) all the sent frames of background data traffic (which should not be impacted by PFC), and (2) zero frames of test data traffic (PFC pause storm was triggered after traffic flow was started).
- Then, check if (1) the total number of transmitted bytes of test data traffic is smaller than the switch shared buffer size for no pfc response delay added, or very small pfc response delay values,
- And also check if the total number of transmitted bytes of test data traffic is (2) greater than the switch shared buffer size for higher pfc response delay values since the headroom limit would be exceeded. The exact values for the pfc delay response times is dependent on the switch.
- Lastly, check if (1) the number of dropped packets on the ingress port (for the specific priority(s)) equals zero when the PFC headroom is not exceeded and (2) is greater than zero when the PFC headroom is exceeded.
- Stop PFC pause storm