Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Minor changes for snappi_tests for T2-ixia, change udp port calculations. #15906

Merged
merged 4 commits into from
Dec 11, 2024

Conversation

rraghav-cisco
Copy link
Contributor

@rraghav-cisco rraghav-cisco commented Dec 5, 2024

Description of PR

Currently the traffic for RDMA tests use randomly selected udp ports. This sometimes ends up sending traffic in a non-load-balanced way, and causes the tests to fail.

In this PR:

  1. For multiple scripts that use udp: We serialize the UDP port numbers instead of randomly picking the numbers. This allows for more uniform load-balancing spread, and allows tests to pass more.
  2. in multi_lossless helper: We reduce the background traffic rate, and raise the rate for lossless traffic, thus keeping the overall rate still same. This allows for accounting the speed of backplane.

Type of change

  • Bug fix
  • Testbed and Framework(new/improvement)
  • Test case(new/improvement)

Back port request

  • 202012
  • 202205
  • 202305
  • 202311
  • 202405

@sdszhang
Copy link
Contributor

sdszhang commented Dec 5, 2024

@rraghav-cisco can you separate this PR into two? one for the fixture change, another one for the UDP port and flow rate change.

@rraghav-cisco rraghav-cisco marked this pull request as ready for review December 6, 2024 01:21
@sdszhang
Copy link
Contributor

sdszhang commented Dec 6, 2024

@selldinesh @amitpawar12 for review

@rraghav-cisco rraghav-cisco changed the title Committing final code changes, minor ones for T2-ixia. Minor changes for snappi_tests for T2-ixia, change udp port calculations. Dec 6, 2024
@rraghav-cisco
Copy link
Contributor Author

@auspham , @sdszhang : Pls let me know what the failure in pre-checks mean. Thanks.

@auspham
Copy link
Contributor

auspham commented Dec 7, 2024

@auspham , @sdszhang : Pls let me know what the failure in pre-checks mean. Thanks.

retriggered for you

Copy link
Contributor

@sdszhang sdszhang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@sdszhang
Copy link
Contributor

sdszhang commented Dec 9, 2024

@rraghav-cisco can you update the description with more detail on why these changes are needed? and test result as well?

@rraghav-cisco
Copy link
Contributor Author

results:

<style> </style>
False snappi_tests/multidut/ecn/test_multidut_dequeue_ecn_with_snappi.py test_dequeue_ecn[multidut_port_info0] skipped ECN tests are not supported on Cisco switches yet. 62.713
False snappi_tests/multidut/ecn/test_multidut_dequeue_ecn_with_snappi.py test_dequeue_ecn[multidut_port_info1] skipped ECN tests are not supported on Cisco switches yet. 302.22
False snappi_tests/multidut/ecn/test_multidut_ecn_marking_with_snappi.py test_ecn_marking_port_toggle[multidut_port_info0] failure json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) 77.15
False snappi_tests/multidut/ecn/test_multidut_ecn_marking_with_snappi.py test_ecn_marking_port_toggle[multidut_port_info1] success   310.16
True snappi_tests/multidut/ecn/test_multidut_ecn_marking_with_snappi.py test_ecn_marking_lossless_prio[multidut_port_info0-test_flow_percent0] error failed on teardown with "Failed: Processes "['analyze_logs--']" failed with exit code "1" Exception: match: 1 expected_match: 0 expected_missing_match: 0 Match Messages: 2024 Dec  6 06:51:14.180812 yy39top-lc4 ERR monit[889]: 'routeCheck' status failed (255) -- Failure results: {{#12    "asic0": {#12        "missed_INTF_TABLE_entries": [#12            "fc00:10::1/128"#12        ]#12    }#12}}#012Failed. Look at reported mismatches above#012add: {#12    "asic0": [#12        "0.0.0.0/0",#12        "::/0"#12    ],#12    "asic1": [],#12    "asic2": []#12}#012del: {#12    "asic0": [#12        "20.1.1.2/31",#12        "20.1.1.2/32"#12    ],#12    "asic1": [],#12    "asic2": []#12} Traceback: Traceback (most recent call last):   File "/data/tests/common/helpers/parallel.py", line 35, in run     Process.run(self)   File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run     self._target(*self._args, **self._kwargs)   File "/data/tests/common/helpers 142.33
False snappi_tests/multidut/ecn/test_multidut_ecn_marking_with_snappi.py test_ecn_marking_lossless_prio[multidut_port_info0-test_flow_percent1] success   146.18
True snappi_tests/multidut/ecn/test_multidut_ecn_marking_with_snappi.py test_ecn_marking_lossless_prio[multidut_port_info0-test_flow_percent2] error failed on teardown with "Failed: Processes "['analyze_logs--']" failed with exit code "1" Exception: match: 1 expected_match: 0 expected_missing_match: 0 Match Messages: 2024 Dec  6 06:56:14.306734 yy39top-lc4 ERR monit[889]: 'routeCheck' status failed (255) -- Failure results: {{#12    "asic0": {#12        "missed_INTF_TABLE_entries": [#12            "fc00:10::1/128"#12        ]#12    }#12}}#012Failed. Look at reported mismatches above#012add: {#12    "asic0": [],#12    "asic1": [],#12    "asic2": []#12}#012del: {#12    "asic0": [],#12    "asic1": [],#12    "asic2": []#12} Traceback: Traceback (most recent call last):   File "/data/tests/common/helpers/parallel.py", line 35, in run     Process.run(self)   File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run     self._target(*self._args, **self._kwargs)   File "/data/tests/common/helpers/parallel.py", line 245, in wrapper     target(*args, **kwargs)   File "/data/tests/common/plugins/loganalyze 124.83
False snappi_tests/multidut/ecn/test_multidut_ecn_marking_with_snappi.py test_ecn_marking_lossless_prio[multidut_port_info0-test_flow_percent3] success   127.94
True snappi_tests/multidut/ecn/test_multidut_ecn_marking_with_snappi.py test_ecn_marking_lossless_prio[multidut_port_info0-test_flow_percent4] error failed on teardown with "Failed: Processes "['analyze_logs--']" failed with exit code "1" Exception: match: 1 expected_match: 0 expected_missing_match: 0 Match Messages: 2024 Dec  6 07:01:14.433095 yy39top-lc4 ERR monit[889]: 'routeCheck' status failed (255) -- Failure results: {{#12    "asic0": {#12        "missed_INTF_TABLE_entries": [#12            "fc00:10::1/128"#12        ]#12    }#12}}#012Failed. Look at reported mismatches above#012add: {#12    "asic0": [],#12    "asic1": [],#12    "asic2": []#12}#012del: {#12    "asic0": [],#12    "asic1": [],#12    "asic2": []#12} Traceback: Traceback (most recent call last):   File "/data/tests/common/helpers/parallel.py", line 35, in run     Process.run(self)   File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run     self._target(*self._args, **self._kwargs)   File "/data/tests/common/helpers/parallel.py", line 245, in wrapper     target(*args, **kwargs)   File "/data/tests/common/plugins/loganalyze 130.37
False snappi_tests/multidut/ecn/test_multidut_ecn_marking_with_snappi.py test_ecn_marking_lossless_prio[multidut_port_info1-test_flow_percent0] success   158.04
True snappi_tests/multidut/ecn/test_multidut_ecn_marking_with_snappi.py test_ecn_marking_lossless_prio[multidut_port_info1-test_flow_percent1] error failed on teardown with "Failed: Processes "['analyze_logs--']" failed with exit code "1" Exception: match: 1 expected_match: 0 expected_missing_match: 0 Match Messages: 2024 Dec  6 07:06:14.544941 yy39top-lc4 ERR monit[889]: 'routeCheck' status failed (255) -- Failure results: {{#12    "asic0": {#12        "missed_INTF_TABLE_entries": [#12            "fc00:10::1/128"#12        ]#12    }#12}}#012Failed. Look at reported mismatches above#012add: {#12    "asic0": [],#12    "asic1": [],#12    "asic2": []#12}#012del: {#12    "asic0": [],#12    "asic1": [],#12    "asic2": []#12} Traceback: Traceback (most recent call last):   File "/data/tests/common/helpers/parallel.py", line 35, in run     Process.run(self)   File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run     self._target(*self._args, **self._kwargs)   File "/data/tests/common/helpers/parallel.py", line 245, in wrapper     target(*args, **kwargs)   File "/data/tests/common/plugins/loganalyze 138.67
False snappi_tests/multidut/ecn/test_multidut_ecn_marking_with_snappi.py test_ecn_marking_lossless_prio[multidut_port_info1-test_flow_percent2] success   152.44
True snappi_tests/multidut/ecn/test_multidut_ecn_marking_with_snappi.py test_ecn_marking_lossless_prio[multidut_port_info1-test_flow_percent3] error failed on teardown with "Failed: Processes "['analyze_logs--']" failed with exit code "1" Exception: match: 1 expected_match: 0 expected_missing_match: 0 Match Messages: 2024 Dec  6 07:11:14.663132 yy39top-lc4 ERR monit[889]: 'routeCheck' status failed (255) -- Failure results: {{#12    "asic0": {#12        "missed_INTF_TABLE_entries": [#12            "fc00:10::1/128"#12        ]#12    }#12}}#012Failed. Look at reported mismatches above#012add: {#12    "asic0": [],#12    "asic1": [],#12    "asic2": []#12}#012del: {#12    "asic0": [],#12    "asic1": [],#12    "asic2": []#12} Traceback: Traceback (most recent call last):   File "/data/tests/common/helpers/parallel.py", line 35, in run     Process.run(self)   File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run     self._target(*self._args, **self._kwargs)   File "/data/tests/common/helpers/parallel.py", line 245, in wrapper     target(*args, **kwargs)   File "/data/tests/common/plugins/loganalyze 139.58
False snappi_tests/multidut/ecn/test_multidut_ecn_marking_with_snappi.py test_ecn_marking_lossless_prio[multidut_port_info1-test_flow_percent4] success   173.16
False snappi_tests/multidut/ecn/test_multidut_red_accuracy_with_snappi.py test_red_accuracy[multidut_port_info0] skipped ECN tests are not supported on Cisco switches yet. 36.081
False snappi_tests/multidut/ecn/test_multidut_red_accuracy_with_snappi.py test_red_accuracy[multidut_port_info1] skipped ECN tests are not supported on Cisco switches yet. 306.12
False snappi_tests/multidut/pfc/test_lossless_response_to_external_pause_storms.py test_lossless_response_to_external_pause_storms_test[multidut_port_info0] success   233.72
False snappi_tests/multidut/pfc/test_lossless_response_to_external_pause_storms.py test_lossless_response_to_external_pause_storms_test[multidut_port_info1] success   178.52
True snappi_tests/multidut/pfc/test_lossless_response_to_throttling_pause_storms.py test_lossless_response_to_throttling_pause_storms[multidut_port_info0] error failed on teardown with "Failed: Processes "['analyze_logs--']" failed with exit code "1" Exception: match: 1 expected_match: 0 expected_missing_match: 0 Match Messages: 2024 Dec  6 07:28:16.717190 yy39top-lc4 ERR monit[889]: 'routeCheck' status failed (255) -- Failure results: {{#12    "asic0": {#12        "missed_INTF_TABLE_entries": [#12            "fc00:10::1/128"#12        ]#12    }#12}}#012Failed. Look at reported mismatches above#012add: {#12    "asic0": [],#12    "asic1": [],#12    "asic2": []#12}#012del: {#12    "asic0": [],#12    "asic1": [],#12    "asic2": []#12} Traceback: Traceback (most recent call last):   File "/data/tests/common/helpers/parallel.py", line 35, in run     Process.run(self)   File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run     self._target(*self._args, **self._kwargs)   File "/data/tests/common/helpers/parallel.py", line 245, in wrapper     target(*args, **kwargs)   File "/data/tests/common/plugins/loganalyze 243.59
True snappi_tests/multidut/pfc/test_lossless_response_to_throttling_pause_storms.py test_lossless_response_to_throttling_pause_storms[multidut_port_info1] error failed on teardown with "Failed: Processes "['analyze_logs--']" failed with exit code "1" Exception: match: 1 expected_match: 0 expected_missing_match: 0 Match Messages: 2024 Dec  6 07:33:16.804286 yy39top-lc4 ERR monit[889]: 'routeCheck' status failed (255) -- Failure results: {{#12    "asic0": {#12        "missed_INTF_TABLE_entries": [#12            "fc00:10::1/128"#12        ]#12    }#12}}#012Failed. Look at reported mismatches above#012add: {#12    "asic0": [],#12    "asic1": [],#12    "asic2": []#12}#012del: {#12    "asic0": [],#12    "asic1": [],#12    "asic2": []#12} Traceback: Traceback (most recent call last):   File "/data/tests/common/helpers/parallel.py", line 35, in run     Process.run(self)   File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run     self._target(*self._args, **self._kwargs)   File "/data/tests/common/helpers/parallel.py", line 245, in wrapper     target(*args, **kwargs)   File "/data/tests/common/plugins/loganalyze 187.59
True snappi_tests/multidut/pfc/test_m2o_fluctuating_lossless.py test_m2o_fluctuating_lossless[multidut_port_info0] error failed on teardown with "Failed: Processes "['analyze_logs--']" failed with exit code "1" Exception: match: 1 expected_match: 0 expected_missing_match: 0 Match Messages: 2024 Dec  6 07:38:16.906663 yy39top-lc4 ERR monit[889]: 'routeCheck' status failed (255) -- Failure results: {{#12    "asic0": {#12        "missed_INTF_TABLE_entries": [#12            "fc00:10::1/128"#12        ]#12    }#12}}#012Failed. Look at reported mismatches above#012add: {#12    "asic0": [],#12    "asic1": [],#12    "asic2": []#12}#012del: {#12    "asic0": [],#12    "asic1": [],#12    "asic2": []#12} Traceback: Traceback (most recent call last):   File "/data/tests/common/helpers/parallel.py", line 35, in run     Process.run(self)   File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run     self._target(*self._args, **self._kwargs)   File "/data/tests/common/helpers/parallel.py", line 245, in wrapper     target(*args, **kwargs)   File "/data/tests/common/plugins/loganalyze 245.73
False snappi_tests/multidut/pfc/test_m2o_fluctuating_lossless.py test_m2o_fluctuating_lossless[multidut_port_info1] success   195.78
True snappi_tests/multidut/pfc/test_m2o_oversubscribe_lossless.py test_m2o_oversubscribe_lossless[multidut_port_info0] error failed on teardown with "Failed: Processes "['analyze_logs--']" failed with exit code "1" Exception: match: 1 expected_match: 0 expected_missing_match: 0 Match Messages: 2024 Dec  6 07:43:17.016411 yy39top-lc4 ERR monit[889]: 'routeCheck' status failed (255) -- Failure results: {{#12    "asic0": {#12        "missed_INTF_TABLE_entries": [#12            "fc00:10::1/128"#12        ]#12    }#12}}#012Failed. Look at reported mismatches above#012add: {#12    "asic0": [],#12    "asic1": [],#12    "asic2": []#12}#012del: {#12    "asic0": [],#12    "asic1": [],#12    "asic2": []#12} Traceback: Traceback (most recent call last):   File "/data/tests/common/helpers/parallel.py", line 35, in run     Process.run(self)   File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run     self._target(*self._args, **self._kwargs)   File "/data/tests/common/helpers/parallel.py", line 245, in wrapper     target(*args, **kwargs)   File "/data/tests/common/plugins/loganalyze 253.45
True snappi_tests/multidut/pfc/test_m2o_oversubscribe_lossless.py test_m2o_oversubscribe_lossless[multidut_port_info1] error failed on teardown with "Failed: Processes "['analyze_logs--']" failed with exit code "1" Exception: match: 1 expected_match: 0 expected_missing_match: 0 Match Messages: 2024 Dec  6 07:48:17.134079 yy39top-lc4 ERR monit[889]: 'routeCheck' status failed (255) -- Failure results: {{#12    "asic0": {#12        "missed_INTF_TABLE_entries": [#12            "fc00:10::1/128"#12        ]#12    }#12}}#012Failed. Look at reported mismatches above#012add: {#12    "asic0": [],#12    "asic1": [],#12    "asic2": []#12}#012del: {#12    "asic0": [],#12    "asic1": [],#12    "asic2": []#12} Traceback: Traceback (most recent call last):   File "/data/tests/common/helpers/parallel.py", line 35, in run     Process.run(self)   File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run     self._target(*self._args, **self._kwargs)   File "/data/tests/common/helpers/parallel.py", line 245, in wrapper     target(*args, **kwargs)   File "/data/tests/common/plugins/loganalyze 177.21
False snappi_tests/multidut/pfc/test_m2o_oversubscribe_lossless_lossy.py test_m2o_oversubscribe_lossless_lossy[multidut_port_info0] success   246.52
True snappi_tests/multidut/pfc/test_m2o_oversubscribe_lossless_lossy.py test_m2o_oversubscribe_lossless_lossy[multidut_port_info1] error failed on teardown with "Failed: Processes "['analyze_logs--']" failed with exit code "1" Exception: match: 1 expected_match: 0 expected_missing_match: 0 Match Messages: 2024 Dec  6 07:53:17.231843 yy39top-lc4 ERR monit[889]: 'routeCheck' status failed (255) -- Failure results: {{#12    "asic0": {#12        "missed_INTF_TABLE_entries": [#12            "fc00:10::1/128"#12        ]#12    }#12}}#012Failed. Look at reported mismatches above#012add: {#12    "asic0": [],#12    "asic1": [],#12    "asic2": []#12}#012del: {#12    "asic0": [],#12    "asic1": [],#12    "asic2": []#12} Traceback: Traceback (most recent call last):   File "/data/tests/common/helpers/parallel.py", line 35, in run     Process.run(self)   File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run     self._target(*self._args, **self._kwargs)   File "/data/tests/common/helpers/parallel.py", line 245, in wrapper     target(*args, **kwargs)   File "/data/tests/common/plugins/loganalyze 180.63
True snappi_tests/multidut/pfc/test_m2o_oversubscribe_lossy.py test_m2o_oversubscribe_lossy[multidut_port_info0] error failed on teardown with "Failed: Processes "['analyze_logs--']" failed with exit code "1" Exception: match: 1 expected_match: 0 expected_missing_match: 0 Match Messages: 2024 Dec  6 07:58:17.338889 yy39top-lc4 ERR monit[889]: 'routeCheck' status failed (255) -- Failure results: {{#12    "asic0": {#12        "missed_INTF_TABLE_entries": [#12            "fc00:10::1/128"#12        ]#12    }#12}}#012Failed. Look at reported mismatches above#012add: {#12    "asic0": [],#12    "asic1": [],#12    "asic2": []#12}#012del: {#12    "asic0": [],#12    "asic1": [],#12    "asic2": []#12} Traceback: Traceback (most recent call last):   File "/data/tests/common/helpers/parallel.py", line 35, in run     Process.run(self)   File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run     self._target(*self._args, **self._kwargs)   File "/data/tests/common/helpers/parallel.py", line 245, in wrapper     target(*args, **kwargs)   File "/data/tests/common/plugins/loganalyze 239.49
False snappi_tests/multidut/pfc/test_m2o_oversubscribe_lossy.py test_m2o_oversubscribe_lossy[multidut_port_info1] success   198.89
False snappi_tests/multidut/pfc/test_multidut_global_pause_with_snappi.py test_global_pause[multidut_port_info0] skipped Global pause is not supported in cisco-8000. 0.001
False snappi_tests/multidut/pfc/test_multidut_global_pause_with_snappi.py test_global_pause[multidut_port_info1] skipped Global pause is not supported in cisco-8000. 0.001
False snappi_tests/multidut/pfc/test_multidut_pfc_pause_lossless_with_snappi.py test_pfc_pause_single_lossless_prio[multidut_port_info0-yy39top-lc4|3] success   301.02
False snappi_tests/multidut/pfc/test_multidut_pfc_pause_lossless_with_snappi.py test_pfc_pause_single_lossless_prio[multidut_port_info0-yy39top-lc4|4] success   230.76
True snappi_tests/multidut/pfc/test_multidut_pfc_pause_lossless_with_snappi.py test_pfc_pause_single_lossless_prio[multidut_port_info1-yy39top-lc4|3] error failed on teardown with "Failed: Processes "['analyze_logs--']" failed with exit code "1" Exception: match: 1 expected_match: 0 expected_missing_match: 0 Match Messages: 2024 Dec  6 08:13:17.709316 yy39top-lc4 ERR monit[889]: 'routeCheck' status failed (255) -- Failure results: {{#12    "asic0": {#12        "missed_INTF_TABLE_entries": [#12            "fc00:10::1/128"#12        ]#12    }#12}}#012Failed. Look at reported mismatches above#012add: {#12    "asic0": [],#12    "asic1": [],#12    "asic2": []#12}#012del: {#12    "asic0": [],#12    "asic1": [],#12    "asic2": []#12} Traceback: Traceback (most recent call last):   File "/data/tests/common/helpers/parallel.py", line 35, in run     Process.run(self)   File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run     self._target(*self._args, **self._kwargs)   File "/data/tests/common/helpers/parallel.py", line 245, in wrapper     target(*args, **kwargs)   File "/data/tests/common/plugins/loganalyze 223.13
True snappi_tests/multidut/pfc/test_multidut_pfc_pause_lossless_with_snappi.py test_pfc_pause_single_lossless_prio[multidut_port_info1-yy39top-lc4|4] error failed on teardown with "Failed: Processes "['analyze_logs--']" failed with exit code "1" Exception: match: 1 expected_match: 0 expected_missing_match: 0 Match Messages: 2024 Dec  6 08:18:17.817387 yy39top-lc4 ERR monit[889]: 'routeCheck' status failed (255) -- Failure results: {{#12    "asic0": {#12        "missed_INTF_TABLE_entries": [#12            "fc00:10::1/128"#12        ]#12    }#12}}#012Failed. Look at reported mismatches above#012add: {#12    "asic0": [],#12    "asic1": [],#12    "asic2": []#12}#012del: {#12    "asic0": [],#12    "asic1": [],#12    "asic2": []#12} Traceback: Traceback (most recent call last):   File "/data/tests/common/helpers/parallel.py", line 35, in run     Process.run(self)   File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run     self._target(*self._args, **self._kwargs)   File "/data/tests/common/helpers/parallel.py", line 245, in wrapper     target(*args, **kwargs)   File "/data/tests/common/plugins/loganalyze 218.15
True snappi_tests/multidut/pfc/test_multidut_pfc_pause_lossless_with_snappi.py test_pfc_pause_multi_lossless_prio[multidut_port_info0] error failed on teardown with "Failed: Processes "['analyze_logs--']" failed with exit code "1" Exception: match: 1 expected_match: 0 expected_missing_match: 0 Match Messages: 2024 Dec  6 08:23:17.925535 yy39top-lc4 ERR monit[889]: 'routeCheck' status failed (255) -- Failure results: {{#12    "asic0": {#12        "missed_INTF_TABLE_entries": [#12            "fc00:10::1/128"#12        ]#12    }#12}}#012Failed. Look at reported mismatches above#012add: {#12    "asic0": [],#12    "asic1": [],#12    "asic2": []#12}#012del: {#12    "asic0": [],#12    "asic1": [],#12    "asic2": []#12} Traceback: Traceback (most recent call last):   File "/data/tests/common/helpers/parallel.py", line 35, in run     Process.run(self)   File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run     self._target(*self._args, **self._kwargs)   File "/data/tests/common/helpers/parallel.py", line 245, in wrapper     target(*args, **kwargs)   File "/data/tests/common/plugins/loganalyze 283.76
True snappi_tests/multidut/pfc/test_multidut_pfc_pause_lossless_with_snappi.py test_pfc_pause_multi_lossless_prio[multidut_port_info1] error failed on teardown with "Failed: Processes "['analyze_logs--']" failed with exit code "1" Exception: match: 1 expected_match: 0 expected_missing_match: 0 Match Messages: 2024 Dec  6 08:28:18.051596 yy39top-lc4 ERR monit[889]: 'routeCheck' status failed (255) -- Failure results: {{#12    "asic0": {#12        "missed_INTF_TABLE_entries": [#12            "fc00:10::1/128"#12        ]#12    }#12}}#012Failed. Look at reported mismatches above#012add: {#12    "asic0": [],#12    "asic1": [],#12    "asic2": []#12}#012del: {#12    "asic0": [],#12    "asic1": [],#12    "asic2": []#12} Traceback: Traceback (most recent call last):   File "/data/tests/common/helpers/parallel.py", line 35, in run     Process.run(self)   File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run     self._target(*self._args, **self._kwargs)   File "/data/tests/common/helpers/parallel.py", line 245, in wrapper     target(*args, **kwargs)   File "/data/tests/common/plugins/loganalyze 293.32
False snappi_tests/multidut/pfc/test_multidut_pfc_pause_lossless_with_snappi.py test_pfc_pause_single_lossless_prio_reboot[multidut_port_info0-warm-yy39top-lc4|4] skipped Reboot type warm is not supported on cisco-8000 switches 28.549
False snappi_tests/multidut/pfc/test_multidut_pfc_pause_lossless_with_snappi.py test_pfc_pause_single_lossless_prio_reboot[multidut_port_info0-cold-yy39top-lc4|4] success   1030.8
False snappi_tests/multidut/pfc/test_multidut_pfc_pause_lossless_with_snappi.py test_pfc_pause_single_lossless_prio_reboot[multidut_port_info0-fast-yy39top-lc4|4] skipped Reboot type fast is not supported on cisco-8000 switches 28.353
False snappi_tests/multidut/pfc/test_multidut_pfc_pause_lossless_with_snappi.py test_pfc_pause_single_lossless_prio_reboot[multidut_port_info1-warm-yy39top-lc4|4] skipped Reboot type warm is not supported on cisco-8000 switches 27.894
False snappi_tests/multidut/pfc/test_multidut_pfc_pause_lossless_with_snappi.py test_pfc_pause_single_lossless_prio_reboot[multidut_port_info1-cold-yy39top-lc4|4] success   1087.9
False snappi_tests/multidut/pfc/test_multidut_pfc_pause_lossless_with_snappi.py test_pfc_pause_single_lossless_prio_reboot[multidut_port_info1-fast-yy39top-lc4|4] skipped Reboot type fast is not supported on cisco-8000 switches 27.881
False snappi_tests/multidut/pfc/test_multidut_pfc_pause_lossless_with_snappi.py test_pfc_pause_multi_lossless_prio_reboot[multidut_port_info0-warm] skipped Reboot type warm is not supported on cisco-8000 switches 28.115
False snappi_tests/multidut/pfc/test_multidut_pfc_pause_lossless_with_snappi.py test_pfc_pause_multi_lossless_prio_reboot[multidut_port_info0-cold] success   1126.5
False snappi_tests/multidut/pfc/test_multidut_pfc_pause_lossless_with_snappi.py test_pfc_pause_multi_lossless_prio_reboot[multidut_port_info0-fast] skipped Reboot type fast is not supported on cisco-8000 switches 27.847
False snappi_tests/multidut/pfc/test_multidut_pfc_pause_lossless_with_snappi.py test_pfc_pause_multi_lossless_prio_reboot[multidut_port_info1-warm] skipped Reboot type warm is not supported on cisco-8000 switches 28.376
False snappi_tests/multidut/pfc/test_multidut_pfc_pause_lossless_with_snappi.py test_pfc_pause_multi_lossless_prio_reboot[multidut_port_info1-cold] success   1103.6
False snappi_tests/multidut/pfc/test_multidut_pfc_pause_lossless_with_snappi.py test_pfc_pause_multi_lossless_prio_reboot[multidut_port_info1-fast] skipped Reboot type fast is not supported on cisco-8000 switches 60.532
False snappi_tests/multidut/pfc/test_multidut_pfc_pause_lossy_with_snappi.py test_pfc_pause_single_lossy_prio[multidut_port_info0-yy39top-lc4|0] success   326.31
True snappi_tests/multidut/pfc/test_multidut_pfc_pause_lossy_with_snappi.py test_pfc_pause_single_lossy_prio[multidut_port_info0-yy39top-lc4|1] error failed on teardown with "Failed: Processes "['analyze_logs--']" failed with exit code "1" Exception: match: 1 expected_match: 0 expected_missing_match: 0 Match Messages: 2024 Dec  6 09:51:37.380865 yy39top-lc4 ERR monit[877]: 'routeCheck' status failed (255) -- Failure results: {{#12    "asic0": {#12        "missed_INTF_TABLE_entries": [#12            "fc00:10::1/128"#12        ]#12    }#12}}#012Failed. Look at reported mismatches above#012add: {#12    "asic0": [],#12    "asic1": [],#12    "asic2": []#12}#012del: {#12    "asic0": [],#12    "asic1": [],#12    "asic2": []#12} Traceback: Traceback (most recent call last):   File "/data/tests/common/helpers/parallel.py", line 35, in run     Process.run(self)   File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run     self._target(*self._args, **self._kwargs)   File "/data/tests/common/helpers/parallel.py", line 245, in wrapper     target(*args, **kwargs)   File "/data/tests/common/plugins/loganalyze 230.04
True snappi_tests/multidut/pfc/test_multidut_pfc_pause_lossy_with_snappi.py test_pfc_pause_single_lossy_prio[multidut_port_info0-yy39top-lc4|2] error failed on teardown with "Failed: Processes "['analyze_logs--']" failed with exit code "1" Exception: match: 1 expected_match: 0 expected_missing_match: 0 Match Messages: 2024 Dec  6 09:56:37.479253 yy39top-lc4 ERR monit[877]: 'routeCheck' status failed (255) -- Failure results: {{#12    "asic0": {#12        "missed_INTF_TABLE_entries": [#12            "fc00:10::1/128"#12        ]#12    }#12}}#012Failed. Look at reported mismatches above#012add: {#12    "asic0": [],#12    "asic1": [],#12    "asic2": []#12}#012del: {#12    "asic0": [],#12    "asic1": [],#12    "asic2": []#12} Traceback: Traceback (most recent call last):   File "/data/tests/common/helpers/parallel.py", line 35, in run     Process.run(self)   File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run     self._target(*self._args, **self._kwargs)   File "/data/tests/common/helpers/parallel.py", line 245, in wrapper     target(*args, **kwargs)   File "/data/tests/common/plugins/loganalyze 236.33
True snappi_tests/multidut/pfc/test_multidut_pfc_pause_lossy_with_snappi.py test_pfc_pause_single_lossy_prio[multidut_port_info0-yy39top-lc4|5] error failed on teardown with "Failed: Processes "['analyze_logs--']" failed with exit code "1" Exception: match: 1 expected_match: 0 expected_missing_match: 0 Match Messages: 2024 Dec  6 10:01:37.603101 yy39top-lc4 ERR monit[877]: 'routeCheck' status failed (255) -- Failure results: {{#12    "asic0": {#12        "missed_INTF_TABLE_entries": [#12            "fc00:10::1/128"#12        ]#12    }#12}}#012Failed. Look at reported mismatches above#012add: {#12    "asic0": [],#12    "asic1": [],#12    "asic2": []#12}#012del: {#12    "asic0": [],#12    "asic1": [],#12    "asic2": []#12} Traceback: Traceback (most recent call last):   File "/data/tests/common/helpers/parallel.py", line 35, in run     Process.run(self)   File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run     self._target(*self._args, **self._kwargs)   File "/data/tests/common/helpers/parallel.py", line 245, in wrapper     target(*args, **kwargs)   File "/data/tests/common/plugins/loganalyze 240.17
False snappi_tests/multidut/pfc/test_multidut_pfc_pause_lossy_with_snappi.py test_pfc_pause_single_lossy_prio[multidut_port_info0-yy39top-lc4|6] success   231.56
True snappi_tests/multidut/pfc/test_multidut_pfc_pause_lossy_with_snappi.py test_pfc_pause_single_lossy_prio[multidut_port_info1-yy39top-lc4|0] error failed on teardown with "Failed: Processes "['analyze_logs--']" failed with exit code "1" Exception: match: 1 expected_match: 0 expected_missing_match: 0 Match Messages: 2024 Dec  6 10:06:37.703934 yy39top-lc4 ERR monit[877]: 'routeCheck' status failed (255) -- Failure results: {{#12    "asic0": {#12        "missed_INTF_TABLE_entries": [#12            "fc00:10::1/128"#12        ]#12    }#12}}#012Failed. Look at reported mismatches above#012add: {#12    "asic0": [],#12    "asic1": [],#12    "asic2": []#12}#012del: {#12    "asic0": [],#12    "asic1": [],#12    "asic2": []#12} Traceback: Traceback (most recent call last):   File "/data/tests/common/helpers/parallel.py", line 35, in run     Process.run(self)   File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run     self._target(*self._args, **self._kwargs)   File "/data/tests/common/helpers/parallel.py", line 245, in wrapper     target(*args, **kwargs)   File "/data/tests/common/plugins/loganalyze 247.79
True snappi_tests/multidut/pfc/test_multidut_pfc_pause_lossy_with_snappi.py test_pfc_pause_single_lossy_prio[multidut_port_info1-yy39top-lc4|1] error failed on teardown with "Failed: Processes "['analyze_logs--']" failed with exit code "1" Exception: match: 1 expected_match: 0 expected_missing_match: 0 Match Messages: 2024 Dec  6 10:11:37.848936 yy39top-lc4 ERR monit[877]: 'routeCheck' status failed (255) -- Failure results: {{#12    "asic0": {#12        "missed_INTF_TABLE_entries": [#12            "fc00:10::1/128"#12        ]#12    }#12}}#012Failed. Look at reported mismatches above#012add: {#12    "asic0": [],#12    "asic1": [],#12    "asic2": []#12}#012del: {#12    "asic0": [],#12    "asic1": [],#12    "asic2": []#12} Traceback: Traceback (most recent call last):   File "/data/tests/common/helpers/parallel.py", line 35, in run     Process.run(self)   File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run     self._target(*self._args, **self._kwargs)   File "/data/tests/common/helpers/parallel.py", line 245, in wrapper     target(*args, **kwargs)   File "/data/tests/common/plugins/loganalyze 284.39
True snappi_tests/multidut/pfc/test_multidut_pfc_pause_lossy_with_snappi.py test_pfc_pause_single_lossy_prio[multidut_port_info1-yy39top-lc4|2] error failed on teardown with "Failed: Processes "['analyze_logs--']" failed with exit code "1" Exception: match: 1 expected_match: 0 expected_missing_match: 0 Match Messages: 2024 Dec  6 10:16:37.975617 yy39top-lc4 ERR monit[877]: 'routeCheck' status failed (255) -- Failure results: {{#12    "asic0": {#12        "missed_INTF_TABLE_entries": [#12            "fc00:10::1/128"#12        ]#12    }#12}}#012Failed. Look at reported mismatches above#012add: {#12    "asic0": [],#12    "asic1": [],#12    "asic2": []#12}#012del: {#12    "asic0": [],#12    "asic1": [],#12    "asic2": []#12} Traceback: Traceback (most recent call last):   File "/data/tests/common/helpers/parallel.py", line 35, in run     Process.run(self)   File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run     self._target(*self._args, **self._kwargs)   File "/data/tests/common/helpers/parallel.py", line 245, in wrapper     target(*args, **kwargs)   File "/data/tests/common/plugins/loganalyze 245.03
True snappi_tests/multidut/pfc/test_multidut_pfc_pause_lossy_with_snappi.py test_pfc_pause_single_lossy_prio[multidut_port_info1-yy39top-lc4|5] error failed on teardown with "Failed: Processes "['analyze_logs--']" failed with exit code "1" Exception: match: 1 expected_match: 0 expected_missing_match: 0 Match Messages: 2024 Dec  6 10:21:38.081387 yy39top-lc4 ERR monit[877]: 'routeCheck' status failed (255) -- Failure results: {{#12    "asic0": {#12        "missed_INTF_TABLE_entries": [#12            "fc00:10::1/128"#12        ]#12    }#12}}#012Failed. Look at reported mismatches above#012add: {#12    "asic0": [],#12    "asic1": [],#12    "asic2": []#12}#012del: {#12    "asic0": [],#12    "asic1": [],#12    "asic2": []#12} Traceback: Traceback (most recent call last):   File "/data/tests/common/helpers/parallel.py", line 35, in run     Process.run(self)   File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run     self._target(*self._args, **self._kwargs)   File "/data/tests/common/helpers/parallel.py", line 245, in wrapper     target(*args, **kwargs)   File "/data/tests/common/plugins/loganalyze 239.3
True snappi_tests/multidut/pfc/test_multidut_pfc_pause_lossy_with_snappi.py test_pfc_pause_single_lossy_prio[multidut_port_info1-yy39top-lc4|6] error failed on teardown with "Failed: Processes "['analyze_logs--']" failed with exit code "1" Exception: match: 1 expected_match: 0 expected_missing_match: 0 Match Messages: 2024 Dec  6 10:26:38.193405 yy39top-lc4 ERR monit[877]: 'routeCheck' status failed (255) -- Failure results: {{#12    "asic0": {#12        "missed_INTF_TABLE_entries": [#12            "fc00:10::1/128"#12        ]#12    }#12}}#012Failed. Look at reported mismatches above#012add: {#12    "asic0": [],#12    "asic1": [],#12    "asic2": []#12}#012del: {#12    "asic0": [],#12    "asic1": [],#12    "asic2": []#12} Traceback: Traceback (most recent call last):   File "/data/tests/common/helpers/parallel.py", line 35, in run     Process.run(self)   File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run     self._target(*self._args, **self._kwargs)   File "/data/tests/common/helpers/parallel.py", line 245, in wrapper     target(*args, **kwargs)   File "/data/tests/common/plugins/loganalyze 270.97
True snappi_tests/multidut/pfc/test_multidut_pfc_pause_lossy_with_snappi.py test_pfc_pause_multi_lossy_prio[multidut_port_info0] error failed on teardown with "Failed: Processes "['analyze_logs--']" failed with exit code "1" Exception: match: 1 expected_match: 0 expected_missing_match: 0 Match Messages: 2024 Dec  6 10:31:38.311949 yy39top-lc4 ERR monit[877]: 'routeCheck' status failed (255) -- Failure results: {{#12    "asic0": {#12        "missed_INTF_TABLE_entries": [#12            "fc00:10::1/128"#12        ]#12    }#12}}#012Failed. Look at reported mismatches above#012add: {#12    "asic0": [],#12    "asic1": [],#12    "asic2": []#12}#012del: {#12    "asic0": [],#12    "asic1": [],#12    "asic2": []#12} Traceback: Traceback (most recent call last):   File "/data/tests/common/helpers/parallel.py", line 35, in run     Process.run(self)   File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run     self._target(*self._args, **self._kwargs)   File "/data/tests/common/helpers/parallel.py", line 245, in wrapper     target(*args, **kwargs)   File "/data/tests/common/plugins/loganalyze 495.94
True snappi_tests/multidut/pfc/test_multidut_pfc_pause_lossy_with_snappi.py test_pfc_pause_multi_lossy_prio[multidut_port_info1] error failed on teardown with "Failed: Processes "['analyze_logs--']" failed with exit code "1" Exception: match: 2 expected_match: 0 expected_missing_match: 0 Match Messages: 2024 Dec  6 10:36:38.440184 yy39top-lc4 ERR monit[877]: 'routeCheck' status failed (255) -- Failure results: {{#12    "asic0": {#12        "missed_INTF_TABLE_entries": [#12            "fc00:10::1/128"#12        ]#12    }#12}}#012Failed. Look at reported mismatches above#012add: {#12    "asic0": [],#12    "asic1": [],#12    "asic2": []#12}#012del: {#12    "asic0": [],#12    "asic1": [],#12    "asic2": []#12} 2024 Dec  6 10:41:38.565555 yy39top-lc4 ERR monit[877]: 'routeCheck' status failed (255) -- Failure results: {{#12    "asic0": {#12        "missed_INTF_TABLE_entries": [#12            "fc00:10::1/128"#12        ]#12    }#12}}#012Failed. Look at reported mismatches above#012add: {#12    "asic0": [],#12    "asic1": [],#12    "asic2": []#12}#012del: {#12    "asic0": [],#12    "asic1" 518.66
False snappi_tests/multidut/pfc/test_multidut_pfc_pause_lossy_with_snappi.py test_pfc_pause_single_lossy_prio_reboot[multidut_port_info0-warm-yy39top-lc4|6] skipped Reboot type warm is not supported on cisco-8000 switches 13.451
False snappi_tests/multidut/pfc/test_multidut_pfc_pause_lossy_with_snappi.py test_pfc_pause_single_lossy_prio_reboot[multidut_port_info0-cold-yy39top-lc4|6] success   1101.5
False snappi_tests/multidut/pfc/test_multidut_pfc_pause_lossy_with_snappi.py test_pfc_pause_single_lossy_prio_reboot[multidut_port_info0-fast-yy39top-lc4|6] skipped Reboot type fast is not supported on cisco-8000 switches 13.668
False snappi_tests/multidut/pfc/test_multidut_pfc_pause_lossy_with_snappi.py test_pfc_pause_single_lossy_prio_reboot[multidut_port_info1-warm-yy39top-lc4|6] skipped Reboot type warm is not supported on cisco-8000 switches 13.21
False snappi_tests/multidut/pfc/test_multidut_pfc_pause_lossy_with_snappi.py test_pfc_pause_single_lossy_prio_reboot[multidut_port_info1-cold-yy39top-lc4|6] success   1106.1
False snappi_tests/multidut/pfc/test_multidut_pfc_pause_lossy_with_snappi.py test_pfc_pause_single_lossy_prio_reboot[multidut_port_info1-fast-yy39top-lc4|6] skipped Reboot type fast is not supported on cisco-8000 switches 13.228
False snappi_tests/multidut/pfc/test_multidut_pfc_pause_lossy_with_snappi.py test_pfc_pause_multi_lossy_prio_reboot[multidut_port_info0-warm] skipped Reboot type warm is not supported on cisco-8000 switches 13.026
False snappi_tests/multidut/pfc/test_multidut_pfc_pause_lossy_with_snappi.py test_pfc_pause_multi_lossy_prio_reboot[multidut_port_info0-cold] success   1353
False snappi_tests/multidut/pfc/test_multidut_pfc_pause_lossy_with_snappi.py test_pfc_pause_multi_lossy_prio_reboot[multidut_port_info0-fast] skipped Reboot type fast is not supported on cisco-8000 switches 13.077
False snappi_tests/multidut/pfc/test_multidut_pfc_pause_lossy_with_snappi.py test_pfc_pause_multi_lossy_prio_reboot[multidut_port_info1-warm] skipped Reboot type warm is not supported on cisco-8000 switches 13.298
False snappi_tests/multidut/pfc/test_multidut_pfc_pause_lossy_with_snappi.py test_pfc_pause_multi_lossy_prio_reboot[multidut_port_info1-cold] success   1329.8
False snappi_tests/multidut/pfc/test_multidut_pfc_pause_lossy_with_snappi.py test_pfc_pause_multi_lossy_prio_reboot[multidut_port_info1-fast] skipped Reboot type fast is not supported on cisco-8000 switches 62.379
False snappi_tests/multidut/pfcwd/test_multidut_pfcwd_a2a_with_snappi.py test_multidut_pfcwd_all_to_all[multidut_port_info0-False] success   214.67
False snappi_tests/multidut/pfcwd/test_multidut_pfcwd_a2a_with_snappi.py test_multidut_pfcwd_all_to_all[multidut_port_info1-False] success   147.51
False snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py test_pfcwd_basic_single_lossless_prio[multidut_port_info0-True] success   204.33
False snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py test_pfcwd_basic_single_lossless_prio[multidut_port_info0-False] success   107.25
False snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py test_pfcwd_basic_single_lossless_prio[multidut_port_info1-True] success   136.08
False snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py test_pfcwd_basic_single_lossless_prio[multidut_port_info1-False] success   96.692
True snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py test_pfcwd_basic_multi_lossless_prio[multidut_port_info0-True] error failed on teardown with "Failed: Processes "['analyze_logs--']" failed with exit code "1" Exception: match: 1 expected_match: 0 expected_missing_match: 0 Match Messages: 2024 Dec  6 12:25:15.970461 yy39top-lc4 ERR monit[879]: 'routeCheck' status failed (255) -- Failure results: {{#12    "asic0": {#12        "missed_INTF_TABLE_entries": [#12            "fc00:10::1/128"#12        ]#12    }#12}}#012Failed. Look at reported mismatches above#012add: {#12    "asic0": [],#12    "asic1": [],#12    "asic2": []#12}#012del: {#12    "asic0": [],#12    "asic1": [],#12    "asic2": []#12} Traceback: Traceback (most recent call last):   File "/data/tests/common/helpers/parallel.py", line 35, in run     Process.run(self)   File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run     self._target(*self._args, **self._kwargs)   File "/data/tests/common/helpers/parallel.py", line 245, in wrapper     target(*args, **kwargs)   File "/data/tests/common/plugins/loganalyze 124.21
False snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py test_pfcwd_basic_multi_lossless_prio[multidut_port_info0-False] success   121.66
False snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py test_pfcwd_basic_multi_lossless_prio[multidut_port_info1-True] success   115.91
True snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py test_pfcwd_basic_multi_lossless_prio[multidut_port_info1-False] error failed on teardown with "Failed: Processes "['analyze_logs--']" failed with exit code "1" Exception: match: 1 expected_match: 0 expected_missing_match: 0 Match Messages: 2024 Dec  6 12:30:16.082405 yy39top-lc4 ERR monit[879]: 'routeCheck' status failed (255) -- Failure results: {{#12    "asic0": {#12        "missed_INTF_TABLE_entries": [#12            "fc00:10::1/128"#12        ]#12    }#12}}#012Failed. Look at reported mismatches above#012add: {#12    "asic0": [],#12    "asic1": [],#12    "asic2": []#12}#012del: {#12    "asic0": [],#12    "asic1": [],#12    "asic2": []#12} Traceback: Traceback (most recent call last):   File "/data/tests/common/helpers/parallel.py", line 35, in run     Process.run(self)   File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run     self._target(*self._args, **self._kwargs)   File "/data/tests/common/helpers/parallel.py", line 245, in wrapper     target(*args, **kwargs)   File "/data/tests/common/plugins/loganalyze 110.61
False snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py test_pfcwd_basic_single_lossless_prio_reboot[multidut_port_info0-warm-yy39top-lc4|3-True] skipped Reboot type warm is not supported on cisco-8000 switches 13.713
False snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py test_pfcwd_basic_single_lossless_prio_reboot[multidut_port_info0-warm-yy39top-lc4|3-False] skipped Reboot type warm is not supported on cisco-8000 switches 13.399
False snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py test_pfcwd_basic_single_lossless_prio_reboot[multidut_port_info0-cold-yy39top-lc4|3-True] success   940.43
False snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py test_pfcwd_basic_single_lossless_prio_reboot[multidut_port_info0-cold-yy39top-lc4|3-False] success   953.08
False snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py test_pfcwd_basic_single_lossless_prio_reboot[multidut_port_info0-fast-yy39top-lc4|3-True] skipped Reboot type fast is not supported on cisco-8000 switches 12.971
False snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py test_pfcwd_basic_single_lossless_prio_reboot[multidut_port_info0-fast-yy39top-lc4|3-False] skipped Reboot type fast is not supported on cisco-8000 switches 13.45
False snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py test_pfcwd_basic_single_lossless_prio_reboot[multidut_port_info1-warm-yy39top-lc4|3-True] skipped Reboot type warm is not supported on cisco-8000 switches 13.628
False snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py test_pfcwd_basic_single_lossless_prio_reboot[multidut_port_info1-warm-yy39top-lc4|3-False] skipped Reboot type warm is not supported on cisco-8000 switches 13.046
False snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py test_pfcwd_basic_single_lossless_prio_reboot[multidut_port_info1-cold-yy39top-lc4|3-True] success   941.89
False snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py test_pfcwd_basic_single_lossless_prio_reboot[multidut_port_info1-cold-yy39top-lc4|3-False] success   920.81
False snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py test_pfcwd_basic_single_lossless_prio_reboot[multidut_port_info1-fast-yy39top-lc4|3-True] skipped Reboot type fast is not supported on cisco-8000 switches 13.062
False snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py test_pfcwd_basic_single_lossless_prio_reboot[multidut_port_info1-fast-yy39top-lc4|3-False] skipped Reboot type fast is not supported on cisco-8000 switches 13.617
False snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py test_pfcwd_basic_multi_lossless_prio_reboot[multidut_port_info0-warm-True] skipped Reboot type warm is not supported on cisco-8000 switches 13.001
False snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py test_pfcwd_basic_multi_lossless_prio_reboot[multidut_port_info0-warm-False] skipped Reboot type warm is not supported on cisco-8000 switches 13.362
False snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py test_pfcwd_basic_multi_lossless_prio_reboot[multidut_port_info0-cold-True] success   948.72
False snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py test_pfcwd_basic_multi_lossless_prio_reboot[multidut_port_info0-cold-False] success   949.03
False snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py test_pfcwd_basic_multi_lossless_prio_reboot[multidut_port_info0-fast-True] skipped Reboot type fast is not supported on cisco-8000 switches 13.136
False snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py test_pfcwd_basic_multi_lossless_prio_reboot[multidut_port_info0-fast-False] skipped Reboot type fast is not supported on cisco-8000 switches 13.724
False snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py test_pfcwd_basic_multi_lossless_prio_reboot[multidut_port_info1-warm-True] skipped Reboot type warm is not supported on cisco-8000 switches 13.502
False snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py test_pfcwd_basic_multi_lossless_prio_reboot[multidut_port_info1-warm-False] skipped Reboot type warm is not supported on cisco-8000 switches 13.174
False snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py test_pfcwd_basic_multi_lossless_prio_reboot[multidut_port_info1-cold-True] success   951.02
False snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py test_pfcwd_basic_multi_lossless_prio_reboot[multidut_port_info1-cold-False] success   947.27
False snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py test_pfcwd_basic_multi_lossless_prio_reboot[multidut_port_info1-fast-True] skipped Reboot type fast is not supported on cisco-8000 switches 13.143
False snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py test_pfcwd_basic_multi_lossless_prio_reboot[multidut_port_info1-fast-False] skipped Reboot type fast is not supported on cisco-8000 switches 13.312
False snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py test_pfcwd_basic_single_lossless_prio_service_restart[multidut_port_info0-True-swss] success   1331.5
False snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py test_pfcwd_basic_single_lossless_prio_service_restart[multidut_port_info0-False-swss] success   1336.5
False snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py test_pfcwd_basic_single_lossless_prio_service_restart[multidut_port_info1-True-swss] success   1326.4
False snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py test_pfcwd_basic_single_lossless_prio_service_restart[multidut_port_info1-False-swss] success   1329.3
False snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py test_pfcwd_basic_multi_lossless_prio_restart_service[multidut_port_info0-True-swss] success   1330.4
False snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py test_pfcwd_basic_multi_lossless_prio_restart_service[multidut_port_info0-False-swss] success   1330.4
False snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py test_pfcwd_basic_multi_lossless_prio_restart_service[multidut_port_info1-True-swss] success   1334.2
False snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py test_pfcwd_basic_multi_lossless_prio_restart_service[multidut_port_info1-False-swss] success   1355.9
True snappi_tests/multidut/pfcwd/test_multidut_pfcwd_burst_storm_with_snappi.py test_pfcwd_burst_storm_single_lossless_prio[multidut_port_info0] error failed on teardown with "Failed: Processes "['analyze_logs--']" failed with exit code "1" Exception: match: 1 expected_match: 0 expected_missing_match: 0 Match Messages: 2024 Dec  6 17:40:55.301213 yy39top-lc4 ERR monit[872]: 'routeCheck' status failed (255) -- Failure results: {{#12    "asic0": {#12        "missed_INTF_TABLE_entries": [#12            "fc00:10::1/128"#12        ]#12    }#12}}#012Failed. Look at reported mismatches above#012add: {#12    "asic0": [],#12    "asic1": [],#12    "asic2": []#12}#012del: {#12    "asic0": [],#12    "asic1": [],#12    "asic2": []#12} Traceback: Traceback (most recent call last):   File "/data/tests/common/helpers/parallel.py", line 35, in run     Process.run(self)   File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run     self._target(*self._args, **self._kwargs)   File "/data/tests/common/helpers/parallel.py", line 245, in wrapper     target(*args, **kwargs)   File "/data/tests/common/plugins/loganalyze 234.39
False snappi_tests/multidut/pfcwd/test_multidut_pfcwd_burst_storm_with_snappi.py test_pfcwd_burst_storm_single_lossless_prio[multidut_port_info1] success   152.64
False snappi_tests/multidut/pfcwd/test_multidut_pfcwd_m2o_with_snappi.py test_pfcwd_many_to_one[multidut_port_info0-True] failure ZeroDivisionError: float division by zero 199.56
False snappi_tests/multidut/pfcwd/test_multidut_pfcwd_m2o_with_snappi.py test_pfcwd_many_to_one[multidut_port_info1-True] failure ZeroDivisionError: float division by zero 127.697
True snappi_tests/multidut/pfcwd/test_multidut_pfcwd_m2o_with_snappi.py test_pfcwd_many_to_one[multidut_port_info1-True] error failed on teardown with "Failed: Processes "['analyze_logs--']" failed with exit code "1" Exception: match: 1 expected_match: 0 expected_missing_match: 0 Match Messages: 2024 Dec  6 17:50:55.513076 yy39top-lc4 ERR monit[872]: 'routeCheck' status failed (255) -- Failure results: {{#12    "asic0": {#12        "missed_INTF_TABLE_entries": [#12            "20.1.1.2/32",#12            "fc00:10::1/128"#12        ]#12    }#12}}#012Failed. Look at reported mismatches above#012add: {#12    "asic0": [],#12    "asic1": [],#12    "asic2": []#12}#012del: {#12    "asic0": [],#12    "asic1": [],#12    "asic2": []#12} Traceback: Traceback (most recent call last):   File "/data/tests/common/helpers/parallel.py", line 35, in run     Process.run(self)   File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run     self._target(*self._args, **self._kwargs)   File "/data/tests/common/helpers/parallel.py", line 245, in wrapper     target(*args, **kwargs)   File "/data/t 341.4
False snappi_tests/multidut/pfcwd/test_multidut_pfcwd_runtime_traffic_with_snappi.py test_pfcwd_runtime_traffic[multidut_port_info0] success   227.45
False snappi_tests/multidut/pfcwd/test_multidut_pfcwd_runtime_traffic_with_snappi.py test_pfcwd_runtime_traffic[multidut_port_info1] success   165.51

@yejianquan yejianquan merged commit 47e7797 into sonic-net:master Dec 11, 2024
17 checks passed
mssonicbld pushed a commit to mssonicbld/sonic-mgmt that referenced this pull request Dec 11, 2024
…ons. (sonic-net#15906)

Description of PR
Currently the traffic for RDMA tests use randomly selected udp ports. This sometimes ends up sending traffic in a non-load-balanced way, and causes the tests to fail.

In this PR:

For multiple scripts that use udp: We serialize the UDP port numbers instead of randomly picking the numbers. This allows for more uniform load-balancing spread, and allows tests to pass more.
in multi_lossless helper: We reduce the background traffic rate, and raise the rate for lossless traffic, thus keeping the overall rate still same. This allows for accounting the speed of backplane.

co-authorized by: jianquanye@microsoft.com
@mssonicbld
Copy link
Collaborator

Cherry-pick PR to 202405: #15990

mssonicbld pushed a commit that referenced this pull request Dec 11, 2024
…ons. (#15906)

Description of PR
Currently the traffic for RDMA tests use randomly selected udp ports. This sometimes ends up sending traffic in a non-load-balanced way, and causes the tests to fail.

In this PR:

For multiple scripts that use udp: We serialize the UDP port numbers instead of randomly picking the numbers. This allows for more uniform load-balancing spread, and allows tests to pass more.
in multi_lossless helper: We reduce the background traffic rate, and raise the rate for lossless traffic, thus keeping the overall rate still same. This allows for accounting the speed of backplane.

co-authorized by: jianquanye@microsoft.com
BG_FLOW_NAME = 'Background Flow'
BG_FLOW_AGGR_RATE_PERCENT = 25
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the description:
"in multi_lossless helper: We reduce the background traffic rate, and raise the rate for lossless traffic, thus keeping the overall rate still same. This allows for accounting the speed of backplane."
Earlier lossless + lossy traffic is 30% + 25% = 55% from each ingress.
Now lossless + lossy traffic is 35% + 22.5% = 57.5% from each ingress.

The traffic rate is not same.

@sdszhang , @vmittal-msft

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, @amitpawar12 . That was intentional. We had discussions, and since lossless will get paused anyways, it can be higher, if we want.

mssonicbld pushed a commit to mssonicbld/sonic-mgmt that referenced this pull request Jan 2, 2025
…ons. (sonic-net#15906)

Description of PR
Currently the traffic for RDMA tests use randomly selected udp ports. This sometimes ends up sending traffic in a non-load-balanced way, and causes the tests to fail.

In this PR:

For multiple scripts that use udp: We serialize the UDP port numbers instead of randomly picking the numbers. This allows for more uniform load-balancing spread, and allows tests to pass more.
in multi_lossless helper: We reduce the background traffic rate, and raise the rate for lossless traffic, thus keeping the overall rate still same. This allows for accounting the speed of backplane.

co-authorized by: jianquanye@microsoft.com
@mssonicbld
Copy link
Collaborator

Cherry-pick PR to 202411: #16294

mssonicbld pushed a commit that referenced this pull request Jan 3, 2025
…ons. (#15906)

Description of PR
Currently the traffic for RDMA tests use randomly selected udp ports. This sometimes ends up sending traffic in a non-load-balanced way, and causes the tests to fail.

In this PR:

For multiple scripts that use udp: We serialize the UDP port numbers instead of randomly picking the numbers. This allows for more uniform load-balancing spread, and allows tests to pass more.
in multi_lossless helper: We reduce the background traffic rate, and raise the rate for lossless traffic, thus keeping the overall rate still same. This allows for accounting the speed of backplane.

co-authorized by: jianquanye@microsoft.com
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants