feat: implement MlxResetFW to reset the FW on VF changes #733

tobiasgiese · 2024-07-11T14:26:45Z

Todos:

implement new method to mstfwreset
add feature flag to opt-in/opt-out
add unit test
test it manually or via e2e test job (preferred)
add docs (will be added with docs: add feature gates to our readme #756)

Manual test shows that the implementation is working:

sriov-network-config-daemon 2024-07-11T14:39:36.793959572Z    INFO    mellanox/mellanox_plugin.go:190 mellanox-plugin: resetFW()      {"cmd-args": ["-d", "0000:03:00.0", "-l", "3", "-y", "reset"]}

foo@bar:~$ sudo mlxconfig -d /dev/mst/mt4119_pciconf0 q | grep -e SRIOV_EN -e NUM_OF_VFS
        NUM_OF_VFS                                  4
        SRIOV_EN                                    True(1)

github-actions · 2024-07-11T14:26:57Z

Thanks for your PR,
To run vendors CIs use one of:

/test-all: To run all tests for all vendors.
/test-e2e-all: To run all E2E tests for all vendors.
/test-e2e-nvidia-all: To run all E2E tests for NVIDIA vendor.

To skip the vendors CIs use one of:

/skip-all: To skip all tests for all vendors.
/skip-e2e-all: To skip all E2E tests for all vendors.
/skip-e2e-nvidia-all: To skip all E2E tests for NVIDIA vendor.
Best regards.

pkg/plugins/mellanox/mellanox_plugin.go

github-actions · 2024-07-12T11:57:05Z

Thanks for your PR,
To run vendors CIs use one of:

/test-all: To run all tests for all vendors.
/test-e2e-all: To run all E2E tests for all vendors.
/test-e2e-nvidia-all: To run all E2E tests for NVIDIA vendor.

To skip the vendors CIs use one of:

/skip-all: To skip all tests for all vendors.
/skip-e2e-all: To skip all E2E tests for all vendors.
/skip-e2e-nvidia-all: To skip all E2E tests for NVIDIA vendor.
Best regards.

github-actions · 2024-07-12T15:10:05Z

Thanks for your PR,
To run vendors CIs use one of:

/test-all: To run all tests for all vendors.
/test-e2e-all: To run all E2E tests for all vendors.
/test-e2e-nvidia-all: To run all E2E tests for NVIDIA vendor.

To skip the vendors CIs use one of:

/skip-all: To skip all tests for all vendors.
/skip-e2e-all: To skip all E2E tests for all vendors.
/skip-e2e-nvidia-all: To skip all E2E tests for NVIDIA vendor.
Best regards.

github-actions · 2024-07-15T14:29:13Z

Thanks for your PR,
To run vendors CIs use one of:

/test-all: To run all tests for all vendors.
/test-e2e-all: To run all E2E tests for all vendors.
/test-e2e-nvidia-all: To run all E2E tests for NVIDIA vendor.

To skip the vendors CIs use one of:

/skip-all: To skip all tests for all vendors.
/skip-e2e-all: To skip all E2E tests for all vendors.
/skip-e2e-nvidia-all: To skip all E2E tests for NVIDIA vendor.
Best regards.

github-actions · 2024-07-15T14:30:09Z

Thanks for your PR,
To run vendors CIs use one of:

/test-all: To run all tests for all vendors.
/test-e2e-all: To run all E2E tests for all vendors.
/test-e2e-nvidia-all: To run all E2E tests for NVIDIA vendor.

To skip the vendors CIs use one of:

/skip-all: To skip all tests for all vendors.
/skip-e2e-all: To skip all E2E tests for all vendors.
/skip-e2e-nvidia-all: To skip all E2E tests for NVIDIA vendor.
Best regards.

controllers/sriovoperatorconfig_controller.go

coveralls · 2024-07-15T14:45:37Z

Pull Request Test Coverage Report for Build 10179580977

Details

1 of 63 (1.59%) changed or added relevant lines in 6 files are covered.
6 unchanged lines in 2 files lost coverage.
Overall coverage decreased (-0.2%) to 43.816%

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
pkg/daemon/daemon.go	1	7	14.29%
pkg/helper/mock/mock_helper.go	0	10	0.0%
pkg/plugins/mellanox/mellanox_plugin.go	0	10	0.0%
pkg/vendors/mellanox/mock/mock_mellanox.go	0	10	0.0%
cmd/sriov-network-config-daemon/start.go	0	13	0.0%
pkg/vendors/mellanox/mellanox.go	0	13	0.0%

Files with Coverage Reduction	New Missed Lines	%
controllers/drain_controller.go	1	68.06%
pkg/plugins/mellanox/mellanox_plugin.go	5	7.64%

Totals
Change from base Build 10178846427:	-0.2%
Covered Lines:	6536
Relevant Lines:	14917

💛 - Coveralls

api/v1/sriovoperatorconfig_types.go

pkg/vars/vars.go

Dockerfile.sriov-network-config-daemon

cmd/sriov-network-config-daemon/start.go

github-actions · 2024-07-15T16:17:42Z

Thanks for your PR,
To run vendors CIs use one of:

/test-all: To run all tests for all vendors.
/test-e2e-all: To run all E2E tests for all vendors.
/test-e2e-nvidia-all: To run all E2E tests for NVIDIA vendor.

To skip the vendors CIs use one of:

/skip-all: To skip all tests for all vendors.
/skip-e2e-all: To skip all E2E tests for all vendors.
/skip-e2e-nvidia-all: To skip all E2E tests for NVIDIA vendor.
Best regards.

tobiasgiese · 2024-07-15T16:30:05Z

/test-all

github-actions · 2024-07-16T07:32:16Z

Thanks for your PR,
To run vendors CIs use one of:

/test-all: To run all tests for all vendors.
/test-e2e-all: To run all E2E tests for all vendors.
/test-e2e-nvidia-all: To run all E2E tests for NVIDIA vendor.

To skip the vendors CIs use one of:

/skip-all: To skip all tests for all vendors.
/skip-e2e-all: To skip all E2E tests for all vendors.
/skip-e2e-nvidia-all: To skip all E2E tests for NVIDIA vendor.
Best regards.

github-actions · 2024-07-16T07:38:27Z

Thanks for your PR,
To run vendors CIs use one of:

/test-all: To run all tests for all vendors.
/test-e2e-all: To run all E2E tests for all vendors.
/test-e2e-nvidia-all: To run all E2E tests for NVIDIA vendor.

To skip the vendors CIs use one of:

/skip-all: To skip all tests for all vendors.
/skip-e2e-all: To skip all E2E tests for all vendors.
/skip-e2e-nvidia-all: To skip all E2E tests for NVIDIA vendor.
Best regards.

github-actions · 2024-07-16T08:38:43Z

Thanks for your PR,
To run vendors CIs use one of:

/test-all: To run all tests for all vendors.
/test-e2e-all: To run all E2E tests for all vendors.
/test-e2e-nvidia-all: To run all E2E tests for NVIDIA vendor.

To skip the vendors CIs use one of:

/skip-all: To skip all tests for all vendors.
/skip-e2e-all: To skip all E2E tests for all vendors.
/skip-e2e-nvidia-all: To skip all E2E tests for NVIDIA vendor.
Best regards.

tobiasgiese · 2024-07-16T09:13:33Z

/test-all

github-actions · 2024-07-16T10:52:04Z

Thanks for your PR,
To run vendors CIs use one of:

/test-all: To run all tests for all vendors.
/test-e2e-all: To run all E2E tests for all vendors.
/test-e2e-nvidia-all: To run all E2E tests for NVIDIA vendor.

To skip the vendors CIs use one of:

/skip-all: To skip all tests for all vendors.
/skip-e2e-all: To skip all E2E tests for all vendors.
/skip-e2e-nvidia-all: To skip all E2E tests for NVIDIA vendor.
Best regards.

tobiasgiese · 2024-07-16T10:53:55Z

@zeeke @adrianchiris the GH Action ci-triggers change seems to work.
After a rebase and push no new comment has been added.
But after closing and reopening there is one. 👍🏻

tobiasgiese · 2024-07-16T11:01:12Z

/test-all

pkg/vendors/mellanox/mellanox.go

tobiasgiese · 2024-07-24T13:22:20Z

/test-all

SchSeba

in general looks good.

I only have an issue on what can be the case if one of the pciAddress for reset doesn't work and we return an error.

and if we reset the primary nic of the system we will not have an API connection to reconcile and we can leave the system in a bad state...

pkg/consts/constants.go

pkg/daemon/daemon.go

SchSeba · 2024-07-28T12:26:09Z

pkg/plugins/mellanox/mellanox_plugin.go

+		return err
+	}
+	if vars.MlxPluginFwReset {
+		return p.helpers.MlxResetFW(pciAddressesToReset)


return an error here can be problematic if we have multiple PCIAddress and one them them failed (for example the primary one)

If I understand you correctly your idea would be to introduce a errs variable to append the errors and return it at the end of the function, right?

tobiasgiese · 2024-07-29T08:19:10Z

and if we reset the primary nic of the system we will not have an API connection to reconcile and we can leave the system in a bad state...

Usually the reset of the fw should work. Also the code of the mstfwrest binary is tested. Additionally, we have a feature gate top opt-in, which means this is not enabled for the default usage. IMHO we should rely on the robustness of the reset.
Or have you already had experience with the reset and that's why concerns?
WDYT @adrianchiris

Dockerfile.sriov-network-config-daemon

cmd/sriov-network-config-daemon/start.go

pkg/daemon/daemon.go

pkg/vendors/mellanox/mellanox.go

SchSeba · 2024-08-04T07:23:27Z

Hi can you please rebase this one

adrianchiris · 2024-08-04T08:51:44Z

LGTM from my sided. just needs rebase

Signed-off-by: Tobias Giese <tgiese@nvidia.com>

To not enable the new feature by default we want to add a feature flag first. Signed-off-by: Tobias Giese <tgiese@nvidia.com>

tobiasgiese · 2024-08-05T05:54:17Z

Rebased, thanks for the hint. Didn't noticed the wrong commits

tobiasgiese · 2024-08-05T05:54:31Z

/test-all

adrianchiris · 2024-08-12T09:54:29Z

@SchSeba can you give this one a look ? would be great to get this one merged this week.

SchSeba

LGTM

nice work!

SalDaniele · 2024-09-27T14:42:43Z

@tobiasgiese @adrianchiris do you know which models fail without this featureGate? We expect the Nvidia DGXH100 will fail to apply mstconfig settings without this change as well.

adrianchiris · 2024-10-09T06:57:38Z

@SalDaniele this feature originated from specific servers utilizing intel 4th generation Xeon "Eagle Stream" platform.
where upon reboot it does not send PCIe PERST# signal to the NIC.

why do you think DGX H100 will also fail to apply mstconfig settings without this change ?

SalDaniele · 2024-10-14T13:33:32Z

We received a customer case (OCPBUGS-42246) involving the DGX H100 platform with a cx-7 nic. Inspecting the logs, the card fails to update num vfs from the sriov config daemon on reboot, causing a boot loop. Cold boot of the system allows the vfs to be configured as expected.

SalDaniele · 2024-10-14T13:34:20Z

After further inspection, it is possible it is due to a conflict between the SNO and Nvidia Network Operator

tobiasgiese commented Jul 12, 2024

View reviewed changes

pkg/plugins/mellanox/mellanox_plugin.go Outdated Show resolved Hide resolved

tobiasgiese force-pushed the impl-mlx-fw-reset branch from 1673a95 to 83927ff Compare July 12, 2024 15:09

tobiasgiese force-pushed the impl-mlx-fw-reset branch from 83927ff to 05f9ca2 Compare July 15, 2024 14:29

tobiasgiese force-pushed the impl-mlx-fw-reset branch from 05f9ca2 to 7192b84 Compare July 15, 2024 14:29

adrianchiris reviewed Jul 15, 2024

View reviewed changes

controllers/sriovoperatorconfig_controller.go Outdated Show resolved Hide resolved

adrianchiris reviewed Jul 15, 2024

View reviewed changes

api/v1/sriovoperatorconfig_types.go Outdated Show resolved Hide resolved

pkg/vars/vars.go Outdated Show resolved Hide resolved

wizhaoredhat reviewed Jul 15, 2024

View reviewed changes

Dockerfile.sriov-network-config-daemon Show resolved Hide resolved

wizhaoredhat reviewed Jul 15, 2024

View reviewed changes

cmd/sriov-network-config-daemon/start.go Outdated Show resolved Hide resolved

tobiasgiese force-pushed the impl-mlx-fw-reset branch from 7192b84 to 76a8f95 Compare July 15, 2024 16:17

tobiasgiese force-pushed the impl-mlx-fw-reset branch from 76a8f95 to f18352a Compare July 16, 2024 07:32

tobiasgiese force-pushed the impl-mlx-fw-reset branch from f18352a to 7202ecc Compare July 16, 2024 07:38

tobiasgiese force-pushed the impl-mlx-fw-reset branch from 7202ecc to ac6f6ff Compare July 16, 2024 08:38

tobiasgiese force-pushed the impl-mlx-fw-reset branch from ac6f6ff to 796da5a Compare July 16, 2024 10:49

tobiasgiese closed this Jul 16, 2024

tobiasgiese reopened this Jul 16, 2024

tobiasgiese force-pushed the impl-mlx-fw-reset branch 3 times, most recently from 0bcc6fe to d4986d4 Compare July 24, 2024 12:09

noama-nv reviewed Jul 24, 2024

View reviewed changes

pkg/vendors/mellanox/mellanox.go Outdated Show resolved Hide resolved

tobiasgiese force-pushed the impl-mlx-fw-reset branch from d4986d4 to 69b14ce Compare July 24, 2024 13:08

tobiasgiese changed the title ~~[WIP] feat: implement MlxResetFW to reset the FW on VF changes~~ feat: implement MlxResetFW to reset the FW on VF changes Jul 24, 2024

SchSeba reviewed Jul 28, 2024

View reviewed changes

tobiasgiese force-pushed the impl-mlx-fw-reset branch from 69b14ce to 0908678 Compare July 29, 2024 08:01

adrianchiris reviewed Jul 31, 2024

View reviewed changes

tobiasgiese force-pushed the impl-mlx-fw-reset branch from 0908678 to d80add9 Compare July 31, 2024 11:45

tobiasgiese added 2 commits August 5, 2024 07:52

feat: implement MlxResetFW to reset the FW on VF changes

0f1a72e

Signed-off-by: Tobias Giese <tgiese@nvidia.com>

chore: add feature flag for mellanox fw reset

08b524e

To not enable the new feature by default we want to add a feature flag first. Signed-off-by: Tobias Giese <tgiese@nvidia.com>

tobiasgiese force-pushed the impl-mlx-fw-reset branch from d80add9 to 08b524e Compare August 5, 2024 05:53

adrianchiris approved these changes Aug 5, 2024

View reviewed changes

tobiasgiese requested a review from SchSeba August 7, 2024 09:03

tobiasgiese mentioned this pull request Aug 12, 2024

docs: add feature gates to our readme #756

Merged

SchSeba approved these changes Aug 12, 2024

View reviewed changes

SchSeba merged commit 88017db into k8snetworkplumbingwg:master Aug 12, 2024
13 of 14 checks passed

tobiasgiese deleted the impl-mlx-fw-reset branch August 12, 2024 12:01

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: implement MlxResetFW to reset the FW on VF changes #733

feat: implement MlxResetFW to reset the FW on VF changes #733

tobiasgiese commented Jul 11, 2024 •

edited

Loading

github-actions bot commented Jul 11, 2024

github-actions bot commented Jul 12, 2024

github-actions bot commented Jul 12, 2024

github-actions bot commented Jul 15, 2024

github-actions bot commented Jul 15, 2024

coveralls commented Jul 15, 2024 •

edited

Loading

github-actions bot commented Jul 15, 2024

tobiasgiese commented Jul 15, 2024

github-actions bot commented Jul 16, 2024

github-actions bot commented Jul 16, 2024

github-actions bot commented Jul 16, 2024

tobiasgiese commented Jul 16, 2024

github-actions bot commented Jul 16, 2024

tobiasgiese commented Jul 16, 2024

tobiasgiese commented Jul 16, 2024

tobiasgiese commented Jul 24, 2024

SchSeba left a comment

SchSeba Jul 28, 2024

tobiasgiese Jul 29, 2024

tobiasgiese commented Jul 29, 2024

SchSeba commented Aug 4, 2024

adrianchiris commented Aug 4, 2024

tobiasgiese commented Aug 5, 2024

tobiasgiese commented Aug 5, 2024

adrianchiris commented Aug 12, 2024

SchSeba left a comment

SalDaniele commented Sep 27, 2024

adrianchiris commented Oct 9, 2024

SalDaniele commented Oct 14, 2024

SalDaniele commented Oct 14, 2024

feat: implement MlxResetFW to reset the FW on VF changes #733

feat: implement MlxResetFW to reset the FW on VF changes #733

Conversation

tobiasgiese commented Jul 11, 2024 • edited Loading

github-actions bot commented Jul 11, 2024

github-actions bot commented Jul 12, 2024

github-actions bot commented Jul 12, 2024

github-actions bot commented Jul 15, 2024

github-actions bot commented Jul 15, 2024

coveralls commented Jul 15, 2024 • edited Loading

Pull Request Test Coverage Report for Build 10179580977

Details

💛 - Coveralls

github-actions bot commented Jul 15, 2024

tobiasgiese commented Jul 15, 2024

github-actions bot commented Jul 16, 2024

github-actions bot commented Jul 16, 2024

github-actions bot commented Jul 16, 2024

tobiasgiese commented Jul 16, 2024

github-actions bot commented Jul 16, 2024

tobiasgiese commented Jul 16, 2024

tobiasgiese commented Jul 16, 2024

tobiasgiese commented Jul 24, 2024

SchSeba left a comment

Choose a reason for hiding this comment

SchSeba Jul 28, 2024

Choose a reason for hiding this comment

tobiasgiese Jul 29, 2024

Choose a reason for hiding this comment

tobiasgiese commented Jul 29, 2024

SchSeba commented Aug 4, 2024

adrianchiris commented Aug 4, 2024

tobiasgiese commented Aug 5, 2024

tobiasgiese commented Aug 5, 2024

adrianchiris commented Aug 12, 2024

SchSeba left a comment

Choose a reason for hiding this comment

SalDaniele commented Sep 27, 2024

adrianchiris commented Oct 9, 2024

SalDaniele commented Oct 14, 2024

SalDaniele commented Oct 14, 2024

tobiasgiese commented Jul 11, 2024 •

edited

Loading

coveralls commented Jul 15, 2024 •

edited

Loading