Adding new TSG for troubleshooting AKS node auto-repair errors #1629
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,40 @@ | ||
| --- | ||
| title: Troubleshoot common node auto-repair errors | ||
| description: Troubleshoot scenarios where node auto-repair returns an error code when trying to repair your NotReady node. | ||
| ms.date: 10/01/2024 | ||
| ms.reviewer: | ||
| ms.service: azure-kubernetes-service | ||
| #Customer intent: As an Azure Kubernetes user, I want to make sure the automatic repair actions from AKS node auto-repair do not cause any impacts on my applications or cluster health. | ||
| ms.custom: sap:Node/node pool availability and performance | ||
| --- | ||
| # Troubleshoot common node auto-repair errors | ||
|
|
||
| When AKS detects a node which reports the NotReady status for more than 5 minutes, we will attempt to automatically repair your node. Learn more about the node auto-repair process [here](https://learn.microsoft.com/en-us/azure/aks/node-auto-repair). | ||
|
|
||
| During this process, AKS will initiate reboot, reimage, and redeploy actions on your unhealthy node. These repair actions may be unsuccessful due to an underlying cause, resulting in an error code. This article will discuss common errors with their potential causes and next steps, as well as best practices for monitoring node auto-repair. | ||
|
|
||
| ## Prerequisites | ||
| To determine what type of node auto-repair error has occured, you will need to look for the following Kubernetes event: | ||
| "Node auto-repair [reboot/reimage/redeploy] action failed due to an operation failure: [error code]." | ||
|
|
||
| ## Common error codes | ||
| The table below contains the most common node auto-repair errors. | ||
|
|
||
| | Error code | Potential causes | Next steps | | ||
| |---|---|---| | ||
| | ClientSecretCredential authentication failed | | | | ||
| | ARM ErrorCode: VMExtensionProvisioningError | | | | ||
|
There was a problem hiding this comment. probably relevant sometimes: https://learn.microsoft.com/en-us/troubleshoot/azure/azure-kubernetes/availability-performance/node-not-ready-custom-script-extension-errors sometimes this or the other specific CSE exit code TSGs: https://learn.microsoft.com/en-us/troubleshoot/azure/azure-kubernetes/create-upgrade-delete/error-code-outboundconnfailvmextensionerror |
||
| | ARM ErrorCode: InvalidParameter | | | | ||
|
There was a problem hiding this comment. I think this is mostly an issue with spot VM node objects still existing for a little while after the VMs are preempted. |
||
| | scaleSetNameAndInstanceIDFromProviderID failed | | | | ||
|
|
||
| | ManagedIdentityCredential authentication failed | | | | ||
|
There was a problem hiding this comment. Looks good, as long as we have some common errors , next steps. This is good to go. |
||
| | ARM ErrorCode: VMRedeploymentFailed | | | | ||
| | ARM ErrorCode: TooManyVMRedeploymentRequests | | | | ||
| | ARM ErrorCode: OutboundConnectivityNotEnabledOnVMSS | | | | ||
| | ARM ErrorCode: NotFound | | | | ||
|
|
||
|
|
||
| ## Best practices for monitoring | ||
|
|
||
| - By default, AKS stores Kubernetes events from the past 1 hour. We recommend for you to enable Container Insights to store events for up to 90 days. Enabling Container Insights will also allow you to query events and configure alerts to quickly detect node auto-repair errors. | ||
| - Configure alerts to quickly detect when node auto-repair errors occur. To configure an alert on a specific event, see instructions [here](LINK). | ||
| - Node auto-repair is a best-effort service and does not guarantee that your node will be restored back to Ready status. We highly recommend that you actively monitor and alert on node NotReady issues, and conduct your own troubleshooting and resolution of these issues. See [basic troubleshooting of node NotReady issues](LINK) for more details. | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
as mentioned on teams, I think we can remove this one and add some other top errors. Since this is an issue with misclassifying some MSI clusters which there is a fix rolling out for.