-
Notifications
You must be signed in to change notification settings - Fork 8
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add Application Troubleshooting V2 (#242)
* Add stubs for app troubleshooting * Add cached diffs for commits * Add parse log exceptions functionality * Add parser updates for sli exception monitoring * Finish report formatter * Add github issue functionality * Clone to PWD for runner
- Loading branch information
Showing
9 changed files
with
1,018 additions
and
14 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,107 @@ | ||
*** Settings *** | ||
Documentation Measures the number of exception stacktraces present in an application's logs over a time period. | ||
Metadata Author jon-funk | ||
Metadata Display Name Kubernetes Application Monitor | ||
Metadata Supports Kubernetes,AKS,EKS,GKE,OpenShift | ||
|
||
Library BuiltIn | ||
Library RW.Core | ||
Library RW.CLI | ||
Library RW.K8sApplications | ||
Library RW.platform | ||
Library RW.NextSteps | ||
Library OperatingSystem | ||
|
||
Suite Setup Suite Initialization | ||
|
||
|
||
*** Tasks *** | ||
Measure Application Exceptions | ||
[Documentation] Examines recent logs for exceptions, providing a count of them. | ||
[Tags] resource application workload logs state exceptions errors | ||
${cmd}= Set Variable | ||
... ${KUBERNETES_DISTRIBUTION_BINARY} --context=${CONTEXT} -n ${NAMESPACE} logs $(${KUBERNETES_DISTRIBUTION_BINARY} --context=${CONTEXT} -n ${NAMESPACE} get deployment,statefulset -l ${LABELS} -oname | head -n 1) --tail=${MAX_LOG_LINES} --limit-bytes=256000 --since=${LOGS_SINCE} --container=${CONTAINER_NAME} | ||
IF $EXCLUDE_PATTERN != "" | ||
${cmd}= Set Variable | ||
... ${cmd} | grep -Eiv "${EXCLUDE_PATTERN}" || true | ||
END | ||
|
||
${logs}= RW.CLI.Run Cli | ||
... cmd=${cmd} | ||
... render_in_commandlist=true | ||
... env=${env} | ||
... secret_file__kubeconfig=${kubeconfig} | ||
${parsed_exceptions}= RW.K8sApplications.Parse Exceptions ${logs.stdout} | ||
${count}= Evaluate len($parsed_exceptions) | ||
RW.Core.Push Metric ${count} | ||
|
||
|
||
*** Keywords *** | ||
Suite Initialization | ||
${kubeconfig}= RW.Core.Import Secret | ||
... kubeconfig | ||
... type=string | ||
... description=The kubernetes kubeconfig yaml containing connection configuration used to connect to cluster(s). | ||
... pattern=\w* | ||
... example=For examples, start here https://kubernetes.io/docs/concepts/configuration/organize-cluster-access-kubeconfig/ | ||
${NAMESPACE}= RW.Core.Import User Variable NAMESPACE | ||
... type=string | ||
... description=The name of the Kubernetes namespace to scope actions and searching to. | ||
... pattern=\w* | ||
... example=my-namespace | ||
... default=sock-shop | ||
${CONTEXT}= RW.Core.Import User Variable CONTEXT | ||
... type=string | ||
... description=Which Kubernetes context to operate within. | ||
... pattern=\w* | ||
... example=sandbox-cluster-1 | ||
... default=sandbox-cluster-1 | ||
${KUBERNETES_DISTRIBUTION_BINARY}= RW.Core.Import User Variable KUBERNETES_DISTRIBUTION_BINARY | ||
... type=string | ||
... description=Which binary to use for Kubernetes CLI commands. | ||
... enum=[kubectl,oc] | ||
... example=kubectl | ||
... default=kubectl | ||
${LABELS}= RW.Core.Import User Variable LABELS | ||
... type=string | ||
... description=The Kubernetes labels used to select the resource for logs. | ||
... pattern=\w* | ||
${LOGS_SINCE}= RW.Core.Import User Variable | ||
... LOGS_SINCE | ||
... type=string | ||
... description=How far back to fetch logs from containers in Kubernetes. Making this too recent and running the codebundle often could cause adverse performance. | ||
... pattern=\w* | ||
... example=15m | ||
... default=15m | ||
${EXCLUDE_PATTERN}= RW.Core.Import User Variable | ||
... EXCLUDE_PATTERN | ||
... type=string | ||
... description=Grep pattern to use to exclude exceptions that don't indicate a critical issue. | ||
... pattern=\w* | ||
... example=FalseError|SecondErrorToSkip | ||
... default=FalseError|SecondErrorToSkip | ||
${CONTAINER_NAME}= RW.Core.Import User Variable | ||
... CONTAINER_NAME | ||
... type=string | ||
... description=The name of the container within the selected pod that represents the application to troubleshoot. | ||
... pattern=\w* | ||
... example=myapp | ||
${MAX_LOG_LINES}= RW.Core.Import User Variable | ||
... MAX_LOG_LINES | ||
... type=string | ||
... description=The max number of log lines to request from Kubernetes workloads to be parsed. Setting this too high can adversely effect performance. | ||
... pattern=\w* | ||
... example=300 | ||
... default=300 | ||
Set Suite Variable ${kubeconfig} ${kubeconfig} | ||
Set Suite Variable ${KUBERNETES_DISTRIBUTION_BINARY} ${KUBERNETES_DISTRIBUTION_BINARY} | ||
Set Suite Variable ${CONTEXT} ${CONTEXT} | ||
Set Suite Variable ${LABELS} ${LABELS} | ||
Set Suite Variable ${NAMESPACE} ${NAMESPACE} | ||
Set Suite Variable ${LOGS_SINCE} ${LOGS_SINCE} | ||
Set Suite Variable ${EXCLUDE_PATTERN} ${EXCLUDE_PATTERN} | ||
Set Suite Variable ${CONTAINER_NAME} ${CONTAINER_NAME} | ||
Set Suite Variable ${MAX_LOG_LINES} ${MAX_LOG_LINES} | ||
Set Suite Variable | ||
... ${env} | ||
... {"LABELS":"${LABELS}", "KUBECONFIG":"./${kubeconfig.key}", "KUBERNETES_DISTRIBUTION_BINARY":"${KUBERNETES_DISTRIBUTION_BINARY}", "CONTEXT":"${CONTEXT}", "NAMESPACE":"${NAMESPACE}"} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
from .k8s_applications import * |
Oops, something went wrong.