Skip to content

Commit

Permalink
Merge pull request #5 from empear-analytics/notify-cancelled-and-skip…
Browse files Browse the repository at this point in the history
…ped-jobs

Notifies of cancelled and skipped test jobs
  • Loading branch information
ulrikawiss authored Aug 3, 2023
2 parents e12b761 + 83f4ce1 commit 8cd792a
Show file tree
Hide file tree
Showing 15 changed files with 374 additions and 885 deletions.
45 changes: 43 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,10 @@
# Sending JUnit report to Slack
# Sending xUnit report to Slack

## Buildning
This Github Action can be used in a Github Workflow to [report test results into a Slack Channel](https://api.slack.com/messaging/webhooks).

It expects a path to an xUnit-like result file. This file is parsed into a nice message, and posted to a Slack channel. It will also report skipped or cancelled test steps, and missing result files.

## Building

If you make changes, the action needs to be built and the resulting files need to be checked in.

Expand All @@ -9,3 +13,40 @@ https://docs.github.com/en/actions/creating-actions/creating-a-javascript-action
1. `npm i -g @vercel/ncc` if you didn't already do that
2. `ncc build index.ts --license LICENSE` or `npm run build`
3. commit, push, tag

## Usage

See [action.yml](action.yml) for parameters.

In the workflow yaml, add a step like this:

```
- name: My test step
id: my-test-step
...
- name: My reporting step
id: my-test-report
if: always()
uses: empear-analytics/xunit-xml-slack-action@v0.5
with:
test-step-outcome: ${{ steps.my-test-step.outcome }}
slack-webhook-url: ${{ secrets.MY_SLACK_WEBHOOK_URL }}
directory-path: test/results.xml
```
- The test step needs an **id** so its outcome can be used in `test-step-outcome`.
- The reporting step should have an **id**, this will then show up in the report. See the examples below: only the last report (UNKNOWN RESULT) has no id for the reporting step.
- Using `if: always()` for the reporting step is necessary, otherwise it will only report on successful tests.

## Examples

### When there are test results

<img src="img_3.png" alt="failing" style="width:400px;"/>

### Different types of no-result reports

<img src="img_1.png" alt="drawing" style="width:400px;"/>



3 changes: 3 additions & 0 deletions action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,9 @@ branding:
icon: "bell"
color: "green"
inputs:
test-step-outcome:
description: 'Outcome of the test step to be reported on. Can be found in the github steps context. Possible values are success, failure, cancelled, or skipped.'
required: true
slack-webhook-url:
description: 'Webhook URL for Slack'
required: true
Expand Down
3 changes: 2 additions & 1 deletion app/action-info.ts
Original file line number Diff line number Diff line change
@@ -1,12 +1,13 @@

import * as github from '@actions/github';

export default class ActionInfo{
workflowName: string;
stepId: string;
runUrl: string;
buildUrl: string;
constructor(){
this.workflowName = github.context.workflow;
this.stepId = github.context.action;
this.runUrl = `${github.context.serverUrl}/${github.context.repo.owner}/${github.context.repo.repo}/actions/runs/${github.context.runId}`;
this.buildUrl = `${github.context.serverUrl}/${github.context.repo.owner}/${github.context.repo.repo}/runs/${github.context.runNumber}`;
}
Expand Down
78 changes: 3 additions & 75 deletions results.xml → app/results-all-ok.xml
Original file line number Diff line number Diff line change
@@ -1,76 +1,6 @@
<testsuites id="" name="" tests="8" failures="1" skipped="1" errors="0" time="195.091">
<testsuite name="cloud/test-4f.spec.ts" timestamp="1655667311825" hostname="" tests="1" failures="1" skipped="0" time="57.449" errors="0">
<testsuites id="" name="" tests="8" failures="0" skipped="0" errors="0" time="195.091">
<testsuite name="cloud/test-4f.spec.ts" timestamp="1655667311825" hostname="" tests="1" failures="0" skipped="0" time="57.449" errors="0">
<testcase name="4 factors dashboard Check 4 factors @dashboard on @cloud" classname="[staging] › cloud/test-4f.spec.ts:11:5 › 4 factors dashboard › Check 4 factors @dashboard on @cloud" time="57.449">
<failure message="test-4f.spec.ts:11:5 Check 4 factors @dashboard on @cloud" type="FAILURE">
[staging] › cloud/test-4f.spec.ts:11:5 › 4 factors dashboard › Check 4 factors @dashboard on @cloud

Error: expect(received).toBeVisible()
Call log:
- expect.toBeVisible with timeout 5000ms
- waiting for selector "[data-automation-id='hotspot-code-health-div']"


15 | const analysisResultsPage = await projectsPage.clickOnProjectTitle(projectName);
16 | let newDashboardPage = await analysisResultsPage.load4f();
> 17 | await newDashboardPage.verifyVisibilityOfHotspotCodeHealth();
| ^
18 | let hotspotsPage = await newDashboardPage.clickOnViewHotspotsLink();
19 | await hotspotsPage.verifyHotspotsTab();
20 | await page.goBack();

at CodeHealthPage.verifyVisibilityOfHotspotCodeHealth (/Users/emir/Codebase/codescene/testing/ui-tests-playwright/pages/common-pages/analysis-results-pages/new-dashboard-page/code-health-page.ts:57:60)
at /Users/emir/Codebase/codescene/testing/ui-tests-playwright/tests/cloud/test-4f.spec.ts:17:30

Error: expect(received).toBeVisible()
Call log:
- expect.toBeVisible with timeout 5000ms
- waiting for selector "[data-automation-id='view-hotspots-in-hotspot-code-health']"


15 | const analysisResultsPage = await projectsPage.clickOnProjectTitle(projectName);
16 | let newDashboardPage = await analysisResultsPage.load4f();
> 17 | await newDashboardPage.verifyVisibilityOfHotspotCodeHealth();
| ^
18 | let hotspotsPage = await newDashboardPage.clickOnViewHotspotsLink();
19 | await hotspotsPage.verifyHotspotsTab();
20 | await page.goBack();

at CodeHealthPage.verifyVisibilityOfHotspotCodeHealth (/Users/emir/Codebase/codescene/testing/ui-tests-playwright/pages/common-pages/analysis-results-pages/new-dashboard-page/code-health-page.ts:58:50)
at /Users/emir/Codebase/codescene/testing/ui-tests-playwright/tests/cloud/test-4f.spec.ts:17:7

locator.click: Timeout 20000ms exceeded.
=========================== logs ===========================
waiting for selector "[data-automation-id='view-hotspots-in-hotspot-code-health']"
============================================================

16 | let newDashboardPage = await analysisResultsPage.load4f();
17 | await newDashboardPage.verifyVisibilityOfHotspotCodeHealth();
> 18 | let hotspotsPage = await newDashboardPage.clickOnViewHotspotsLink();
| ^
19 | await hotspotsPage.verifyHotspotsTab();
20 | await page.goBack();
21 | await newDashboardPage.verifyVisibilityOfAverageCodeHealth();

at CodeHealthPage.clickOnViewHotspotsLink (/Users/emir/Codebase/codescene/testing/ui-tests-playwright/pages/common-pages/analysis-results-pages/new-dashboard-page/code-health-page.ts:89:37)
at /Users/emir/Codebase/codescene/testing/ui-tests-playwright/tests/cloud/test-4f.spec.ts:18:49

attachment #1: video (video/webm) --------------------------------------------------------------
test-results/cloud-test-4f-4-factors-dashboard-Check-4-factors-dashboard-on-cloud-staging/video.webm
------------------------------------------------------------------------------------------------

attachment #2: trace (application/zip) ---------------------------------------------------------
test-results/cloud-test-4f-4-factors-dashboard-Check-4-factors-dashboard-on-cloud-staging/trace.zip
Usage:

npx playwright show-trace test-results/cloud-test-4f-4-factors-dashboard-Check-4-factors-dashboard-on-cloud-staging/trace.zip

------------------------------------------------------------------------------------------------

attachment #3: screenshot (image/png) ----------------------------------------------------------
test-results/cloud-test-4f-4-factors-dashboard-Check-4-factors-dashboard-on-cloud-staging/test-failed-1.png
------------------------------------------------------------------------------------------------

</failure>
<system-out>

[[ATTACHMENT|../test-results/cloud-test-4f-4-factors-dashboard-Check-4-factors-dashboard-on-cloud-staging/video.webm]]
Expand Down Expand Up @@ -152,10 +82,8 @@
</system-out>
</testcase>
</testsuite>
<testsuite name="cloud/test-project-architecture.spec.ts" timestamp="1655667311825" hostname="" tests="1" failures="0" skipped="1" time="0" errors="0">
<testsuite name="cloud/test-project-architecture.spec.ts" timestamp="1655667311825" hostname="" tests="1" failures="0" skipped="0" time="0" errors="0">
<testcase name="Architectural components Generate architectural components on @cloud" classname="[staging] › cloud/test-project-architecture.spec.ts:15:8 › Architectural components › Generate architectural components on @cloud" time="0">
<skipped>
</skipped>
</testcase>
</testsuite>
</testsuites>
17 changes: 0 additions & 17 deletions app/results-parser.test.ts

This file was deleted.

File renamed without changes.
56 changes: 56 additions & 0 deletions app/slack-message-no-results.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
import ActionInfo from "./action-info";


export default function noResultsSlackMessage(actionInfo: ActionInfo, title: string, text: string): object {
const noResultBlocks: string = getBlocks(actionInfo, title, text);
return {
text: `${actionInfo.workflowName} - ${title}`,
blocks: JSON.parse(noResultBlocks)
}
}

function getBlocks(actionInfo: ActionInfo, title: string, text: string): string {
return `
[
{
"type": "context",
"elements": [
{
"type": "plain_text",
"text": "Workflow: ${actionInfo.workflowName} :: Step: ${actionInfo.stepId}",
"emoji": true
}
]
},
{
"type": "divider"
},
{
"type": "section",
"text": {
"type": "mrkdwn",
"text": ":Question: *${title}:* ${text}"
}
},
{
"type": "divider"
},
{
"type": "actions",
"elements": [
{
"type": "button",
"text": {
"type": "plain_text",
"text": "Go to Action",
"emoji": true
},
"value": "action_go",
"url": "${actionInfo.runUrl}"
}
]
}
]
`;
}

99 changes: 99 additions & 0 deletions app/slack-message-with-results.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,99 @@
import ActionInfo from './action-info';
import ResultsParser from './results-parser';


export default function withResultSlackMessage(actionInfo: ActionInfo, testResults: ResultsParser): object {
const resultBlocks = getBlocks(testResults, actionInfo);
return {
text: `${actionInfo.workflowName} - ${testResults.failedTests > 0 ? "Failed" : "Passed"}`,
blocks: JSON.parse(resultBlocks)
}
}


function getFailedTestsSections(failed, failedTestsList: string[]): string {
const template = (testName: string, isFailed: boolean) => `{
"type": "section",
"text": {
"type": "mrkdwn",
"text": "${isFailed ? ":red_circle: " + testName : ":tada: *ALL PASSED*"} "
}
},`;
if (!failed) {
return template("", false);
} else {
return failedTestsList.map(testName => template(testName, true)).join("\n");
}
}

function getOverralTestsSection(passedTests, skippedTests, failedTests): string {
const passedSubstring = passedTests > 0 ? `:large_green_circle: *PASSED: ${passedTests}*` : "";
const failedSubstring = failedTests > 0 ? `:red_circle: *FAILED: ${failedTests}*` : "";
const skippedSubstring = skippedTests > 0 ? `:white_circle: *SKIPPED: ${skippedTests}*` : "";
const template = (passedTests, skippedTests, failedTests) => `{
"type": "section",
"text": {
"type": "mrkdwn",
"text": "${passedSubstring} ${failedSubstring} ${skippedSubstring}"
}
},`;
return template(passedTests, skippedTests, failedTests);
}


function getBlocks(testResults: ResultsParser, actionInfo: ActionInfo): string {
const failedTests = testResults.failedTests;
const skippedTests = testResults.skippedTests;
const passedTests = testResults.passedTests;
const failedTestsList = testResults.failedTestsList;
const failed = failedTests > 0;
const failedTestsSections = getFailedTestsSections(failed, failedTestsList);
const overralTestsSection = getOverralTestsSection(passedTests, skippedTests, failedTests);
return `
[
{
"type": "context",
"elements": [
{
"type": "plain_text",
"text": "Workflow: ${actionInfo.workflowName} :: Step: ${actionInfo.stepId}",
"emoji": true
}
]
},
{
"type": "divider"
},
{
"type": "section",
"text": {
"type": "mrkdwn",
"text": ":clock1: *Execution time:* ${testResults.executionTime}"
}
},
${overralTestsSection}
{
"type": "divider"
},
${failedTestsSections}
{
"type": "divider"
},
{
"type": "actions",
"elements": [
{
"type": "button",
"text": {
"type": "plain_text",
"text": "Go to Action",
"emoji": true
},
"value": "action_go",
"url": "${actionInfo.runUrl}"
}
]
}
]
`;
}
Loading

0 comments on commit 8cd792a

Please sign in to comment.