Skip to content

Commit

Permalink
fix: masking buy and earn webviews in playwright screenshots so rando…
Browse files Browse the repository at this point in the history 
…m font issues don't affect tests
  • Loading branch information
@ggilchrist-ledger
ggilchrist-ledger committed Jul 25, 2023
1 parent 91818fd commit a0de0e0
Show file tree
Hide file tree
Showing 2 changed files with 6 additions and 2 deletions.
4 changes: 3 additions & 1 deletion apps/ledger-live-desktop/tests/specs/services/buy.spec.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,9 @@ test("Buy / Sell @smoke", async ({ page }) => {
await portfolioPage.startBuyFlow();
await expect(await liveAppWebview.waitForCorrectTextInWebview("theme: dark")).toBe(true);
await expect(await liveAppWebview.waitForCorrectTextInWebview("lang: en")).toBe(true);
await expect.soft(page).toHaveScreenshot("buy-app-opened.png");
await expect
.soft(page)
.toHaveScreenshot("buy-app-opened.png", { mask: [page.locator("webview")] });

Check failure on line 73 in apps/ledger-live-desktop/tests/specs/services/buy.spec.ts

View workflow job for this annotation

GitHub Actions / Desktop E2E (Linux)

services/buy.spec.ts:52:5 › Buy / Sell @smoke 

  1) services/buy.spec.ts:52:5 › Buy / Sell @smoke › Navigate to Buy app from portfolio banner ─────
    Error: Screenshot comparison failed:

      447740 pixels (ratio 0.57 of all image pixels) are different.

    Call log:
      - expect.soft.toHaveScreenshot(buy-app-opened.png) with timeout 30000ms
      -   verifying given screenshot expectation
      - taking page screenshot
      -   disabled all CSS animations
      - 447740 pixels (ratio 0.57 of all image pixels) are different.
      - waiting 100ms before taking screenshot
      - taking page screenshot
      -   disabled all CSS animations
      - captured a stable screenshot
      - 447740 pixels (ratio 0.57 of all image pixels) are different.

    Expected: /runner/_work/ledger-live/ledger-live/apps/ledger-live-desktop/tests/artifacts/test-results/services-buy-Buy-Sell-smoke/buy-app-opened-expected.png
    Received: /runner/_work/ledger-live/ledger-live/apps/ledger-live-desktop/tests/artifacts/test-results/services-buy-Buy-Sell-smoke/buy-app-opened-actual.png
        Diff: /runner/_work/ledger-live/ledger-live/apps/ledger-live-desktop/tests/artifacts/test-results/services-buy-Buy-Sell-smoke/buy-app-opened-diff.png

      71 |     await expect
      72 |       .soft(page)
    > 73 |       .toHaveScreenshot("buy-app-opened.png", { mask: [page.locator("webview")] });
         |        ^
      74 |   });
      75 |
      76 |   await test.step("Navigate to Buy app from market", async () => {

        at /runner/_work/ledger-live/ledger-live/apps/ledger-live-desktop/tests/specs/services/buy.spec.ts:73:8
        at /runner/_work/ledger-live/ledger-live/apps/ledger-live-desktop/tests/specs/services/buy.spec.ts:67:3

Check failure on line 73 in apps/ledger-live-desktop/tests/specs/services/buy.spec.ts

View workflow job for this annotation

GitHub Actions / Desktop E2E (Linux)

services/buy.spec.ts:52:5 › Buy / Sell @smoke 

  1) services/buy.spec.ts:52:5 › Buy / Sell @smoke › Navigate to Buy app from portfolio banner ─────

    Retry #1 ───────────────────────────────────────────────────────────────────────────────────────
    Error: Screenshot comparison failed:

      447740 pixels (ratio 0.57 of all image pixels) are different.

    Call log:
      - expect.soft.toHaveScreenshot(buy-app-opened.png) with timeout 30000ms
      -   verifying given screenshot expectation
      - taking page screenshot
      -   disabled all CSS animations
      - 447740 pixels (ratio 0.57 of all image pixels) are different.
      - waiting 100ms before taking screenshot
      - taking page screenshot
      -   disabled all CSS animations
      - captured a stable screenshot
      - 447740 pixels (ratio 0.57 of all image pixels) are different.

    Expected: /runner/_work/ledger-live/ledger-live/apps/ledger-live-desktop/tests/artifacts/test-results/services-buy-Buy-Sell-smoke-retry1/buy-app-opened-expected.png
    Received: /runner/_work/ledger-live/ledger-live/apps/ledger-live-desktop/tests/artifacts/test-results/services-buy-Buy-Sell-smoke-retry1/buy-app-opened-actual.png
        Diff: /runner/_work/ledger-live/ledger-live/apps/ledger-live-desktop/tests/artifacts/test-results/services-buy-Buy-Sell-smoke-retry1/buy-app-opened-diff.png

      71 |     await expect
      72 |       .soft(page)
    > 73 |       .toHaveScreenshot("buy-app-opened.png", { mask: [page.locator("webview")] });
         |        ^
      74 |   });
      75 |
      76 |   await test.step("Navigate to Buy app from market", async () => {

        at /runner/_work/ledger-live/ledger-live/apps/ledger-live-desktop/tests/specs/services/buy.spec.ts:73:8
        at /runner/_work/ledger-live/ledger-live/apps/ledger-live-desktop/tests/specs/services/buy.spec.ts:67:3

Check failure on line 73 in apps/ledger-live-desktop/tests/specs/services/buy.spec.ts

View workflow job for this annotation

GitHub Actions / Desktop E2E (Linux)

services/buy.spec.ts:52:5 › Buy / Sell @smoke 

  1) services/buy.spec.ts:52:5 › Buy / Sell @smoke › Navigate to Buy app from portfolio banner ─────

    Retry #2 ───────────────────────────────────────────────────────────────────────────────────────
    Error: Screenshot comparison failed:

      447740 pixels (ratio 0.57 of all image pixels) are different.

    Call log:
      - expect.soft.toHaveScreenshot(buy-app-opened.png) with timeout 30000ms
      -   verifying given screenshot expectation
      - taking page screenshot
      -   disabled all CSS animations
      - 447740 pixels (ratio 0.57 of all image pixels) are different.
      - waiting 100ms before taking screenshot
      - taking page screenshot
      -   disabled all CSS animations
      - captured a stable screenshot
      - 447740 pixels (ratio 0.57 of all image pixels) are different.

    Expected: /runner/_work/ledger-live/ledger-live/apps/ledger-live-desktop/tests/artifacts/test-results/services-buy-Buy-Sell-smoke-retry2/buy-app-opened-expected.png
    Received: /runner/_work/ledger-live/ledger-live/apps/ledger-live-desktop/tests/artifacts/test-results/services-buy-Buy-Sell-smoke-retry2/buy-app-opened-actual.png
        Diff: /runner/_work/ledger-live/ledger-live/apps/ledger-live-desktop/tests/artifacts/test-results/services-buy-Buy-Sell-smoke-retry2/buy-app-opened-diff.png

      71 |     await expect
      72 |       .soft(page)
    > 73 |       .toHaveScreenshot("buy-app-opened.png", { mask: [page.locator("webview")] });
         |        ^
      74 |   });
      75 |
      76 |   await test.step("Navigate to Buy app from market", async () => {

        at /runner/_work/ledger-live/ledger-live/apps/ledger-live-desktop/tests/specs/services/buy.spec.ts:73:8
        at /runner/_work/ledger-live/ledger-live/apps/ledger-live-desktop/tests/specs/services/buy.spec.ts:67:3
});

await test.step("Navigate to Buy app from market", async () => {
Expand Down
4 changes: 3 additions & 1 deletion apps/ledger-live-desktop/tests/specs/services/earn.spec.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,8 @@ test("Earn @smoke", async ({ page }) => {
await expect(await liveAppWebview.waitForCorrectTextInWebview("currencyTicker: USD")).toBe(
true,
);
await expect.soft(page).toHaveScreenshot("earn-app-opened.png");
await expect
.soft(page)
.toHaveScreenshot("earn-app-opened.png", { mask: [page.locator("webview")] });

Check failure on line 60 in apps/ledger-live-desktop/tests/specs/services/earn.spec.ts

View workflow job for this annotation

GitHub Actions / Desktop E2E (Linux)

services/earn.spec.ts:42:5 › Earn @smoke 

  2) services/earn.spec.ts:42:5 › Earn @smoke › Navigate to Buy app from portfolio banner ──────────
    Error: Screenshot comparison failed:

      446468 pixels (ratio 0.57 of all image pixels) are different.

    Call log:
      - expect.soft.toHaveScreenshot(earn-app-opened.png) with timeout 30000ms
      -   verifying given screenshot expectation
      - taking page screenshot
      -   disabled all CSS animations
      - 446468 pixels (ratio 0.57 of all image pixels) are different.
      - waiting 100ms before taking screenshot
      - taking page screenshot
      -   disabled all CSS animations
      - captured a stable screenshot
      - 446468 pixels (ratio 0.57 of all image pixels) are different.

    Expected: /runner/_work/ledger-live/ledger-live/apps/ledger-live-desktop/tests/artifacts/test-results/services-earn-Earn-smoke/earn-app-opened-expected.png
    Received: /runner/_work/ledger-live/ledger-live/apps/ledger-live-desktop/tests/artifacts/test-results/services-earn-Earn-smoke/earn-app-opened-actual.png
        Diff: /runner/_work/ledger-live/ledger-live/apps/ledger-live-desktop/tests/artifacts/test-results/services-earn-Earn-smoke/earn-app-opened-diff.png

      58 |     await expect
      59 |       .soft(page)
    > 60 |       .toHaveScreenshot("earn-app-opened.png", { mask: [page.locator("webview")] });
         |        ^
      61 |   });
      62 | });
      63 |

        at /runner/_work/ledger-live/ledger-live/apps/ledger-live-desktop/tests/specs/services/earn.spec.ts:60:8
        at /runner/_work/ledger-live/ledger-live/apps/ledger-live-desktop/tests/specs/services/earn.spec.ts:51:3

Check failure on line 60 in apps/ledger-live-desktop/tests/specs/services/earn.spec.ts

View workflow job for this annotation

GitHub Actions / Desktop E2E (Linux)

services/earn.spec.ts:42:5 › Earn @smoke 

  2) services/earn.spec.ts:42:5 › Earn @smoke › Navigate to Buy app from portfolio banner ──────────

    Retry #1 ───────────────────────────────────────────────────────────────────────────────────────
    Error: Screenshot comparison failed:

      446468 pixels (ratio 0.57 of all image pixels) are different.

    Call log:
      - expect.soft.toHaveScreenshot(earn-app-opened.png) with timeout 30000ms
      -   verifying given screenshot expectation
      - taking page screenshot
      -   disabled all CSS animations
      - 446468 pixels (ratio 0.57 of all image pixels) are different.
      - waiting 100ms before taking screenshot
      - taking page screenshot
      -   disabled all CSS animations
      - captured a stable screenshot
      - 446468 pixels (ratio 0.57 of all image pixels) are different.

    Expected: /runner/_work/ledger-live/ledger-live/apps/ledger-live-desktop/tests/artifacts/test-results/services-earn-Earn-smoke-retry1/earn-app-opened-expected.png
    Received: /runner/_work/ledger-live/ledger-live/apps/ledger-live-desktop/tests/artifacts/test-results/services-earn-Earn-smoke-retry1/earn-app-opened-actual.png
        Diff: /runner/_work/ledger-live/ledger-live/apps/ledger-live-desktop/tests/artifacts/test-results/services-earn-Earn-smoke-retry1/earn-app-opened-diff.png

      58 |     await expect
      59 |       .soft(page)
    > 60 |       .toHaveScreenshot("earn-app-opened.png", { mask: [page.locator("webview")] });
         |        ^
      61 |   });
      62 | });
      63 |

        at /runner/_work/ledger-live/ledger-live/apps/ledger-live-desktop/tests/specs/services/earn.spec.ts:60:8
        at /runner/_work/ledger-live/ledger-live/apps/ledger-live-desktop/tests/specs/services/earn.spec.ts:51:3
});
});

0 comments on commit a0de0e0

Please sign in to comment.