Merge pull request #97 from OasisDEX/readme-updates-2

README file updates and fixes for flaky tests

README.md

@@ -6,20 +6,30 @@
 
 # Repository with E2E tests for Summer.fi DeFi web application
 
-## Introduction
+1. [ Introduction ](#1-introduction)
+2. [ Installation ](#2-installation)
+3. [ Environment variables](#3-environment-variables)
+4. [ Browsers ](#4-browsers)
+5. [ Running the tests ](#5-running-the-tests)
+6. [ Test reports ](#6-test-reports)
+7. [ Tags ](#7-tags)
+8. [ Projects ](#8-projects)
+9. [ License ](#9-license)
+
+## 1. Introduction
 This repository contains a repository of tests for the e2e testing of the Summer.fi DeFi web application.
 
 There are two main categories of e2e tests:
 - **No wallet**: 
 Tests without a wallet connected. 
 - **With wallet**: 
-Tests with a wallet (and a Tenderly fork) connected.
+Tests with a wallet connected (Metamask) and, in most of the cases, with a Tenderly fork.
 
 The tests and framework have been written using Playwright and Typescript. 
-- These 'No wallet' tests use Playwright only.
+- The 'No wallet' tests use Playwright only.
 - The 'With wallet' tests use Playwright, Synpress (for Metamask wallet) and Tenderly APIs (for forks).
 
-## Installation
+## 2. Installation
 First, install the programs required to run the application:
 
 - [Node.js](https://nodejs.org/en/download/)
@@ -36,17 +46,13 @@ yarn install
 ```
 
 
-## Environment variables
+## 3. Environment variables
 To run the tests locally you will need to create a `.env` file. You can create a copy from `.env.example` file by running: 
 ```
 cp .env.example .env
 ```
 
-The following environment variable is **mandatory** for both all tests:
-
-| Variable | Description |
-| --- | --- |
-| BASE_URL | This is the base url used to run the tests. As per `.env.example` file, you'll probably use `https://staging.summer.fi` as base url|
+There aren't any mandatory environment variables for **no wallet** tests.
 
 The following environment variables are **mandatory** for **with wallet** tests:
 
@@ -61,25 +67,25 @@ You can also use these other environment variables:
 
 | Variable | Default | Description |
 | --- | --- | -- |
+| BASE_URL | This is the base url used to run the tests. As per `.env.example` file. If you don't set this variable the tests will run with `https://staging.summer.fi` as base url|
 | FULLY_PARALLEL | false | Playwright Test runs tests in parallel. In order to achieve that, it runs several worker processes that run at the same time. By default, **test files** are run in parallel. However, tests in a single file are run in order, in the same worker process. You can configure entire test run to concurrently execute all tests in all files using this option. Note that **with wallet** tests can run **only** serial mode, so this variable will be applied only to **no wallet tests**. |
 | WORKERS | 1 | Number of workers used to run the tests. To run the tests in pralallel you need to use more than 1 workers. Note that **with wallet** tests can run **only** with 1 worker, so this variable will be applied only to **no wallet tests**. |
 | TIMEOUT | 30000 | Timeout for each test in milliseconds. Note that some timeouts have been defined in `./utils/config.ts` file, so this TIMEOUT variable might not have any effect in some cases. |
-| RETRIES | 0 | The maximum number of retry attempts given to failed tests. By default failing tests are not retried. |
+| RETRIES | 2 | The maximum number of retry attempts given to failed tests. By default failing tests are not retried. |
 | HEADLESS | true | Whether to run browser in headless mode. Note that **with wallet** tests can run **only** in headed mode, so this variable will be applied only to **no wallet tests**. |
 | FLAGS | '' | You can pass one or more flags to enable hidden features in staging and dev environments. To pass more than one flag separate them with an empty space: `'flag1:true flag2:false'` |
 
-## Browsers
+## 4. Browsers
 For now the tests have been setup so that they run only on desktop Chrome.
 
-## Running the tests
+## 5. Running the tests
 **Note** that in oder to considerably reduce the test execution time, several scripts have been created so that **with wallet** tests can be run in parallel jobs (docker containers) in GitHub actions.
 
 1. Create `.env` file.
 
-2. Set `BASE_URL` in `.env` file. For example,  `BASE_URL='https://staging.summer.fi'`.
-For running **with wallet** tests also set `WALLET_ADDRESS`, `TENDERLY_PROJECT`, `TENDERLY_USER` and `TENDERLY_ACCESS_KEY` variables.
+2. For running **with wallet** tests, set `WALLET_ADDRESS`, `TENDERLY_PROJECT`, `TENDERLY_USER` and `TENDERLY_ACCESS_KEY` variables.
 
-3. Run tests by running:
+3. Run tests by running, for example:
 - Run all tests:
 `yarn test:e2e`
 - Run all **regression** tests:
@@ -93,19 +99,26 @@ For running **with wallet** tests also set `WALLET_ADDRESS`, `TENDERLY_PROJECT`,
 - Run **with wallet regression** tests:
 `yarn with-wallet:all:regression`
 
-## Running the tests report
-After running one or more tests you can run the following commands to open one or more test reports in the browser:
+Check all the scripts in [ package.json ](https://github.com/OasisDEX/e2e-tests/blob/main/package.json) file
+
+## 6. Test reports
+> The test reports for the test runs executed in gitHub can be found in https://github.com/OasisDEX/e2e-tests/actions.
+> 1. Click on the test run you wish to see the report for.
+> 2. Scroll down to the **Artifacts** section and click on the report name you wish to download.
+> 3. Unzip the report file you just downloaded and double click on the **index.html** file to open the report in the browser.  
+
+After running one or more tests locally you can run the following commands to open test reports in the browser:
 - `yarn test:report:no-wallet`
 - `yarn test:report:with-wallet:aave:ethereum`
 - `yarn test:report:with-wallet:aave:other`
 - `yarn test:report:with-wallet:maker-and-spark`  
 
-## Tags 
-Tests can be tagged so that we  can only run the tests that have the certain tag. 
+## 7. Tags 
+Tests can be tagged so that we can run only the tests that have the certain tag. 
 At the moment we are using the following tags:
 - `@regression`: Add this tag to all the tests that should be part of the regression test suite.
 
-## Projects
+## 8. Projects
 A project is a logical group of tests running with the same configuration. For now we are using projects so that we can achieve the two following goals:
 - Applying different configuration to **no wallet** and **with wallet** tests.
 - Splitting **with wallet** tests so that they can run in parallel containers, and so reducing significantly the test execution time.
@@ -116,7 +129,7 @@ At the moment we are using the following projects:
 - `with-wallet-aave-ethereum`
 - `with-wallet-maker-and-spark`
 
-## License
+## 9. License
 
 Copyright (C) 2021 Oazo Apps Limited, Licensed under the Apache License, Version 2.0 (the
 "License"); you may not use this file except in compliance with the License. You may obtain a copy

tests/noWallet/aaveV3/base/aaveV3EarnBase.spec.ts

@@ -79,7 +79,7 @@ test.describe('Aave v3 Earn Base', async () => {
 		});
 	});
 
-	test('It should allow to simulate an Aave V3 Base Earn position before opening it - Adjust risk - Up and Down  - No wallet connected @regression', async ({
+	test('It should allow to simulate an Aave V3 Base Earn position before opening it - Adjust risk - Up and Down - No wallet connected @regression', async ({
 		app,
 	}) => {
 		test.info().annotations.push({
@@ -111,7 +111,7 @@ test.describe('Aave v3 Earn Base', async () => {
 			exactAmount: true,
 		});
 		await app.position.setup.shouldHaveLiquidationPrice({
-			amount: '[0-3].[0-9]{3,4} CBETH',
+			amount: '[0-5].[0-9]{3,4} CBETH',
 		});
 
 		await app.position.overview.shouldHaveLiquidationPriceAfterPill('[0-3].[0-9]{2}');
@@ -129,10 +129,10 @@ test.describe('Aave v3 Earn Base', async () => {
 			exactAmount: true,
 		});
 		await app.position.setup.shouldHaveLiquidationPrice({
-			amount: '[2-6].[0-9]{3,4} CBETH',
+			amount: '[0-9]{1,2}.[0-9]{1,2}([0-9]{1,2})? CBETH',
 		});
 
-		await app.position.overview.shouldHaveLiquidationPriceAfterPill('[2-6].[0-9]{2}');
+		await app.position.overview.shouldHaveLiquidationPriceAfterPill('[0-9]{1,2}.[0-9]{2}');
 		const updatedLiqPrice2 = await app.position.manage.getLiquidationPrice();
 		const updatedLoanToValue2 = await app.position.manage.getLoanToValue();
 		expect(updatedLiqPrice2).toBeGreaterThan(updatedLiqPrice);

tests/withWallet/aaveV3/ethereum/aaveV3EarnEthereum.spec.ts

@@ -83,7 +83,7 @@ test.describe('Aave V3 Earn - Ethereum - Wallet connected', async () => {
 		await app.position.manage.ok();
 
 		await app.position.overview.shouldHaveNetValue({
-			value: '[0-9]{2}.[0-9]{2}ETH',
+			value: '[0-9]{2}(.[0-9]{1,2})?ETH',
 		});
 		await app.position.overview.shouldHaveTotalCollateral({ amount: '20.00000', token: 'WSTETH' });
 	});