Run only tests relevant to the changeset.
“Insanity is doing the same thing over and over and expecting different results.”
Albert Einstein, probably
Imagine we have the following dependencies structure:
If the 📦Login module is changed, it would only affect the 📦LoginUI and the 📱MainApp.
Does it make sense to test all the modules if we know only the 📦Login module is changed? No. We can only run 50% of the tests and get the same results.
This technique saves time when testing locally and on the CI.
- Your project must have multiple targets or modules
Add to Xcode as SPM dependency.
- Open your project or workspace in Xcode
- Select yout project in the file list in Xcode
- In the right menu select "Project", open tab "Package Dependencies"
- Select "+"
- In the new window, paste
git@github.com:mikeger/XcodeSelectiveTestingin the search field - Select project if necessary, put a checkbox on "XcodeSelectiveTesting" in the list
- Click "Add Package"
Alternatively, you can use a prebuilt binary release of the tool distributed under releases section.
Add .package(url: "git@github.com:mikeger/XcodeSelectiveTesting", .upToNextMajor(from: "0.10.4")) to your Package.swift's dependencies section.
Use SPM to run the command: swift run xcode-selective-test.
Alternatively, you can use a prebuilt binary release of the tool distributed under releases section.
Using Mint
mint install mikeger/XcodeSelectiveTesting@0.10.4
- Checkout this repository
- Compile the tool:
swift build -c release
In case you are using Swift Package Manager without Xcode project or workspace:
Run swift test --filter "$(swift run xcode-selective-test . --json | jq -r ". | map(.name) | join(\"|\")")"
NB: This command assumes you have jq tool installed. You can install it with Homebrew via brew install jq.
- Install the tool (see Installation: Xcode)
- Select your project in the Xcode's file list
- Right-click on it and select
SelectiveTestingPlugin - Wait for the tool to run
- Run tests normally, SelectiveTesting would modify your test plan according to the local changes
Alternatively, you can use CLI to achieve the same result:
- Run
mint run mikeger/XcodeSelectiveTesting@0.10.4 YourWorkspace.xcworkspace --test-plan YourTestPlan.xctestplan - Run tests normally, XcodeSelectiveTesting would modify your test plan according to the local changes
- Requires jq installed (
brew install jq)
- Add code to install the tool
- Use xcodebuild to run only selected tests:
xcodebuild test -workspace Workspace.xcworkspace -scheme Scheme $(mint run --silent XcodeSelectiveTesting@provide-if-target-is-test-target --json | jq -r "[.[] | select(.testTarget == true)] | map(\"-only-testing:\" + .name) | join(\" \")")
- Add code to install the tool
- Add a CI step before you execute your tests:
mint run mikeger/XcodeSelectiveTesting@0.10.4 YourWorkspace.xcworkspace --test-plan YourTestPlan.xctestplan --base-branch $PR_BASE_BRANCH - Execute your tests
Git allows us to find what files were touched in the changeset.
Root
├── Dependencies
│ └── Login
│ ├── ❗️LoginAssembly.swift
│ └── ...
├── MyProject.xcodeproj
└── SourcesGoing from the project to its dependencies, to its dependencies, to dependencies of the dependencies, ...
Dependencies between packages can be parsed with swift package dump-package and dependencies between Xcode projects and targets can be parsed with XcodeProj.
BTW, This is the moment your Leetcode graph exercises would pay off
This is important, so we'll know which files affect which targets.
Go from every changed dependency all the way up, and save a set of dependencies you've touched.
This is the hardest part: dealing with obscure Xcode formats. But if we get that far, we will not be scared by 10-year-old XMLs.
--help: Display all command line options--base-branch: Branch to compare against to find the relevant changes. If emitted, a local changeset is used (development mode).--test-plan: Path to the test plan. If not given, tool would try to infer the path.--json: Provide output in JSON format (STDOUT).--dependency-graph: Opens Safari with a dependency graph visualization. Attention: if you don't trust Javascript ecosystem prefer using--dotoption. More info here.--dot: Output dependency graph in Dot (Graphviz) format. To be used with Graphviz:brew install graphviz, thenxcode-selective-test --dot | dot -Tsvg > output.svg && open output.svg--turbo: Turbo mode: run tests only for directly affected targets.--verbose: Provide verbose output.
It is possible to define the configuration in a separate file. The tool would look for this file in the current directory.
Options available are (see selective-testing-config-example.yml for an example):
basePath: Relative or absolute path to the project. If set, the command line option can be emitted.testPlan: Relative or absolute path to the test plan to configure.exclude: List of relative paths to exclude when looking for Swift packages.extra/dependencies: Options allowing to hint tool about dependencies between targets or packages.extra/targetsFiles: Options allowing to hint tool about the files affecting targets or packages.
Supported operating systems:
- macOS 12.0+ (Monterey) : Xcode 14.2 and above
- Linux: Swift 5.8 and above
Contributions are welcome. Consider checking existing issues and creating a new one if you plan to contribute.
See LICENSE
- 🇺🇦 Michael Gerasymenko <mike (at) gera.cx>
If you like this product, consider donating to my hometown's charity project Monsters Corporation 🤝