Skip to content

Commit

Permalink
delete-orphaned-images option
Browse files Browse the repository at this point in the history
  • Loading branch information
@rohanmars
rohanmars committed Sep 16, 2024
1 parent 6448a08 commit 98b4022
Show file tree
Hide file tree
Showing 13 changed files with 302 additions and 138 deletions.
12 changes: 12 additions & 0 deletions .github/workflows/ci.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -129,6 +129,11 @@ jobs:
folder: '6c_partial-images'
delete-partial-images: true

- title: '6d - Orphaned Images'
purpose: 'Deletes Orphaned images'
folder: '6d_orphaned-images'
delete-orphaned-images: true

- title: '7a - Single: Untagging Images'
purpose: 'Untagging multi tagged images (single arch)'
folder: '7a_single-untagging'
Expand Down Expand Up @@ -214,6 +219,11 @@ jobs:
DELETE_PARTIAL_IMAGES="${{ matrix.tests.delete-partial-images }}"
fi
DELETE_ORPHANED_IMAGES=""
if [ ! -z "${{ matrix.tests.delete-orphaned-images }}" ]; then
DELETE_ORPHANED_IMAGES="${{ matrix.tests.delete-orphaned-images }}"
fi
KEEP_N_TAGGED=""
if [ ! -z "${{ matrix.tests.keep-n-tagged }}" ]; then
KEEP_N_TAGGED="${{ matrix.tests.keep-n-tagged }}"
Expand Down Expand Up @@ -263,6 +273,7 @@ jobs:
echo "DELETE_UNTAGGED=$DELETE_UNTAGGED" >> $GITHUB_ENV
echo "DELETE_GHOST_IMAGES=$DELETE_GHOST_IMAGES" >> $GITHUB_ENV
echo "DELETE_PARTIAL_IMAGES=$DELETE_PARTIAL_IMAGES" >> $GITHUB_ENV
echo "DELETE_ORPHANED_IMAGES=$DELETE_ORPHANED_IMAGES" >> $GITHUB_ENV
echo "KEEP_N_TAGGED=$KEEP_N_TAGGED" >> $GITHUB_ENV
echo "KEEP_N_UNTAGGED=$KEEP_N_UNTAGGED" >> $GITHUB_ENV
echo "EXCLUDE_TAGS=$EXCLUDE_TAGS" >> $GITHUB_ENV
Expand All @@ -286,6 +297,7 @@ jobs:
delete-untagged: ${{ env.DELETE_UNTAGGED }}
delete-ghost-images: ${{ env.DELETE_GHOST_IMAGES }}
delete-partial-images: ${{ env.DELETE_PARTIAL_IMAGES }}
delete-orphaned-images: ${{ env.DELETE_ORPHANED_IMAGES }}
keep-n-tagged: ${{ env.KEEP_N_TAGGED }}
keep-n-untagged: ${{ env.KEEP_N_UNTAGGED }}
exclude-tags: ${{ env.EXCLUDE_TAGS }}
Expand Down
93 changes: 83 additions & 10 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,30 @@ simulate the cleanup action but will not delete any images/packages. This is
especially important when using a wildcard/regular expression syntaxes or the
`older-than` option to select images.

## How It Works

The high level processing of the action occurs as follows:

1. For each package.
1. Download all package metadata and their manifests and put them into a working
'filter set'.
1. Remove all child images from the working filter set (including referrers and
cosign images).
1. Remove `exclude-tags` images from filter set.
1. Remove images which are younger than the `older-than` option from the filter
set.
1. Stage for deletion `delete-tags` images present in the filter set.
1. Stage for deletion `delete-ghost-images`, `delete-partial-iamges` and
`delete-orphaned-images` images present in the filter set.
1. Process `keep-n-tagged` images from the filter set, stage remainder tagged
images for deletion.
1. Process `keep-n-untagged` images from the filter set, stage remainder
untagged images for deletion.
1. Or process `delete-untagged`, staging all untagged images in filter set for
deletion.
1. Preform the deletion on all staged packages, including their children if
present.

## Action Options

### Repository Options
Expand All @@ -99,16 +123,17 @@ automatically set from the project environment where the action is running.

### Cleanup Options

| Option | Required | Defaults | Description |
| --------------------- | :------: | --------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| delete-tags | no | | Comma-separated list of tags to delete (supports wildcard syntax), can be abbreviated as `tags`. A regular expression selector can be used instead by setting the `use-regex` option to true |
| exclude-tags | no | | Comma-separated list of tags strictly to be preserved/excluded from deletion (supports wildcard syntax). A regular expression selector can be used instead by setting the `use-regex` option to true |
| delete-untagged | no | depends\* | Delete all untagged images |
| keep-n-untagged | no | | Number of untagged images to keep, sorted by date, keeping the latest |
| keep-n-tagged | no | | Number of tagged images to keep, sorted by date, keeping the latest |
| delete-ghost-images | no | false | Delete multi-architecture images where all underlying platform images are missing |
| delete-partial-images | no | false | Delete multi-architecture images where some (but not all) underlying platform images are missing |
| older-than | no | | Only include images for processing that are older than this interval (eg 5 days, 6 months or 1 year) |
| Option | Required | Defaults | Description |
| ---------------------- | :------: | --------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| delete-tags | no | | Comma-separated list of tags to delete (supports wildcard syntax), can be abbreviated as `tags`. A regular expression selector can be used instead by setting the `use-regex` option to true |
| exclude-tags | no | | Comma-separated list of tags strictly to be preserved/excluded from deletion (supports wildcard syntax). A regular expression selector can be used instead by setting the `use-regex` option to true |
| delete-untagged | no | depends\* | Delete all untagged images |
| keep-n-untagged | no | | Number of untagged images to keep, sorted by date, keeping the latest |
| keep-n-tagged | no | | Number of tagged images to keep, sorted by date, keeping the latest |
| delete-ghost-images | no | false | Delete multi-architecture images where all underlying platform images are missing |
| delete-partial-images | no | false | Delete multi-architecture images where some (but not all) underlying platform images are missing |
| delete-orphaned-images | no | false | Delete tagged images which have no parent (e.g. referrers and cosign tags missing their parent) |
| older-than | no | | Only include images for processing that are older than this interval (eg 5 days, 6 months or 1 year) |

\* If no delete or keep options are set on the action then the action defaults
the option `delete-untagged` to "true" and will delete all untagged images.
Expand Down Expand Up @@ -242,6 +267,13 @@ jobs:
token: ${{ secrets.GITHUB_TOKEN }}
```

### `delete-orphaned-images`

This option removes tagged images where the assoicated parent image does not
exist. It searches for images with tags starting with "sha256-" and then
searches for the equivalent sha256: digest. If an image digest is not found then
its flagged for deletion. This picks up orphaned referrers and cosign images.

## Keep Options

### `keep-n-untagged`
Expand Down Expand Up @@ -285,6 +317,9 @@ jobs:
token: ${{ secrets.GITHUB_TOKEN }}
```

This option operates on all tagged entries. To narrow it's scope use the
`exclude-tag` option also.

## Personal Access Tokens (PAT's)

The default injected token (`secret.GITHUB_TOKEN`) is sufficient for packages
Expand Down Expand Up @@ -345,6 +380,9 @@ jobs:
token: ${{ secrets.GHCR_PAT }}
```

Multiple package execution can also be achived by using the GitHub workflow
matrix mechanism.

## Sample Action Setups

### Complex example `keep-n-tagged`
Expand All @@ -369,6 +407,29 @@ jobs:
token: ${{ secrets.GITHUB_TOKEN }}
```

### Delete all untagged images and keep 3 latest (rc) images

This sample operates on multiple packages and makes use of a regular expression
selector. It excludes all versioned images and the latest and main tags from
processing.

```yaml
cleanup-images:
name: cleanup-images
runs-on: ubuntu-latest
concurrency:
group: cleanup-images
steps:
- uses: dataaxiom/ghcr-cleanup-action@v1
with:
packages: 'tiecd/k8s,tiecd/okd,tiecd/gke,tiecd/eks,tiecd/aks,tiecd/node18,tiecd/node20'
delete-untagged: true
keep-n-tagged: 3
exclude-tags: "^\\d+\\.\\d+\\.\\d+$|^latest$|^main$"
use-regex: true
token: ${{ secrets.GITHUB_TOKEN }}
```

### Delete an image when a pull request is closed

```yaml
Expand Down Expand Up @@ -451,6 +512,18 @@ deleted it requires that no other package publishing or deleting process is
occurring at the same time. It's recommended to use a GitHub concurrency group
with this action in complex/busy repositories.

```yaml
cleanup-images:
name: cleanup-images
runs-on: ubuntu-latest
concurrency:
group: cleanup-images
steps:
- uses: dataaxiom/ghcr-cleanup-action@v1
with:
token: ${{ secrets.GITHUB_TOKEN }}
```

### Validate Option

Set the `validate` option to true to enable a full scan of the image repository
Expand Down
6 changes: 6 additions & 0 deletions action.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,12 @@ inputs:
platform images are missing (true/false) Default: false
required: false

delete-orphaned-images:
description: >
Delete orphaned images (referrers/cosign etc) where the parent image
doesn't exist. Uses a tagged based check.
required: false

older-than:
description: >
Only include packages for processing that are older than this value. Use
Expand Down
13 changes: 11 additions & 2 deletions citester/index.js
View file
Original file line number Diff line number Diff line change
Expand Up @@ -37935,6 +37935,7 @@ class Config {
deleteUntagged;
deleteGhostImages;
deletePartialImages;
deleteOrphanedImages;
keepNuntagged;
keepNtagged;
dryRun;
Expand All @@ -37951,8 +37952,8 @@ class Config {
throttle: {
onRateLimit: (retryAfter, options, octokit, retryCount) => {
lib_core.info(`Octokit - request quota exhausted for request ${options.method} ${options.url}`);
if (retryCount < 1) {
// only retries once
if (retryCount < 3) {
// try upto 3 times
lib_core.info(`Octokit - retrying after ${retryAfter} seconds!`);
return true;
}
Expand Down Expand Up @@ -38106,6 +38107,7 @@ function buildConfig() {
!core.getInput('delete-tags') &&
!core.getInput('delete-ghost-images') &&
!core.getInput('delete-partial-images') &&
!core.getInput('delete-orphaned-images') &&
!core.getInput('keep-n-untagged') &&
!core.getInput('keep-n-tagged')) {
config.deleteUntagged = true;
Expand All @@ -38120,6 +38122,9 @@ function buildConfig() {
if (core.getInput('delete-partial-images')) {
config.deletePartialImages = core.getBooleanInput('delete-partial-images');
}
if (core.getInput('delete-orphaned-images')) {
config.deleteOrphanedImages = core.getBooleanInput('delete-orphaned-images');
}
if (core.getInput('dry-run')) {
config.dryRun = core.getBooleanInput('dry-run');
if (config.dryRun) {
Expand Down Expand Up @@ -38189,6 +38194,9 @@ function buildConfig() {
if (config.deletePartialImages !== undefined) {
optionsMap.add('delete-partial-images', `${config.deletePartialImages}`);
}
if (config.deleteOrphanedImages !== undefined) {
optionsMap.add('delete-orphaned-images', `${config.deleteOrphanedImages}`);
}
if (config.keepNtagged !== undefined) {
optionsMap.add('keep-n-tagged', `${config.keepNtagged}`);
}
Expand Down Expand Up @@ -43984,6 +43992,7 @@ class Registry {
// upgrade token
let putToken;
const auth = lib_axios.create();
esm(auth, { retries: 3 });
try {
await auth.put(`https://ghcr.io/v2/${this.config.owner}/${this.targetPackage}/manifests/${tag}`, manifest, config);
}
Expand Down
2 changes: 1 addition & 1 deletion citester/index.js.map
View file

Large diffs are not rendered by default.

Loading

0 comments on commit 98b4022

Please sign in to comment.