Skip to content

Commit

Permalink
Maintain and upload artifacts index (#2839)
Browse files Browse the repository at this point in the history
* Maintain and upload artifacts index

Make the artifacts browsable by maintaining a list of builds. This keeps
it up-to-date even when deleting images from the object storage, and
minimizes queries to the object storage.

* Add favicon

* Apply suggestions from code review

Co-authored-by: Joakim Sørensen <joasoe@gmail.com>

* Move index update outside of the build Matrix

* Add error handling and styling

* Exclude index files

* Add cache flush

* Use separate prefix for indexes

This allows to filter by prefix when generating the main index. Since
the list-objects-v2 is limited to 1000 entries, this will be a bottle
neck soon. Separating indexes allows to support up to 1000 nightly
builds.

* Add missing backslash

* Use cp and fix index format

* Sync index.html as well

* Move OS artifacts index file to root directory

This is not really GitHub related, so it shouldn't live in there.

* Adjust URL for dev builds

---------

Co-authored-by: Joakim Sørensen <joasoe@gmail.com>
  • Loading branch information
agners and ludeeus authored Oct 26, 2023
1 parent ae42fbf commit 9e70ebe
Show file tree
Hide file tree
Showing 3 changed files with 118 additions and 2 deletions.
36 changes: 35 additions & 1 deletion .github/workflows/build.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -202,17 +202,24 @@ jobs:

- name: Upload artifacts
if: ${{ github.event_name != 'release' }}
working-directory: output/images/
env:
AWS_ACCESS_KEY_ID: ${{ secrets.R2_OS_ARTIFACTS_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.R2_OS_ARTIFACTS_KEY }}
run: |
aws s3 sync \
output/images/ \
./ \
s3://${{ secrets.R2_OS_ARTIFACTS_BUCKET }}/${{ needs.prepare.outputs.version_full }}/ \
--exclude "*" \
--include "haos_*" \
--endpoint-url ${{ secrets.R2_OS_ARTIFACTS_ENDPOINT }}

ls haos_* | jq --raw-input | jq --slurp > "${{ needs.prepare.outputs.version_full }}.json"
aws s3 cp \
"${{ needs.prepare.outputs.version_full }}.json" \
s3://${{ secrets.R2_OS_ARTIFACTS_BUCKET }}/indexes/ \
--endpoint-url ${{ secrets.R2_OS_ARTIFACTS_ENDPOINT }}

- name: Upload release assets
if: ${{ github.event_name == 'release' }}
uses: shogo82148/actions-upload-release-asset@v1
Expand Down Expand Up @@ -255,6 +262,33 @@ jobs:
email: ${{ secrets.GIT_EMAIL }}
token: ${{ secrets.GIT_TOKEN }}

- name: Regenerate artifacts index
if: ${{ github.event_name != 'release' }}
env:
AWS_ACCESS_KEY_ID: ${{ secrets.R2_OS_ARTIFACTS_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.R2_OS_ARTIFACTS_KEY }}
run: |
aws s3api list-objects-v2 \
--bucket "${{ secrets.R2_OS_ARTIFACTS_BUCKET }}" \
--endpoint-url "${{ secrets.R2_OS_ARTIFACTS_ENDPOINT }}" \
--prefix "indexes/" \
--query 'Contents[].Key' | jq 'map(capture("indexes/(?<version>[[:digit:]].+).json").version) | sort' > .os-artifacts/index.json
aws s3 sync \
.os-artifacts/ \
s3://${{ secrets.R2_OS_ARTIFACTS_BUCKET }}/ \
--endpoint-url ${{ secrets.R2_OS_ARTIFACTS_ENDPOINT }}

- name: Flush CloudFlare cache
run: |
curl --silent --show-error --fail -X POST \
"https://api.cloudflare.com/client/v4/zones/${{ secrets.CF_ZONE }}/purge_cache" \
-H "Authorization: Bearer ${{ secrets.CF_TOKEN }}" \
-H "Content-Type: application/json" \
--data '{"files": [
"https://os-artifacts.home-assistant.io/index.html",
"https://os-artifacts.home-assistant.io/index.json"
] }'

- name: Bump Home Assistant OS ${{ needs.prepare.outputs.channel }} channel version
uses: home-assistant/actions/helpers/version-push@master
with:
Expand Down
82 changes: 82 additions & 0 deletions .os-artifacts/index.html
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
<!DOCTYPE html>
<html>
<head>
<title>Home Assistant OS - development builds</title>
<link rel="shortcut icon" href="https://brands.home-assistant.io/homeassistant/icon.png">
<style>
body {
font-family: 'Trebuchet MS', 'Lucida Sans Unicode', 'Lucida Grande', 'Lucida Sans', Arial, sans-serif;
}

.error {
color: maroon;
}
</style>
</head>
<body>
<h1>Home Assistant OS - development builds</h1>
<select id="os-builds"></select>
<div id="os-builds-list"></div>
<script>
const buildSelect = document.getElementById('os-builds');
const osBuildsList = document.getElementById('os-builds-list');

const fillVersions = async () => {
try {
const response = await fetch('/index.json');
if (!response.ok) {
const p = document.createElement('p');
p.className = "error";
p.textContent = "Could not load version index file.";
osBuildsList.appendChild(p);
return;
}
const items = await response.json();
items.reverse();

items.forEach(buildVersion => {
buildSelect.appendChild(new Option(buildVersion, buildVersion));
});

buildSelect.dispatchEvent(new Event('change'));
} catch (error) {
console.error('Error fetching data:', error);
}
}

buildSelect.addEventListener('change', async function() {
osBuildsList.innerHTML = '';

const buildVersion = this.value;

try {
const response = await fetch(`/indexes/${buildVersion}.json`);
if (!response.ok) {
const p = document.createElement('p');
p.className = "error";
p.textContent = `Could not load index file for version ${buildVersion}.`;
osBuildsList.appendChild(p);
return;
}
const images = await response.json();

const ul = document.createElement('ul');
images.forEach(image => {
const li = document.createElement('li');
const a = document.createElement('a');
a.href =`/${buildVersion}/${image}`;
a.textContent = image;
li.appendChild(a);
ul.appendChild(li);
});

osBuildsList.appendChild(ul);
} catch (error) {
console.error('Error fetching images:', error);
}
});

fillVersions();
</script>
</body>
</html>
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,4 +55,4 @@ The Home Assistant Operating System documentation can be found on the [Home Assi

The Development build GitHub Action Workflow is a manually triggered workflow
which creates Home Assistant OS development builds. The development builds are
available at [os-builds.home-assistant.io](https://os-builds.home-assistant.io/).
available at [https://os-artifacts.home-assistant.io/index.html](https://os-artifacts.home-assistant.io/index.html).

0 comments on commit 9e70ebe

Please sign in to comment.