feat: new rafiki release pipeline and nodejs bump to v20 #2510

golobitch · 2024-03-07T15:47:10Z

Changes proposed in this pull request

Update NodeJS version from v18 to v20
- v18.x is Maintenance LTS and it will end on 2025-04-30
- v20.x is Active LTS
New Rafiki release pipeline
- Do everything that now has been done
- Run docker image scan for any known vulnerabilities and break build if severity is HIGH or CRITICAL
- Publish docker image to GitHub repository
- Create tag and release in GitHub

Context

Rafiki release pipeline is now designed that is possible to do quick deploys.

Branching strategy

For every major / minor version change, you need to create new release branch, but not for patch change. If you want to create release v2.1.0, you would need to create following release branch: release/v2.1.0. When there will be push to this repository, patch number will increment. Ideally, when release branch is created, there should be feature freeze, so that we do not need to update minor version. Instead, in release branch, we should push (after creation), only bugfixes, to make this release stable.

Following release branches are supported:

release/v1.0.0
release/v1.0.0-beta
release/v1.0.0-alpha

Also, correct corresponding tags and releases will be created in GitHub.

On branch release/v1.0.0-beta, first tag and release would be v1.0.0-beta and second one v1.0.1-beta

Docker build, scan, publish and release creation will be triggered only on release branches.

For the documentation on branching strategy, tags, etc, I will create new PR with updated documentation.

Checklist

netlify · 2024-03-07T15:47:27Z

❌ Deploy Preview for brilliant-pasca-3e80ec failed.

Name	Link
🔨 Latest commit	`aaf9d49`
🔍 Latest deploy log	https://app.netlify.com/sites/brilliant-pasca-3e80ec/deploys/660d2f57828c9c0008dfa2b2

.github/workflows/pr_title_check.yml

.github/workflows/node-build.yml

sabineschaller · 2024-03-08T08:53:45Z

.github/workflows/node-build.yml

+        uses: actions/checkout@v4
+      - name: Generate CHANGELOG data
+        id: changelog
+        uses: requarks/changelog-action@v1


This creates the changelog based on the template, correct?

This actually generates changelog data based on commits between current and previous same tag. It does not create changelog. If you want, I can also generate CHANGELOG.md here or append the data to it?

Do you know how it orders the commits in the generated changelog?
Right now, we have the following defined: https://github.com/interledger/rafiki/blob/main/.github/release.yaml

Yes, if commit has note BREAKING CHANGE then this breaking changes will be always on top. After that, there will be bug fixes, features, chores, etc. I just need to check for dependencies.

@sabineschaller I'm not sure if this action takes into account the label of the PRs and the breaking > general changes > dependency updates order, since it checks commits and not PRs.

That being said, it does look like though we should get into a habit of doing feat!: ... (or adding BREAKING CHANGE to the to-be squashed commit), so we could see & auto generate breaking changes just from the commits

@mkurapov I think it needs to be in the to-be-squashed commit either way, so we need to start adding the !. And we can try training people to add the BREAKING CHANGE footer.

sabineschaller · 2024-03-08T12:50:56Z

Also, I keep forgetting to increase the version in the package.json. I was thinking about just removing it

chore: remove version from monorepo package.json #2477

but the frontend displays that version so we may have to keep it.
Long story short, can we add a check in the workflow that checks whether the version has been bumped on the release branch?

golobitch · 2024-03-09T19:02:41Z

Also, I keep forgetting to increase the version in the package.json. I was thinking about just removing it

chore: remove version from monorepo package.json #2477

but the frontend displays that version so we may have to keep it. Long story short, can we add a check in the workflow that checks whether the version has been bumped on the release branch?

We can maybe just update the package.json when we are creating changelog data? But that would be only on the release branch.

Action can create PR into main branch with version update in package.json. Should we go with that?

github-advanced-security · 2024-03-10T09:32:01Z

This pull request sets up GitHub code scanning for this repository. Once the scans have completed and the checks have passed, the analysis results for this pull request branch will appear on this overview. Once you merge this pull request, the 'Security' tab will show more code scanning analysis results (for example, for the default branch). Depending on your configuration and choice of analysis tool, future pull requests will be annotated with code scanning analysis results. For more information about GitHub code scanning, check out the documentation.

sabineschaller · 2024-03-11T13:56:18Z

We can maybe just update the package.json when we are creating changelog data? But that would be only on the release branch.

Never mind, the team decided to remove the version.

mkurapov · 2024-03-13T13:31:00Z

README.md

@@ -35,9 +35,9 @@ Never heard of Interledger before? Or would you like to learn more? Here are som



While we are here, in the "Local Development Environment" section above, we can remove the nvm references and just say that the prerequisites are pnpm & node v20.

cc @sabineschaller since it looks like you edited the file

.github/workflows/node-build.yml

mkurapov · 2024-03-13T15:26:26Z

.github/workflows/node-build.yml

+          severity-cutoff: high
+          output-format: table
+
+  docker-trivy:


grype and trivy: how much do they overlap? ie is it good to keep both vs just picking one?

Well, the difference is basically in the data sources that the scanner is using. Trivy is using more data sorces than grype, however, grype is using some of the data sources that trivy is not. So basically, we can keep both, or at least, just Trivy. Your thought?

might as well have a broader check, we can keep both - just wanted to understand the differences

mkurapov · 2024-03-13T15:48:45Z

.github/workflows/node-build.yml

+        uses: actions/checkout@v4
+      - name: Generate CHANGELOG data
+        id: changelog
+        uses: requarks/changelog-action@v1


@sabineschaller I'm not sure if this action takes into account the label of the PRs and the breaking > general changes > dependency updates order, since it checks commits and not PRs.

That being said, it does look like though we should get into a habit of doing feat!: ... (or adding BREAKING CHANGE to the to-be squashed commit), so we could see & auto generate breaking changes just from the commits

.github/workflows/node-build.yml

sabineschaller · 2024-03-15T09:04:24Z

.github/workflows/node-build.yml

+        uses: docker/build-push-action@v5
+        with:
+          push: false
+          platforms: linux/amd64,linux/arm64


Can we only build on both platforms if we actually release? Otherwise, this will take forever.

We can. So basically, build only for linux/amd64 on feature branches and both platforms for release branch?

Yes, I think that is what we used to do.

mkurapov · 2024-03-15T17:42:33Z

BTW I changed the Netlify NODE_VERSION back to 18 (for the rafiki.dev documentation) for now since we have some doc changes in PRs we wanted previews on. So once this is merged in, it can be set to 20.

.github/workflows/node-build.yml

mkurapov · 2024-03-19T17:46:57Z

.github/workflows/node-build.yml

+          severity-cutoff: high
+          output-format: table
+
+  docker-trivy:


might as well have a broader check, we can keep both - just wanted to understand the differences

golobitch · 2024-03-25T16:23:20Z

@sabineschaller @mkurapov we can do another interation of a review now.

mkurapov

👍 good to go after main's merged in

sabineschaller · 2024-03-28T13:07:57Z

I'll re-review as soon as main is merged in :-)

Co-authored-by: Max Kurapov <maxkurapov@gmail.com>

… platform on non release branch

golobitch · 2024-03-29T00:05:36Z

@mkurapov @sabineschaller rebased to main & updated PR

.github/workflows/lint_test_build.yml

golobitch · 2024-04-01T14:25:19Z

@mkurapov @sabineschaller I am not authorized to merge this PR.

.github/workflows/lint_test_build.yml

mkurapov · 2024-04-02T08:23:05Z

@mkurapov @sabineschaller I am not authorized to merge this PR.

I'll give you access... ✅

golobitch · 2024-04-02T19:52:30Z

Netlify builds will fail due to node bump. After we merge this PR into master, we should also bump Netlify

mkurapov · 2024-04-02T20:37:36Z

.github/workflows/node-build.yml

@@ -134,6 +134,29 @@ jobs:
     - name: Perform CodeQL Analysis
       uses: github/codeql-action/analyze@v3

+  integration-test:


I think we should add this to the needs list for the node-build -> we should probably not run node-build if we fail integration tests.

CC @BlairCurrey

yeah I'd say so too.

mkurapov

🚀

mkurapov · 2024-04-03T10:44:54Z

@tadejgolobic We have a branch protection rule set up for all_pr_checks_passed. Since we don't have that anymore, I can change that so that it should be node-build instead.

Maybe also docker-build ?

mkurapov · 2024-04-03T10:53:33Z

@golobitch done, looks like its mergeable

tadejgolobic · 2024-04-03T11:29:06Z

@tadejgolobic We have a branch protection rule set up for all_pr_checks_passed. Since we don't have that anymore, I can change that so that it should be node-build instead.

Maybe also docker-build ?

Should I just create one more job whose name it will be all_pr_checks_passed ? That would probably be the easiest solution

mkurapov · 2024-04-03T15:14:31Z

@tadejgolobic I don't mind just adding docker-build as a separate required check in the settings, it's straightforward enough and doesn't make a big difference.

tl;dr we can merge IMO

github-actions bot added type: ci Changes to the CI pkg: backend Changes in the backend package. pkg: frontend Changes in the frontend package. pkg: auth Changes in the GNAP auth package. pkg: mock-ase type: documentation (archived) Improvements or additions to documentation labels Mar 7, 2024

sabineschaller reviewed Mar 8, 2024

View reviewed changes

sabineschaller requested a review from mkurapov March 13, 2024 10:51

mkurapov reviewed Mar 13, 2024

View reviewed changes

huijing mentioned this pull request Mar 14, 2024

chore: add feature requests to contributing.md #2542

Merged

5 tasks

sabineschaller reviewed Mar 15, 2024

View reviewed changes

mkurapov reviewed Mar 19, 2024

View reviewed changes

golobitch self-assigned this Mar 23, 2024

golobitch requested review from mkurapov and sabineschaller March 25, 2024 21:33

mkurapov previously approved these changes Mar 27, 2024

View reviewed changes

mkurapov mentioned this pull request Mar 27, 2024

feat: misc integration test improvents #2595

Merged

5 tasks

sabineschaller and others added 5 commits March 28, 2024 20:11

ci: update workflows

e7add38

docs: update planning call -> community call

3fc4b03

docs: improve wording

89aba2d

feat(workflows): change nodejs version to latest lts

f247c26

feat(workflows): add new action for release and publish

1810c67

golobitch and others added 4 commits March 28, 2024 20:13

fix(workflows): token introspection openapi specs path

8ba73aa

Co-authored-by: Max Kurapov <maxkurapov@gmail.com>

fix(workflows): auth openapi specs path

1938154

Co-authored-by: Max Kurapov <maxkurapov@gmail.com>

fix(workflows): backend openapi specs path

f982abf

Co-authored-by: Max Kurapov <maxkurapov@gmail.com>

feat(workflows): build multiple platforms only on release branch

762ae43

golobitch dismissed mkurapov’s stale review via 762ae43 March 28, 2024 19:13

golobitch added 3 commits March 28, 2024 21:42

fix(workflows): specify new job for doing a docker build for specific…

66c3398

… platform on non release branch

fix(workflows): use sha instead of refname

0563b11

fix(workflows): reorder mase steps

b02594f

golobitch requested a review from mkurapov March 29, 2024 19:34

mkurapov reviewed Mar 29, 2024

View reviewed changes

.github/workflows/lint_test_build.yml Outdated Show resolved Hide resolved

chore(workflows): remove unnecessary workflows

5c8044f

mkurapov previously approved these changes Mar 29, 2024

View reviewed changes

BlairCurrey reviewed Apr 1, 2024

View reviewed changes

.github/workflows/lint_test_build.yml Show resolved Hide resolved

fix(workflows): add missing integration test

6649d92

golobitch dismissed mkurapov’s stale review via 6649d92 April 2, 2024 18:36

golobitch requested review from BlairCurrey and mkurapov April 2, 2024 19:51

mkurapov reviewed Apr 2, 2024

View reviewed changes

fix(workflows): add integration tests as prereqs for node build

aaf9d49

mkurapov approved these changes Apr 3, 2024

View reviewed changes

sabineschaller approved these changes Apr 3, 2024

View reviewed changes

golobitch merged commit 021a7fa into interledger:main Apr 3, 2024
26 of 30 checks passed

		@@ -35,9 +35,9 @@ Never heard of Interledger before? Or would you like to learn more? Here are som

feat: new rafiki release pipeline and nodejs bump to v20 #2510

feat: new rafiki release pipeline and nodejs bump to v20 #2510

Conversation

golobitch commented Mar 7, 2024

Changes proposed in this pull request

Context

Branching strategy

Checklist

netlify bot commented Mar 7, 2024 • edited Loading

❌ Deploy Preview for brilliant-pasca-3e80ec failed.

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sabineschaller commented Mar 8, 2024 • edited Loading

golobitch commented Mar 9, 2024

github-advanced-security bot commented Mar 10, 2024

sabineschaller commented Mar 11, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mkurapov commented Mar 15, 2024

Choose a reason for hiding this comment

golobitch commented Mar 25, 2024

mkurapov left a comment

Choose a reason for hiding this comment

sabineschaller commented Mar 28, 2024

golobitch commented Mar 29, 2024

golobitch commented Apr 1, 2024

mkurapov commented Apr 2, 2024 • edited Loading

golobitch commented Apr 2, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mkurapov left a comment

Choose a reason for hiding this comment

mkurapov commented Apr 3, 2024 • edited Loading

mkurapov commented Apr 3, 2024

tadejgolobic commented Apr 3, 2024

mkurapov commented Apr 3, 2024

netlify bot commented Mar 7, 2024 •

edited

Loading

sabineschaller commented Mar 8, 2024 •

edited

Loading

mkurapov commented Apr 2, 2024 •

edited

Loading

mkurapov commented Apr 3, 2024 •

edited

Loading