kernel plugin: properly generate Ubuntu kernel .config

[arighi]

Recent Ubuntu kernels (>= Jammy) are now using a new format to store their .config, called annotations [1].

The kernel plugin is trying to assemble a .config using the old config chunks stored in debian.<kernel>/config/* that are not available anymore in the kernels that are using this new annotations model.

Instead of trying to manually assemble the config we should rely on debian/rules genconfigs that is more portable across different Ubuntu kernels and releases.

[1] https://lists.ubuntu.com/archives/kernel-team/2023-June/140230.html

• Have you followed the guidelines for contributing?
• Have you signed the CLA?
• Have you successfully run make lint?
• Have you successfully run pytest tests/unit?

---

[kubiko]

@arighi thank you! this looks good.
Can you please ensure tests pass as well?
Would this work for any kernel done from Focal onwards?

[sergiusens]

@arighi can you take a look at @kubiko's question please?

[arighi]

@arighi can you take a look at @kubiko's question please?

Working on it. I've just updated my pull request, the last one should fix all the kernel test cases.

@kubiko about the support for older kernels, yes, it will work with any kernel back to focal (and even before focal), the target debian/rules genconfigs has been supported since ... forever :) and it's the most reliable way to generate kernel .config for all the supported architectures and flavours, also with older kernels/releases.

[xnox]

@gkoh @kubiko how can arighi fix the pull request CI, if the CI doesn't start and nobody is approving it? Can workflows please be started? or should arighi submit some minor whitespace change to be allowed to trigger workflows on merge proposals?

[gkoh]

@gkoh @kubiko how can arighi fix the pull request CI, if the CI doesn't start and nobody is approving it? Can workflows please be started? or should arighi submit some minor whitespace change to be allowed to trigger workflows on merge proposals?

@xnox
This needs someone in the snapcraft team to approve.
I believe the default Github project configuration is that first-time contributors need manual approval for workflows to execute.

[gkoh]

@gkoh @kubiko how can arighi fix the pull request CI, if the CI doesn't start and nobody is approving it? Can workflows please be started? or should arighi submit some minor whitespace change to be allowed to trigger workflows on merge proposals?

@xnox This needs someone in the snapcraft team to approve. I believe the default Github project configuration is that first-time contributors need manual approval for workflows to execute.

@sergiusens Can you help move this PR along?

[xnox]

@arighi workflows were re-run, but there are now failing tests again, and rebase needed.

[arighi]

@arighi workflows were re-run, but there are now failing tests again, and rebase needed.

I'm trying to setup a dev environment to verify all these tests locally. I'm wondering if there's some documentation on how to verify everything without relying on the CI here in github, I've followed all the steps here (https://github.com/snapcore/snapcraft/blob/main/HACKING.md) and I have created a local dev environment correctly, but then the tests are failing even without my patch applied... I'll investigate more, I need to start from a local environment where everything works, then I'll apply my patch, re-run all the tests and if everything is good I'll rebase+push again.

[kubiko]

@xnox this is now rebased, I will run a test build to make sure things are still working

[kubiko]

@arighi I have struggled with this as well, not well documented. Here are my notes
on focal

$ sudo apt install -y libapt-pkg-dev python3-pip python3-virtualenv 
$ cd <snapcraft project root>
$ virtualenv .venv
$ source .venv/bin/activate
$ pip install -U pip
$ pip install -r requirements-devel.txt
$ pytest tests/unit/parts/plugins/test_kernel.py ; pytest tests/legacy/unit/plugins/v2/test_kernel.py

As part of the snapcraft release there are more tests, those can be run with ./runtests.sh
I have setup LP build for this branch here:
https://launchpad.net/~ondrak/+snap/snapcraft-kernel-initrd-split

[codecov-commenter]

Codecov Report

Merging #4210 (f8b46de) into main (5ec6572) will decrease coverage by 0.14%.
The diff coverage is n/a.

❗ Current head f8b46de differs from pull request most recent head 989b49d. Consider uploading reports for the commit 989b49d to get more accurate results

❗ Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.

@@            Coverage Diff             @@
##             main    #4210      +/-   ##
==========================================
- Coverage   89.11%   88.97%   -0.14%     
==========================================
  Files         300      295       -5     
  Lines       20623    20171     -452     
==========================================
- Hits        18378    17948     -430     
+ Misses       2245     2223      -22     

Files Changed	Coverage Δ
snapcraft_legacy/plugins/v2/_kernel_build.py	86.66% <ø> (ø)

... and 16 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[arighi]

@kubiko it looks like the tests are finally passing now. Is there anything special to do or do I just need to wait for a review/approval? Thanks.

[kubiko]

@arighi that should be all. you just need snapcraft team review now

[xnox]

I am not sure how kernel team can maintain this, without powers to run CI or merge changes.

[kubiko]

@xnox if we are open to improving Ubuntu upstream kernel packaging, I'd propose allowing running
debian/rules clean genconfigs out of the source tree, or providing an output directory. So we do not need to copy over files and clean the source tree after running the steps.

[sergiusens]

@xnox moving this repo to the canonical org is probably long overdue, that should allow running tests without this churn. In any case, until that happens, please give me a list of folks so I can add as contributors.

accident

[kubiko]

LGTM

[xnox]

@arighi

commit 5685218369e6bc93c1fd72d8d59616777d0b0e1b
Merge: f8b46de068 8758571f08
Author: Andrea Righi <r....a@gmail.com>
Date:   Mon Aug 14 11:55:48 2023 +0200

    Merge branch 'main' into kernel-config-fix

It seems the merge was done using @gmail.com address, instead of your @canonical.com address which is now tripping up the cla check. I don't know if you can rewrite that merge commit with @canonical.com address. Or if this was done using github UI to like change github's default email to @canonical.com if it isn't that for things like these.

[arighi]

@arighi

commit 5685218369e6bc93c1fd72d8d59616777d0b0e1b
Merge: f8b46de068 8758571f08
Author: Andrea Righi <r....a@gmail.com>
Date:   Mon Aug 14 11:55:48 2023 +0200

    Merge branch 'main' into kernel-config-fix

It seems the merge was done using @gmail.com address, instead of your @canonical.com address which is now tripping up the cla check. I don't know if you can rewrite that merge commit with @canonical.com address. Or if this was done using github UI to like change github's default email to @canonical.com if it isn't that for things like these.

Hm... not sure why it used my gmail address, my default email in github is already the one @canonical.com and for the commit I've also used @canonical.com: Author: Andrea Righi <andrea.righi@canonical.com>. I guess when I click "Create merge request" here on github is still using the gmail one, because it's the one associated to my account...? I'll investigate a bit.

[xnox]

@sergiusens kernel team approves of this PR. please approve & merge this on our behalf due to lack of ACLs on this.