OEM: Provide Proxmox images #1783

pothos · 2024-03-22T12:15:54Z

This pulls in flatcar/bootengine#91
and flatcar/init#115 to run afterburn for
hostname, network, SSH key, and metadata attribute setup. The afterburn
support for the SSH key and hostname parses the user-data when it's
cloud-init. The coreos-cloudinit support is not there but can be added
in addition: We need to add a new provider that varies from the existing
config drive support because the file is called user-data and not
user_data, and it needs to look for a filesystem label cidata and not
config-2.

Related to: flatcar/Flatcar#1040

How to use

Follow-up: fix coreos-cloudinit as mentioned above, and better don't make it work through the mount unit trigger but explicitly

Testing done

Jenkins

Changelog entries added in the respective changelog/ directory (user-facing change, bug fix, security fix, update)
Inspected CI output for image differences: /boot and /usr size, packages, list files for any missing binaries, kernel modules, config files, kernel modules, etc.

TODO: bump init and bootengine once related PRs are merged

github-actions · 2024-03-22T15:19:32Z

Build action triggered: https://github.com/flatcar/scripts/actions/runs/11665808382

arcln · 2024-04-24T09:24:20Z

Hello, there is an existing Igntion PR that should work at coreos/ignition#1790. Could you include the Ignition PR into this one so we can do further testing?

pothos · 2024-04-24T10:01:00Z

Hello, there is an existing Igntion PR that should work at coreos/ignition#1790. Could you include the Ignition PR into this one so we can do further testing?

This needs to be reworked first and we have to drop the afterburn integration in Flatcar as the afterburn PR relies on cloud-init userdata and because Flatcar will run coreos-cloudinit anyway if it doesn't see Ignition user-data we would have it processed twice. For the network config that can be quite confusing because the afterburn network unit would only support a subset.

pothos · 2024-05-06T13:25:48Z

Triggered a new build with the Ignition PR included

arcln · 2024-05-27T09:48:11Z

@pothos hey, it looks like the build never started, can you take a look please ? (cc @plgueugnon)

tormath1 · 2024-05-27T13:30:11Z

@pothos hey, it looks like the build never started, can you take a look please ? (cc @plgueugnon)

@arcln hey, pothos is AFK for now. I'll take over this PR. A CI build has been triggered :)

tormath1 · 2024-05-28T15:36:09Z

@arcln here's the proxmox image if you want to try it: http://bincache.flatcar-linux.net/images/amd64/9999.0.101+kai-proxmox-support/flatcar_production_proxmoxve_image.img.bz2

arcln · 2024-05-29T16:49:52Z

Thank you, I tried the image and it didn't work because of a problem in the ignition PR (see coreos/ignition#1790 (comment)). Can you please run a new build when the PR is fixed ? Thanks

tormath1 · 2024-05-30T08:19:34Z

Thank you, I tried the image and it didn't work because of a problem in the ignition PR (see coreos/ignition#1790 (comment)). Can you please run a new build when the PR is fixed ? Thanks

@arcln ah I see, I already got this issue once. I started a new build with an updated patch: https://github.com/flatcar/scripts/pull/1783/files#diff-c65e2140094c80c4b4ce5c7c8494b6e4e72b78ef0692d236ffee1fb3932e11c5R157

arcln · 2024-05-30T09:03:21Z

great. it seems that the build needs approval once again @tormath1

tormath1 · 2024-05-30T09:30:10Z

@arcln it builds directly on our Jenkins (github actions only build for qemu targets) - I'll let you know once it's done.

tormath1 · 2024-05-30T11:49:03Z

@arcln image is available here: http://bincache.flatcar-linux.net/images/amd64/9999.0.102+kai-proxmox-support/flatcar_production_proxmoxve_image.img.bz2

tormath1 · 2024-10-24T17:47:20Z

🔈 Proxmox users - Ignition and Afterburn are now released and ingested into Flatcar. I need your help to test this freshly built image: http://bincache.flatcar-linux.net/images/amd64/9999.9.100+kai-proxmox-support/flatcar_production_proxmoxve_image.img.bz2 using this documentation: https://github.com/flatcar/flatcar-website/pull/337/files (or any other method)

@abuisine @arcln @fhemberger @mcbenjemaa

fhemberger · 2024-10-25T10:23:05Z

@tormath1 Tried to test the image following the given documentation, fails to start:

[   57.120718] localhost systemd[1]: Finished ignition-mount.service - Ignition (mount).
[   62.633185] localhost coreos-metadata[704]: Error: failed to run
[   62.633185] localhost coreos-metadata[704]: Caused by:
[   62.633185] localhost coreos-metadata[704]:     0: fetching metadata from provider
[   62.633185] localhost coreos-metadata[704]:     1: maximum number of retries (3) reached
[   62.633185] localhost coreos-metadata[704]:     2: failed to mount (read-only) source '/dev/disk/by-label/cidata' to target '/tmp/afterburn-Kk3KoQ', with type 'iso9660'
[   62.633185] localhost coreos-metadata[704]:     3: ENOENT: No such file or directory

Here is the complete dump of rdsosreport.txt

Ignition config is stored via qm set $VM_ID --cicustom "user=local:snippets/user-data". (Took it from another Flatcar VM I tested before.)

EDIT: It was a documentation issue, one important line was missing which activates the cloud-init support in Proxmox VE.

fhemberger · 2024-10-25T10:45:59Z

@tormath1 Added qm set $VM_ID --ide2 local:cloudinit, restarted the VM and the provisioning works! 🎉

tormath1 · 2024-10-25T10:54:07Z

@tormath1 Added qm set $VM_ID --ide2 local:cloudinit, restarted the VM and the provisioning works! 🎉

Great news, we should add this in the documentation. Thank you so much for testing and investigating this. So Ignition provisioning is working and what about coreos-metadata.service ? Do you have some metadata in /run/metadata/flatcar ? And what about flatcar-digitalocean-network.service ? (the name is legacy, it has nothing to see with Digital Ocean).

fhemberger · 2024-10-25T11:24:18Z

@tormath1 I already amended the documentation PR, this is the process for me:
flatcar/flatcar-website#337 (review)

what about coreos-metadata.service? Do you have some metadata in /run/metadata/flatcar?

core@flatcar ~ $ systemctl status coreos-metadata.service
○ coreos-metadata.service - Flatcar Metadata Agent
     Loaded: loaded (/usr/lib/systemd/system/coreos-metadata.service; disabled; preset: disabled)
     Active: inactive (dead)
  Condition: start condition unmet at Fri 2024-10-25 10:39:50 UTC; 24min ago
             ├─ ConditionKernelCommandLine=|flatcar.oem.id=ec2 was not met
             ├─ ConditionKernelCommandLine=|coreos.oem.id=ec2 was not met
             ├─ ConditionKernelCommandLine=|flatcar.oem.id=azure was not met
             ├─ ConditionKernelCommandLine=|coreos.oem.id=azure was not met
             ├─ ConditionKernelCommandLine=|flatcar.oem.id=digitalocean was not met
             ├─ ConditionKernelCommandLine=|coreos.oem.id=digitalocean was not met
             ├─ ConditionKernelCommandLine=|flatcar.oem.id=gce was not met
             ├─ ConditionKernelCommandLine=|coreos.oem.id=gce was not met
             ├─ ConditionKernelCommandLine=|flatcar.oem.id=openstack was not met
             ├─ ConditionKernelCommandLine=|coreos.oem.id=openstack was not met
             ├─ ConditionKernelCommandLine=|flatcar.oem.id=packet was not met
             ├─ ConditionKernelCommandLine=|coreos.oem.id=packet was not met
             ├─ ConditionKernelCommandLine=|flatcar.oem.id=scaleway was not met
             ├─ ConditionKernelCommandLine=|flatcar.oem.id=hetzner was not met
             ├─ ConditionKernelCommandLine=|flatcar.oem.id=kubevirt was not met
             └─ ConditionKernelCommandLine=|flatcar.oem.id=akamai was not met

The coreos-metadata.service is missing a line ConditionKernelCommandLine=|flatcar.oem.id=proxmoxve, that's why /run/metadata/flatcar wasn't created.

I ran the script manually, after that the metadata was present:

core@flatcar ~ $ sudo /usr/bin/coreos-metadata --cmdline --attributes=/run/metadata/flatcar
Oct 25 11:11:21.706 WARN user-data does not have the expected header `#cloud-config`, ignoring this file

core@flatcar ~ $ cat /run/metadata/flatcar
AFTERBURN_PROXMOXVE_INSTANCE_ID=ae1808cd94103dab0a5f9dcbc2b93e0b0f8d9b51

And what about flatcar-digitalocean-network.service? (the name is legacy, it has nothing to see with Digital Ocean).

core@flatcar ~ $ systemctl status flatcar-digitalocean-network.service
Unit flatcar-digitalocean-network.service could not be found.

There's no such service (or similarly named). 🤷

mcbenjemaa · 2024-10-25T12:23:02Z

This image is failed to boot for me

fhemberger · 2024-10-25T12:32:14Z

@mcbenjemaa Did you follow the procedure here?
flatcar/flatcar-website#337 (review)

Could you please provide a bit more context?

mcbenjemaa · 2024-10-25T12:47:35Z

flatcar/flatcar-website#337 (review)

@mcbenjemaa Did you follow the procedure here? flatcar/flatcar-website#337 (review)

Could you please provide a bit more context?

I just converted the VM into a template and then tried to add the config drive into a new clone, which didn't work.

fhemberger · 2024-10-25T13:31:56Z

@mcbenjemaa You still need to provide your ignition.json. Proxmox' own cloud-init parameters are not enough (yet).

Creating the VM in the GUI doesn't work at the moment, but should work via CLI (steps linked in my comment above).

mcbenjemaa · 2024-10-25T13:36:01Z

@mcbenjemaa You still need to provide your ignition.json. Proxmox' own cloud-init parameters are not enough (yet).

Creating the VM in the GUI doesn't work at the moment, but should work via CLI (steps linked in my comment above).

Can you try to mount /dev/disk/by-label/cidata into some tmp folder, and let me know what is inside there?

tormath1 · 2024-10-25T13:40:47Z

@fhemberger @mcbenjemaa many thanks for trying again Flatcar on Proxmox! I started a discussion to avoid spamming and distracting the PR review. You can continue the discussion here if you need: flatcar/Flatcar#1573

fhemberger · 2024-10-25T14:16:59Z

@pothos Could you please add the following change to your PR:

https://github.com/flatcar/scripts/blob/main/sdk_container/src/third_party/coreos-overlay/coreos-base/afterburn/files/coreos-metadata.service

--- coreos-metadata.service             2024-10-25 16:14:39.499357398 +0200
+++ coreos-metadata.service-proxmoxve   2024-10-25 16:14:55.180153512 +0200
@@ -25,6 +25,8 @@

 ConditionKernelCommandLine=|flatcar.oem.id=akamai

+ConditionKernelCommandLine=|flatcar.oem.id=proxmoxve
+
 Description=Flatcar Metadata Agent

 [Service]

Co-authored-by: Kai Lüke <pothos@users.noreply.github.com> Signed-off-by: Mathieu Tortuyaux <mtortuyaux@microsoft.com>

donch · 2024-10-28T14:54:15Z

Hi @tormath1 , i've tested your build on Proxmox using the cloud-init configuration and it works well (at least for network configuration and SSH keys). By the way, the hostname configuration process strips any FQDN: if you set 'test.vm-1', only 'test' will be retained as the hostname.

Will continue testing the ignition part.

tormath1 · 2024-10-29T12:10:59Z

@pothos Could you please add the following change to your PR:

https://github.com/flatcar/scripts/blob/main/sdk_container/src/third_party/coreos-overlay/coreos-base/afterburn/files/coreos-metadata.service
--- coreos-metadata.service             2024-10-25 16:14:39.499357398 +0200
+++ coreos-metadata.service-proxmoxve   2024-10-25 16:14:55.180153512 +0200
@@ -25,6 +25,8 @@

 ConditionKernelCommandLine=|flatcar.oem.id=akamai

+ConditionKernelCommandLine=|flatcar.oem.id=proxmoxve
+
 Description=Flatcar Metadata Agent

 [Service]

Now available here: http://bincache.flatcar-linux.net/images/amd64/9999.9.101+kai-proxmox-support/flatcar_production_proxmoxve_image.img.bz2

fhemberger · 2024-10-29T13:09:52Z

@tormath1 Works fine for me, /run/metadata/flatcar is now created successfully. 👍

donch · 2024-10-29T15:46:18Z

I was able to sucessfully configure a proxmox VM using ignition including user creation, file creation, custom sysext deployment, systemd unit 🎉
By the way, the service coreos-metadata-sshkeys@core.service is failing when no ssh pub key are supplied but may not be related to this PR

tormath1 · 2024-10-30T08:10:42Z

Thanks everyone involved in the testing of the image, that's truly appreciated ❤️ There is one last thing to sort out: flatcar/Flatcar#1573 (reply in thread) before going ahead.

abuisine · 2024-10-31T10:54:00Z

Thanks everyone involved in the testing of the image, that's truly appreciated ❤️ There is one last thing to sort out: flatcar/Flatcar#1573 (reply in thread) before going ahead.

Hi, we got an extra test with @donch if you do not mind : complete configuration via terraform (upload of snippets and cicustom). We should have the feedback today.

fhemberger · 2024-10-31T12:55:54Z

@abuisine I used Terraform provider bpg/proxmox@0.66.3 to create a VM with an ignition.json config snippet. Worked for me with just this little caveat:

Error: resizing disk: error waiting for VM disk resize: All attempts fail:
│ #1: task "UPID:xxx:0028F421:0E5B0594:67237B38:resize:1000:xxx@xxx!xxx:" failed to complete with exit code: shrinking disks is not supported
│ 
│   with proxmox_virtual_environment_vm.flatcar,
│   on virtual_machine.tf line 1, in resource "proxmox_virtual_environment_vm" "flatcar":
│    1: resource "proxmox_virtual_environment_vm" "flatcar" {

But this is an issue with the provider, starting the VM and everything after worked as expected.

donch · 2024-10-31T14:17:31Z

Thanks everyone involved in the testing of the image, that's truly appreciated ❤️ There is one last thing to sort out: flatcar/Flatcar#1573 (reply in thread) before going ahead.

Hi, we got an extra test with @donch if you do not mind : complete configuration via terraform (upload of snippets and cicustom). We should have the feedback today.

Hi @tormath1 , i was able to create a VM using Terraform with these providers :

community-terraform-providers/ignition v2.3.5 to generate the ignition config
bpg/proxmox v0.66.3 to create the VM and attach the generated snippet.

All is working fine 👌

fhemberger · 2024-10-31T14:34:21Z

@donch Did you run into the resize issue as well? If not, it would be great if you could share your snippet for proxmox_virtual_environment_vm and proxmox_virtual_environment_file (ISO).

donch · 2024-10-31T14:47:09Z

@fhemberger no issue on my side. You may have a disk smaller than the image size ? That's why you get the "shrinking" issue i guess.

Here are my snippets:

resource "proxmox_virtual_environment_file" "cloud_config" {
  content_type = "snippets"
  datastore_id = "mystore"
  node_name    = "pve-1"

  source_raw {
    data      = data.ignition_config.config.rendered
    file_name = local.ignition_filename
  }
}


resource "proxmox_virtual_environment_vm" "flatcar" {
  name        = "donch.flatcar-2"
  node_name   = "pve-1"
  vm_id       = local.vm_id
  started     = false

  clone {
    vm_id = 9051
    full  = true
  }

  memory {
    dedicated = 2048
  }

  cpu {
    cores = 2
  }

  disk {
    size            = 12
    datastore_id    = "mystore"
    interface       = "scsi0"
  }

  network_device {
    bridge = "vmbr0"
    model  = "virtio"
    vlan_id = 1234
  }

  agent {
    enabled = true
  }

  initialization {
    datastore_id = "mystore"
    user_data_file_id = proxmox_virtual_environment_file.cloud_config.id
  }

}

abuisine · 2024-11-04T10:16:33Z

Alright so it seems ok except the default behavior while no configuration is provided.
Is this a go then ? or no go ?

tormath1 · 2024-11-04T14:04:10Z

Alright so it seems ok except the default behavior while no configuration is provided. Is this a go then ? or no go ?

Hi, it's not only a default behavior: the instance does not boot if no configuration is provided. I'm currently checking this. We aim for a Flatcar release by the end of the week, if it's not solved by this time I will merge this PR. It's ok to have this support in Alpha, even incomplete, we will get more feedback on it.

EDIT: Upstream issue: coreos/afterburn#1126

pothos had a problem deploying to development March 22, 2024 12:15 — with GitHub Actions Error

pothos force-pushed the kai/proxmox-support branch from 8b6f5ae to 4bdeec5 Compare March 22, 2024 12:17

pothos temporarily deployed to development March 22, 2024 12:17 — with GitHub Actions Inactive

pothos force-pushed the kai/proxmox-support branch from 4bdeec5 to 9ef9191 Compare March 22, 2024 13:08

pothos had a problem deploying to development March 22, 2024 13:08 — with GitHub Actions Error

pothos force-pushed the kai/proxmox-support branch from 9ef9191 to 709fca5 Compare March 22, 2024 13:14

pothos temporarily deployed to development March 22, 2024 13:14 — with GitHub Actions Inactive

arcln mentioned this pull request Mar 28, 2024

Proxmox VE provider coreos/ignition#1844

Closed

3 tasks

github-actions bot mentioned this pull request Apr 22, 2024

Monthly contributions report 2024-03-22 - 2024-04-21 flatcar/Flatcar#1435

Closed

pothos force-pushed the kai/proxmox-support branch 2 times, most recently from 64b0e74 to 7c833ba Compare May 6, 2024 13:19

pothos temporarily deployed to development May 6, 2024 13:19 — with GitHub Actions Inactive

tormath1 force-pushed the kai/proxmox-support branch from 7c833ba to a52c30b Compare May 27, 2024 15:31

tormath1 temporarily deployed to development May 27, 2024 15:32 — with GitHub Actions Inactive

tormath1 force-pushed the kai/proxmox-support branch from a52c30b to 456e9bb Compare May 30, 2024 08:16

tormath1 had a problem deploying to development May 30, 2024 08:17 — with GitHub Actions Error

tormath1 force-pushed the kai/proxmox-support branch from 456e9bb to c1ce858 Compare May 30, 2024 08:17

tormath1 had a problem deploying to development May 30, 2024 08:17 — with GitHub Actions Failure

tormath1 had a problem deploying to development October 24, 2024 13:07 — with GitHub Actions Error

oem: provide proxmox images

48e1b35

Co-authored-by: Kai Lüke <pothos@users.noreply.github.com> Signed-off-by: Mathieu Tortuyaux <mtortuyaux@microsoft.com>

tormath1 force-pushed the kai/proxmox-support branch from 2bdf792 to 48e1b35 Compare October 25, 2024 14:20

tormath1 had a problem deploying to development October 25, 2024 14:20 — with GitHub Actions Error

fixup! oem: provide proxmox images

7c42743

tormath1 requested a deployment to development October 31, 2024 09:42 — with GitHub Actions Waiting

OEM: Provide Proxmox images #1783

Are you sure you want to change the base?

OEM: Provide Proxmox images #1783

Conversation

pothos commented Mar 22, 2024 • edited by tormath1 Loading

How to use

Testing done

github-actions bot commented Mar 22, 2024 • edited Loading

arcln commented Apr 24, 2024

pothos commented Apr 24, 2024

pothos commented May 6, 2024

arcln commented May 27, 2024

tormath1 commented May 27, 2024

tormath1 commented May 28, 2024

arcln commented May 29, 2024

tormath1 commented May 30, 2024

arcln commented May 30, 2024

tormath1 commented May 30, 2024

tormath1 commented May 30, 2024

tormath1 commented Oct 24, 2024 • edited Loading

fhemberger commented Oct 25, 2024 • edited Loading

fhemberger commented Oct 25, 2024

tormath1 commented Oct 25, 2024

fhemberger commented Oct 25, 2024

mcbenjemaa commented Oct 25, 2024

fhemberger commented Oct 25, 2024

mcbenjemaa commented Oct 25, 2024 • edited Loading

fhemberger commented Oct 25, 2024

mcbenjemaa commented Oct 25, 2024

tormath1 commented Oct 25, 2024

fhemberger commented Oct 25, 2024

donch commented Oct 28, 2024

tormath1 commented Oct 29, 2024

fhemberger commented Oct 29, 2024

donch commented Oct 29, 2024

tormath1 commented Oct 30, 2024

abuisine commented Oct 31, 2024

fhemberger commented Oct 31, 2024 • edited Loading

donch commented Oct 31, 2024

fhemberger commented Oct 31, 2024

donch commented Oct 31, 2024

abuisine commented Nov 4, 2024

tormath1 commented Nov 4, 2024 • edited Loading

pothos commented Mar 22, 2024 •

edited by tormath1

Loading

github-actions bot commented Mar 22, 2024 •

edited

Loading

tormath1 commented Oct 24, 2024 •

edited

Loading

fhemberger commented Oct 25, 2024 •

edited

Loading

mcbenjemaa commented Oct 25, 2024 •

edited

Loading

fhemberger commented Oct 31, 2024 •

edited

Loading

tormath1 commented Nov 4, 2024 •

edited

Loading