Automatic upgrade strategy

[countless-integers]

With AL2 I had a setup where yum-cron would apply security updates automatically. In AL23 this doesn't seem feasible as dnf-automatic (service for automated updates) doesn't pick up security updates as they come in new releases of AL23. I could add a timer+service to update release versions, but it feels a hacky (automatic package updates already felt a bit weird, but on AL2 it seem to have worked reasonably well). What would be a recommended way to tackle this problem on AL23?

[stewartsmith]

Have a look at the following two sections of the User Guide:

https://docs.aws.amazon.com/linux/al2023/ug/managing-repos-os-updates.html

https://docs.aws.amazon.com/linux/al2023/ug/deterministic-upgrades-usage.html

tl;dr: By passing latest as releasever, you can refer to whatever our latest release may be.

[countless-integers]

I don't think dnf-automatic can upgrade the release version on its own, only packages within the release. That's my problem -- it doesn't have an option to pass in the release version/tag. Anyway, I can drop dnf-automatic and just write a script for updates myself. Getting the latest version and running upgrade/update is easy enough, but it's really confusing to asses the state after I run the upgrade command. Take this example after running sudo dnf upgrade --security --assumeyes --releasever=2023.1.20230705 once:

[~]$ sudo dnf upgrade --security --assumeyes --releasever=2023.1.20230705
Last metadata expiration check: 0:15:24 ago on Mon Jul 10 09:49:15 2023.
Dependencies resolved.
Nothing to do.
Complete!
[~]$ dnf check-release-update --latest-only --version-only
2023.1.20230705
[~]$ dnf check-release-update
WARNING:
  A newer release of "Amazon Linux" is available.

  Available Versions:

  Version 2023.1.20230705:
    Run the following command to upgrade to 2023.1.20230705:

      dnf upgrade --releasever=2023.1.20230705

    Release notes:
     https://docs.aws.amazon.com/linux/al2023/release-notes/relnotes.html

which one's output is correct? Is there some metadata caching layer somewhere that I'm not aware of? (I'm not very familiar with rhel/fedora distros tbh)

[celesteking]

Extremely frustrated with your choice of dumping enterprise users. Enterprise doesn't care for instance replacement, it cares for stability and longevity.
You should've offered a streamlined experience for users wanting to stay on a defined release AND wanting to receive security (0day) package updates, even if that would last 2 months. Reprovisioning an instance each day is absurd to point of stupidity.

dnf-automatic is a de-facto standard in RPM/DNF world and should be used to do the update procedure.
The documenation you've provided is confusing and contradicts itself.

[ozbenh]

dnf check-release-update will base its findings on the version of the system-release package that you have installed. As long as that package doesn't get updated, it will continue offerring a new release.

Note that we generally try to be careful with updating packages for security or grave bugs only. So we normally recommend normal upgrades. The main difference is that new releases might also offer new packages you can chose to install, but that's really about it.

[stewartsmith]

Our general recommendation is to do instance replacement: whenever a new AMI comes out, relaunch with that as the base.

That way you have a clear rollback strategy (just launch the previous AMI) as well as having tested it. Doing in-place patching can create a situation where you haven't tested a clean deployment in a long time, and thus don't actually know that any rollback will actually work.

[countless-integers]

Makes sense and we're doing that too. It's just that to minimize the impact of 0-day exploits on some of our critical software, we also want to have an in-place solution in place as a "rapid" response -- so to compliment a AMI-baking pipeline and not as a replacement for it. We had some experience with update process being a bit tardy (and a pipeline will always take a moment, especially if there's no good trigger for it and it runs on schedule) and it got us into problems.

So my takeaway so far is that the rapid-sloppy solution for in-place update would be security upgrade + release package bump and the more proper-robust solution is to have a good AMI-bake-verify-ASG-refresh pipeline.

[caseyoneill]

I agree that I have some concerns with reacting to 0-day exploits. Especially on a small team where we don't have the staff to rebuild the servers every month.

[BansariSheth]

Can't agree more, did anyone find a way to automate the upgrades without having to rebuilt the servers every month and not update the servers manually?
I use deterministic upgrade process where /etc/dnf/vars/releasever is set to latest, but I see instance is not doing the system-release update

[xip-niekneuij]

In terraform, you can define data from AMI marketplace for the latest AL 2023.

Example for getting the latest AMI of Amazon Linux 2023 for ARM:

data "aws_ami" "amazon-linux-2023" {
  most_recent = true
  owners      = ["amazon"]
  filter {
    name   = "name"
    values = ["al2023-ami-2*arm64"]
  }
}

Then refer to the data in aws_instance.

Example of Amazon Linux 2023 from data source with instance type t4g.nano:

resource "aws_instance" "example" {
  ami                    = data.aws_ami.amazon-linux-2023.id
  instance_type          = "t4g.nano"
}

Which means, whenever you update your EC2 instance, you get the latest Amazon Linux 2023 release.