Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ubuntu/jammy: 24.3.1 upstream snapshot release #5678

Merged
merged 50 commits into from
Sep 10, 2024

Conversation

blackboxsw
Copy link
Collaborator

@blackboxsw blackboxsw commented Sep 6, 2024

Additional context

Full package diff review branch

For full package dff please see: blackboxsw#28

The upstream/ubuntu/jammy branch receives multiple unreleased pull request merges between the time the previous 24.2 published release and a new upstream release 24.3 for a couple of reasons:

  • add quilt patches to retain stable behavior when new features are land in main that we know we want to revert on stable branches
  • perform interim unreleased new_upstream_snapshots to fix daily build recipes when commits to main cause drift that quilt is unable to resolve with fuzz handling

Given that this PR is only the latest unreleased changeset proposed into upstream/ubuntu/jammy, it doesn't represent the full debdiff SRU reviewers would expect to see of all unreleased content since latest published cloud-init in jammy-updates. To aid the review of this changeset, we propose a secondary PR to containing the entire diff of all unreleased content to make that coordination and discussion easier in github prior to landing this PR and performing an upload to jammy:

Note: #5669 (comment) approved the fix to d/patches/no-single-process.patch. cherry-picked this change from noble into focal and jammy

Testing

  • quilt push -a ; tox -e py3; quilt pop -a # confirm no remaining functional cloud-init-(network|main).service references
  • build-package
  • -build # no new lintian warnings/errors
  • package upgrade test (also covered anyway in daily integration tests) # specifically validate no systemd unit or target inconsistencies w/ cloud-init-main or cloud-init-network.service vs cloud-init.service

Expected CI failures on this PR

  • packaging tests should fail due to drift in main after 24.3.1 upstream release was cut rendering the no-single-process.patch unable to apply even with fuzz. this broke our daily recipe PPA builder and will be resolved just after this release is cut by performing another new_upstream_snapshot to refresh quilt patches against tip of main

PengpengSun and others added 30 commits August 9, 2024 14:32
…onical#5471)" (canonical#5596)

This reverts commit 2b6fe64.

When there is no IPv6 set to dhcp explicitly, NetworkManager keyfile
defaults to method=auto, may-fail=true. When there is Ipv6 set to dhcp
explictily, NetworkManager keyfile will be set to
method=auto, may-fail=false. The default settings are what we want, so
revert the previous change to keep IPv6 not set explicitly.
…nonical#5598)

chore: add comment explaining the NetworkManager may-fail setting

The value of may-fail in network manager keyfile is a source of
confusion as the default value of it is True for Network Manager and
False for network manager renderer implementation. Add a comment to
explain why the renderer sets may-fail to False in its implementation.
Also shift the format page higher in the explanation page list, since
this is a high traffic page.
…canonical#5602)

Recently noticed that doc file changes in nested subdirs were
not triggering documentation auto label.

Example of subdir match at
https://github.com/actions/labeler?tab=readme-ov-file#basic-examples
)

As noted in the systemd documentation, /etc is reserved for "System
units created by the administrator" while the lib directory should be
used by "System units installed by the distribution package manager".

Fixes canonicalGH-5613
When referencing a command from another environment, it will cause
errors when the other environment already exists. Fix it by avoiding
indirection in environment command definitions.

Additionally, simplify envoronment dependency management by defining two
lists of dependencies: a default one with pinned versions for all
environments, and an unpinned on for "tip" environments. Several
dependencies have been missed in the mypy envornments, so this should
make it easier by standardizing environment dependencies to be
consistent across environments.
Bump Ubuntu version for better pip dependency resolution.
…#5385)

When using NetworkManager, if the base bond interface does not have
subnet information configured, ensure it is disabled with respect to
ipv4 and ipv6. Otherwise, the base bond interface defaults to 'auto'
and will try to configure itself via DHCP. This is problematic when
using a tagged VLAN interface on top of the bond as the base
interface will try to configure itself via DHCP on the untagged VLAN.
…on (canonical#5383)

The cloud-init network config version 1 schema defines the bond
properties with underscores, prepended with 'bond-'. This change
ensures consistency with the schema for the bond property names.

canonicalGH-5366
Avoid exclusive expectations that cloud-init is the only agent
registering certificates in a system to
/etc/ssl/certs/ca-certificates.crt.

On Google Cloud Platform, Google Guest Agent does setup root certs
which makes performing a checksum of ca-certificates.crt incorrect due
to extra certs present in ca-certificates.crt.

Adapt test to assert that cloud-init's cert is contained in
ca-certificates.crt but not exclusive content of the file.

Fixes canonicalGH-5609
* Collect sensitive data by default since we ask for it more often
  than not
* Output warning that we're collecting sensitive data
* Glob most of /run/cloud-init, /etc/cloud, and /var/lib/cloud
* Stop creating empty directories in the tarball
* Require running as root given that the logs are root read-only
* Update apport accordingly

Fixes canonicalGH-5297
In the case of Pro, if either agent or user data is not cloud-config user-data,
combine the parse in `self.userdata_raw` as a #include file so
cloud-init transforms that internally into a multipart data.

Avoid passing strings and lists directly, which confused the processor due the
lack of a mime type.

Being explicit about only loading text/cloud-config parts also allow other
composition of cloud-init features to just work, like jinja templates.

This error was surfaced when testing with empty Landscape
data, but any non-text/cloud-config content type would trigger the same behavior.

Add merge_agent_landscape_data to process agent.yaml or Landscape data and
ignore any empty  files present in .ubuntupro/.cloud-init/
…ical#5562)

Without this fix, rendered module documentation was not rendering the
following text for some objects:

Each object in **<key_name>** list supports the following keys:

See Rsyslog Config schema tab.
…cal#5562)

Document any keys of objects in a list which allows for objects as one
of the alternative types allowed as a list item.

Also, when documenting properties, ensure we skip documentation of
either 'properties' or 'patternProperties' if those properties are
declared in the hidden key.

Fixes canonicalGH-5514
…5562)

When running tox -e doc the following environment variables are
supported:
  CLOUD_INIT_DEBUG_MODULE_DOC=cc_<module_id>
  CLOUD_INIT_DEBUG_MODULE_DOC_FILE=<file_path>

The env var CLOUD_INIT_DEBUG_MODULE_DOC can be set to either
a specific module id, such as cc_rsyslog, or 'all'.

When set the rendered module documentation RST format is printed
inline to stdout to allow for quick analysis of rendered content.

Optionally, if CLOUD_INIT_DEBUG_MODULE_DOC_FILE is set to a writable
file path, the output of the rendered content is written to that file
instead.

This supports development of docs and quick comparison of docs
generated before and after a changeset.
Remove additional \n which is not present if only one ca_cert is in the
instance.
KsenijaS and others added 13 commits August 27, 2024 07:36
Add PPS support for azure-proxy agent and improve error logging.
…nical#5633)

Do not treat the emptiness of .cloud-init/ as an error in the logs
if agent.yaml is present.

Fixes canonicalGH-5632
…ply (canonical#5622)

Perform the same steps that cloud-init daily recipe builds performs
to assert any packaging branch updates will not break daily builds
due to quilt patch apply issues.

Steps of daily build recipe reflected in this workflow:
- checkout main
- merge packaging branch topmost commit
- quilt push -a
- run unittests (via tox -e py3)
- quilt pop -a
…cal#5641)

Reintroduce strict assert that cloud-init's cert in userdata is the
only root cert defined on the platform. Google guest agent was
installed a secondary root cert in ca_certifications.crt for a period of
time and this was determined to be less than ideal practice.

Allow cloud-init's integration tests to remain strict validation of
cert checksum to provide a signal if other platforms or agents
attempt to extend or alter the system-wide CA.
…d field (canonical#5355)

Currently cc_user_groups assumes that "useradd" never locks the password
field of newly created users. This is an incorrect assumption.

Change add_user (in both __init__.py and alpine.py) to
explicitly call either lock_passwd or unlock_passwd at all times to
achieve the desired final result.

For existing users with empty or empty locked passwords, no
password unlock will be performed and warnings will be issued.
To support empty password validation, provide functionality to
parse /etc/shadow and /var/lib/extrausers/shadow to assert
existing users do not have empty passwords before unlocking.

Additionally in this commit: 
- add NetworkBSD.ifs property to avoid subp side-effect in ___init__
  which calls ifconfig -a at every instance initialization

Useradd background:

From the useradd manpage:

'-p, --password PASSWORD
The encrypted password, as returned by crypt(3). The default is to
disable the password.'

That is, if cloud-init runs 'useradd' but does not pass it the "-p"
option (with an encrypted password) then the new user's password field
will be locked by "useradd".

cloud-init only passes the "-p" option when calling "useradd" when
user-data specifies the "passwd" option for a new user. For user-data
that specifies either the "hashed_passwd" or "plain_text_passwd"
options instead then cloud-init calls "useradd" without the "-p" option
and so the password field of such a user will be locked by "useradd".

For user-data that specifies "hashed_passwd" for a new user then
"useradd" is called with no "-p" option, so causing "useradd" to lock the
password field, however then cloud-init calls "chpasswd -e" to set the
encrypted password which also results in the password field being
unlocked.

For user-data that specifies either "plain_text_passwd" for a new user
then "useradd" is called with no "-p" option, so causing "useradd" to
lock the password. cloud-init then calls "chpasswd" to set the password
which also results in the password field being unlocked.

For user-data that specifies no password at all for a new user then
"useradd" is called with no "-p" option, so causing "useradd" to lock
the password. The password field is left locked.

In all the above scenarios "passwd -l" may be called later by
cloud-init to enforce "lock_passwd: true").

Conversely where "lock_passwd: false" applies the above "usermod"
situation (for "hash_passwd", "plain_text_passwd" or no
password) means that newly created users may have password
fields locked when they should be unlocked.

For Alpine, "adduser" does not support any form of password being
passed and it always locks the password field (the same point
applies about password field being unlocked when/if "chpasswd" is
called). Therefore in some situations (i.e. no password specified
in user-data) the password needs to be unlocked if
"lock_passwd: false".
Bump the version in cloudinit/version.py to 24.3 and
update ChangeLog.
Bump the version in cloudinit/version.py to 24.3.1 and
update ChangeLog.
patches:
debian/patches/cli-retain-file-argument-as-main-cmd-arg.patch
debian/patches/no-nocloud-network.patch
debian/patches/revert-551f560d-cloud-config-after-snap-seeding.patch
@holmanb
Copy link
Member

holmanb commented Sep 6, 2024

@blackboxsw it looks like the check patch test is failing on both PRs

@blackboxsw
Copy link
Collaborator Author

@blackboxsw it looks like the check patch test is failing on both PRs

@holmanb This is expected (and started failing on daily builds 3 days ago after the 24.3.1 upstream release cut. We do need to fix daily recipes and can do this w/ an unreleased new_upstream_snapshot.py against tip of main once this branch clears review. I'm adding context to this PR now to aid review for @panlinux

Copy link
Member

@TheRealFalcon TheRealFalcon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

d/changelog has some repeated lines for refreshing patches and the upstream snapshot line but otherwise LGTM.

Copy link
Member

@TheRealFalcon TheRealFalcon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

Copy link

@panlinux panlinux left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 from pre-SRU

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
packaging Supplemental package review requested
Projects
None yet
Development

Successfully merging this pull request may close these issues.