Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Nested container image builds and pushes are failing to start #540

Closed
mikedep333 opened this issue Aug 29, 2023 · 1 comment · Fixed by #541 or mikedep333/pulp-oci-images#6
Closed

Nested container image builds and pushes are failing to start #540

mikedep333 opened this issue Aug 29, 2023 · 1 comment · Fixed by #541 or mikedep333/pulp-oci-images#6
Labels
bug Something isn't working

Comments

@mikedep333
Copy link
Member

mikedep333 commented Aug 29, 2023 •
edited
Loading

https://github.com/pulp/pulp_container/actions/runs/5988444008/job/16243656835#step:10:5275

During pulp_container tests, which tries to run skopeo (and also buildah).

E           pulp_smash.exceptions.CalledProcessError: Command ('podman', 'login', '-u', 'admin', '-p', 'password', 'pulp.example.com:443', '--tls-verify=true') returned non-zero exit status 125.
E           
E           stdout: 
E           
E           stderr: time="2023-08-27T03:04:17Z" level=warning msg="\"/\" is not a shared mount, this could cause issues or missing mounts with rootless containers"
E           time="2023-08-27T03:04:17Z" level=error msg="running `/usr/bin/newuidmap 3755 0 700 1 1 100000 65536`: newuidmap: write to uid_map failed: Operation not permitted\n"
E           Error: cannot set up namespace using "/usr/bin/newuidmap": should have setuid or have filecaps setuid: exit status 1

usr/local/lib/python3.8/site-packages/pulp_smash/cli.py:128: CalledProcessError

This happened when they updated the plugin-template from pulp/pulp-ci-centos:https (about 8 months old) to pulp/pulp-ci-centos:latest . Podman got upgraded from 4.2 to 4.4 and then 4.7, as CentOS 8 went from RHEL 8.7 to 8.8 and then 8.9.
mikedep333 added a commit to mikedep333/pulp-oci-images that referenced this issue Aug 29, 2023
@mikedep333
Fix nested containers
49c8c0d 
fixes: pulp#540
@mikedep333 mikedep333 mentioned this issue Aug 29, 2023
Merged
mikedep333 added a commit to mikedep333/pulp-oci-images that referenced this issue Aug 29, 2023
@mikedep333
Fix nested containers
f1299c7 
fixes: pulp#540
mikedep333 added a commit to mikedep333/pulp-oci-images that referenced this issue Aug 29, 2023
@mikedep333
Fix nested containers
e29b9be 
fixes: pulp#540
mikedep333 added a commit to mikedep333/pulp-oci-images that referenced this issue Aug 29, 2023
@mikedep333
Fix nested containers
e8ef897 
fixes: pulp#540
mikedep333 added a commit to mikedep333/pulp-oci-images that referenced this issue Aug 30, 2023
@mikedep333
Fix nested containers
3c9f57a 
fixes: pulp#540
@lubosmj lubosmj mentioned this issue Sep 4, 2023
Closed
lubosmj added a commit to lubosmj/pulp_container that referenced this issue Sep 8, 2023
@lubosmj
Use an image with updated service files
ccff7da 
This is a temporary workaround that makes the CI green once again. The
referenced image uses the pulp-ci-centos:https base image. On top of it,
there are layers that contain updated service files which allows us to
properly run pulpcore-api and pulpcore-content services.

We will revert the commit once we will be able to resolve the execution
of nested containers.

ref pulp/pulpcore@4def68e#diff-15df6aebb457ef62acf7707d2032bf15dc24646d040d1bf2f892929263ce570dR16
ref pulp/pulp-oci-images#540

[noissue]
lubosmj added a commit to lubosmj/pulp_container that referenced this issue Sep 8, 2023
@lubosmj
Use an image with updated service files
3e6358b 
This is a temporary workaround that makes the CI green once again. The
referenced image uses the pulp-ci-centos:https base image. On top of it,
there are layers that contain updated service files which allows us to
properly run pulpcore-api and pulpcore-content services.

We will revert the commit once we will be able to resolve the execution
of nested containers.

ref pulp/pulpcore@4def68e#diff-15df6aebb457ef62acf7707d2032bf15dc24646d040d1bf2f892929263ce570dR16

ref pulp/pulp-oci-images#540

[noissue]
@lubosmj lubosmj mentioned this issue Sep 8, 2023
Merged
lubosmj added a commit to lubosmj/pulp_container that referenced this issue Sep 8, 2023
@lubosmj
Use the old pulp-ci-centos:https image with updated service files
0ec8473 
This is a temporary workaround that makes the CI green once again. The
referenced image uses the pulp-ci-centos:https base image. On top of it,
there are layers that contain updated service files which allows us to
properly run pulpcore-api and pulpcore-content services.

We will revert the commit once we will be able to resolve the execution
of nested containers.

ref pulp/pulpcore@4def68e#diff-15df6aebb457ef62acf7707d2032bf15dc24646d040d1bf2f892929263ce570dR16

ref pulp/pulp-oci-images#540

[noissue]
lubosmj added a commit to pulp/pulp_container that referenced this issue Sep 8, 2023
@lubosmj
Use the old pulp-ci-centos:https image with updated service files
abdcf7c 
This is a temporary workaround that makes the CI green once again. The
referenced image uses the pulp-ci-centos:https base image. On top of it,
there are layers that contain updated service files which allows us to
properly run pulpcore-api and pulpcore-content services.

We will revert the commit once we will be able to resolve the execution
of nested containers.

ref pulp/pulpcore@4def68e#diff-15df6aebb457ef62acf7707d2032bf15dc24646d040d1bf2f892929263ce570dR16

ref pulp/pulp-oci-images#540

[noissue]
patchback bot pushed a commit to pulp/pulp_container that referenced this issue Sep 9, 2023
@lubosmj @patchback
Use the old pulp-ci-centos:https image with updated service files
fc74008 
This is a temporary workaround that makes the CI green once again. The
referenced image uses the pulp-ci-centos:https base image. On top of it,
there are layers that contain updated service files which allows us to
properly run pulpcore-api and pulpcore-content services.

We will revert the commit once we will be able to resolve the execution
of nested containers.

ref pulp/pulpcore@4def68e#diff-15df6aebb457ef62acf7707d2032bf15dc24646d040d1bf2f892929263ce570dR16

ref pulp/pulp-oci-images#540

[noissue]

(cherry picked from commit abdcf7c)
@patchback patchback bot mentioned this issue Sep 9, 2023
Merged
lubosmj added a commit to pulp/pulp_container that referenced this issue Sep 9, 2023
@lubosmj
Use the old pulp-ci-centos:https image with updated service files
5da2479 
This is a temporary workaround that makes the CI green once again. The
referenced image uses the pulp-ci-centos:https base image. On top of it,
there are layers that contain updated service files which allows us to
properly run pulpcore-api and pulpcore-content services.

We will revert the commit once we will be able to resolve the execution
of nested containers.

ref pulp/pulpcore@4def68e#diff-15df6aebb457ef62acf7707d2032bf15dc24646d040d1bf2f892929263ce570dR16

ref pulp/pulp-oci-images#540

[noissue]

(cherry picked from commit abdcf7c)
@mikedep333 mikedep333 added bug Something isn't working and removed Triage-Needed labels Oct 10, 2023
mikedep333 added a commit to mikedep333/pulp-oci-images that referenced this issue Oct 17, 2023
@mikedep333
Fix nested containers
9d9f7b3 
fixes: pulp#540
@mikedep333 mikedep333 changed the title Nested containers are failing to start Nested container image builds and pushes are failing to start Oct 20, 2023
@mikedep333
Copy link
Member Author

mikedep333 commented Oct 20, 2023 •
edited
Loading

Upon further investigation, we are not trying to run nested containers actually. podman run is not run by pulp_container.

What pulp_container does run is both buildah & skopeo inside of podman.

Getting them working on the latest Podman 4.y (upgrade from 4.2 to 4.4, and now 4.7 while working on this PR) is much easier. They have fewer requirements.

mikedep333 added a commit to mikedep333/pulp-oci-images that referenced this issue Oct 20, 2023
@mikedep333
Fix buildah/skopeo (for pulp_container) when pulp
8075766 
is run inside of a container based on RHEL8.8's podman 4.4 or later.

fixes: pulp#540
@mikedep333 mikedep333 mentioned this issue Oct 20, 2023
Merged
mikedep333 added a commit to mikedep333/pulp-oci-images that referenced this issue Oct 20, 2023
@mikedep333
Fix buildah/skopeo (for pulp_container) when pulp
430911d 
is run inside of a container based on RHEL8.8's podman 4.4 or later.

fixes: pulp#540
patchback bot pushed a commit to pulp/pulp_container that referenced this issue Oct 31, 2023
@lubosmj @patchback
Use the old pulp-ci-centos:https image with updated service files
f379275 
This is a temporary workaround that makes the CI green once again. The
referenced image uses the pulp-ci-centos:https base image. On top of it,
there are layers that contain updated service files which allows us to
properly run pulpcore-api and pulpcore-content services.

We will revert the commit once we will be able to resolve the execution
of nested containers.

ref pulp/pulpcore@4def68e#diff-15df6aebb457ef62acf7707d2032bf15dc24646d040d1bf2f892929263ce570dR16

ref pulp/pulp-oci-images#540

[noissue]

(cherry picked from commit abdcf7c)
@patchback patchback bot mentioned this issue Oct 31, 2023
Merged
lubosmj added a commit to pulp/pulp_container that referenced this issue Oct 31, 2023
@lubosmj
Use the old pulp-ci-centos:https image with updated service files
6ca9ef9 
This is a temporary workaround that makes the CI green once again. The
referenced image uses the pulp-ci-centos:https base image. On top of it,
there are layers that contain updated service files which allows us to
properly run pulpcore-api and pulpcore-content services.

We will revert the commit once we will be able to resolve the execution
of nested containers.

ref pulp/pulpcore@4def68e#diff-15df6aebb457ef62acf7707d2032bf15dc24646d040d1bf2f892929263ce570dR16

ref pulp/pulp-oci-images#540

[noissue]

(cherry picked from commit abdcf7c)
lubosmj added a commit to pulp/pulp_container that referenced this issue Oct 31, 2023
@lubosmj
Use the old pulp-ci-centos:https image with updated service files
a014163 
This is a temporary workaround that makes the CI green once again. The
referenced image uses the pulp-ci-centos:https base image. On top of it,
there are layers that contain updated service files which allows us to
properly run pulpcore-api and pulpcore-content services.

We will revert the commit once we will be able to resolve the execution
of nested containers.

ref pulp/pulpcore@4def68e#diff-15df6aebb457ef62acf7707d2032bf15dc24646d040d1bf2f892929263ce570dR16

ref pulp/pulp-oci-images#540

[noissue]

(cherry picked from commit abdcf7c)
lubosmj added a commit to pulp/pulp_container that referenced this issue Oct 31, 2023
@lubosmj
Use the old pulp-ci-centos:https image with updated service files
49cc438 
This is a temporary workaround that makes the CI green once again. The
referenced image uses the pulp-ci-centos:https base image. On top of it,
there are layers that contain updated service files which allows us to
properly run pulpcore-api and pulpcore-content services.

We will revert the commit once we will be able to resolve the execution
of nested containers.

ref pulp/pulpcore@4def68e#diff-15df6aebb457ef62acf7707d2032bf15dc24646d040d1bf2f892929263ce570dR16

ref pulp/pulp-oci-images#540

[noissue]

(cherry picked from commit abdcf7c)
mikedep333 added a commit to mikedep333/pulp_container that referenced this issue Nov 1, 2023
@mikedep333
Updated CI with the nested container fix
7361e0e 
Related: pulp/pulp-oci-images#540

[noissue]
@mikedep333 mikedep333 mentioned this issue Nov 1, 2023
Merged
lubosmj pushed a commit to pulp/pulp_container that referenced this issue Nov 1, 2023
@mikedep333 @lubosmj
Updated CI with the nested container fix
9ead8f7 
Related: pulp/pulp-oci-images#540

[noissue]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
No open projects
RH Pulp Kanban board
Archived in project
Milestone
No milestone
1 participant
@mikedep333