podman from current rhcontainerbot/packit-builds breaks checkpointing

[martinpitt]

Issue Description

The latest packages in your https://copr.fedorainfracloud.org/coprs/rhcontainerbot/packit-builds/ break checkpointing. This was spotted in containers/podman#19500.

Steps to reproduce the issue

On a current Fedora 38, checkpoint/restore works:

podman run -dit --name test1 docker.io/busybox sh
podman container checkpoint test1
podman container restore test1
podman rm -flt0

Updating to the latest builds from main branches:

dnf copr enable rhcontainerbot/packit-builds
dnf update --disablerepo=updates  # skip kernel updates and unrelated packages

pulls in

Upgrading:
 aardvark-dns
          x86_64 102:1.7.0-1.20230724122917168191.pr358.21.ga713fa6   copr:copr.fedorainfracloud.org:rhcontainerbot:packit-builds 914 k
 container-selinux
          noarch 102:2.219.0-1.20230727135309517446.pr260.3.gcdbd720  copr:copr.fedorainfracloud.org:rhcontainerbot:packit-builds  49 k
 crun     x86_64 102:1.8.6-1.20230804061348476023.pr1254.12.g02ee7c4  copr:copr.fedorainfracloud.org:rhcontainerbot:packit-builds 197 k
 netavark x86_64 102:1.7.0-1.20230804010218038021.pr765.50.gbe2a8ea   copr:copr.fedorainfracloud.org:rhcontainerbot:packit-builds 3.0 M
 podman   x86_64 102:4.7.0~dev-1.20230802175140699202.pr19477.1302.258a3e617.fc38
                                                                      copr:copr.fedorainfracloud.org:rhcontainerbot:packit-builds  14 M
Installing dependencies:
 gvisor-tap-vsock
          x86_64 103:0.7.0-1.20230802140348422016.pr247.1.g5cc1053    copr:copr.fedorainfracloud.org:rhcontainerbot:packit-builds 5.0 M
     replacing  podman-gvproxy.x86_64 5:4.5.1-1.fc38

After that, the podman container checkpoint test1 command fails.

Describe the results you received

Error: configured runtime does not support checkpoint/restore

Describe the results you expected

checkpoint works

podman info output

host:
  arch: amd64
  buildahVersion: 1.32.0-dev
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  - rdma
  - misc
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.7-2.fc38.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.7, commit: '
  cpuUtilization:
    idlePercent: 90.65
    systemPercent: 2.11
    userPercent: 7.24
  cpus: 1
  databaseBackend: boltdb
  distribution:
    distribution: fedora
    variant: cloud
    version: "38"
  eventLogger: journald
  freeLocks: 2047
  hostname: fedora-38-127-0-0-2-2201
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 6.4.6-200.fc38.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 302899200
  memTotal: 1123872768
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.7.0-1.20230724122917168191.pr358.21.ga713fa6.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.8.0-dev
    package: netavark-1.7.0-1.20230804010218038021.pr765.50.gbe2a8ea.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.8.0-dev
  ociRuntime:
    name: crun
    package: crun-1.8.6-1.20230804061348476023.pr1254.12.g02ee7c4.x86_64
    path: /usr/bin/crun
    version: |-
      crun version UNKNOWN
      commit: faa0adb9c2e2f2f24952adf442be4a03bfa54b8f
      rundir: /run/user/0/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20230625.g32660ce-1.fc38.x86_64
    version: |
      pasta 0^20230625.g32660ce-1.fc38.x86_64
      Copyright Red Hat
      GNU Affero GPL version 3 or later <https://www.gnu.org/licenses/agpl-3.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-12.fc38.x86_64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 1120661504
  swapTotal: 1123020800
  uptime: 0h 9m 14.00s
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /usr/share/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 1
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 12798898176
  graphRootUsed: 2164260864
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 4
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.7.0-dev
  Built: 1690998908
  BuiltTime: Wed Aug  2 17:55:08 2023
  GitCommit: ""
  GoVersion: go1.20.6
  Os: linux
  OsArch: linux/amd64
  Version: 4.7.0-dev

Podman in a container

No

Privileged Or Rootless

Privileged

Upstream Latest Release

Yes

Additional environment details

Standard Fedora 38 cloud image. It also reproduces in the Testing farm in cockpit-podman's tests, see test logs and the corresponding screenshot

Additional information

No response

[Luap99]

This sounds to me like crun is build without criu support, podman does a simple checl to run crun checkpoint --help in order to see if checkpoint is supported by the oci runtime.

If crun is build with criu we get unknown command checkpoint, so this looks like a packit build issue.
@lsm5 @giuseppe PTAL

[Luap99]

In the actual fedora spec we have https://src.fedoraproject.org/rpms/crun/blob/rawhide/f/crun.spec#_52 criu as dependency listed. However in the upstream spec it is not: https://github.com/containers/crun/blob/main/rpm/crun.spec

[martinpitt]

That was indeed already the case in the original 6a3d7a7 which introduced the spec file upstream. So apparently that used an outdated version?