Skip to content

Display PersistentVolumeUsage in crc status #2660

Display PersistentVolumeUsage in crc status

Display PersistentVolumeUsage in crc status #2660

GitHub Actions / JUnit Test Report failed Aug 7, 2024 in 0s

21 tests run, 20 passed, 0 skipped, 1 failed.

Annotations

Check failure on line 1 in CRC start usecase

See this annotation in the file changed.

@github-actions github-actions / JUnit Test Report

CRC start usecase

Step starting CRC with default bundle succeeds: command '$env:CRC_DISABLE_UPDATE_CHECK=true; crc start -p 'C:\Users\rhqp\crc-qe\pull-secret' ', expected to succeed, exited with exit code: 1
Command stdout: 
Command stderr: true : The term 'true' is not recognized as the name of a cmdlet, function, script file, or operable program. Check 
the spelling of the name, or if a path was included, verify that the path is correct and try again.
At line:1 char:31
+ $env:CRC_DISABLE_UPDATE_CHECK=true; crc start -p 'C:\Users\rhqp\crc-q ...
+                               ~~~~
    + CategoryInfo          : ObjectNotFound: (true:String) [], CommandNotFoundException
    + FullyQualifiedErrorId : CommandNotFoundException
 
level=debug msg="CRC version: 2.40.0+a2dccd\n"
level=debug msg="OpenShift version: 4.16.4\n"
level=debug msg="MicroShift version: 4.16.4\n"
level=debug msg="Running 'crc start'"
level=debug msg="Total memory of system is 34359267328 bytes"
level=debug msg="No new version available. The latest version is 2.40.0"
level=debug msg="Checking file: C:\\Users\\rhqp\\.crc\\machines\\crc\\.crc-exist"
level=info msg="Using bundle path C:\\Users\\rhqp\\.crc\\cache\\crc_hyperv_4.16.4_amd64.crcbundle"
level=info msg="Checking minimum RAM requirements"
level=debug msg="Total memory of system is 34359267328 bytes"
level=info msg="Check if Podman binary exists in: C:\\Users\\rhqp\\.crc\\bin\\oc"
level=info msg="Checking if running in a shell with administrator rights"
level=debug msg="Running '$currentPrincipal = New-Object Security.Principal.WindowsPrincipal([Security.Principal.WindowsIdentity]::GetCurrent());$currentPrincipal.IsInRole([Security.Principal.WindowsBuiltInRole]::Administrator)'"
level=info msg="Checking Windows release"
level=debug msg="Running '(Get-ItemProperty -Path \"HKLM:\\SOFTWARE\\Microsoft\\Windows NT\\CurrentVersion\" -Name ReleaseId).ReleaseId'"
level=info msg="Checking Windows edition"
level=debug msg="Running '(Get-ItemProperty -Path \"HKLM:\\SOFTWARE\\Microsoft\\Windows NT\\CurrentVersion\").EditionID'"
level=debug msg="Running on Windows Enterprise edition"
level=info msg="Checking if Hyper-V is installed and operational"
level=debug msg="Running '@(Get-Wmiobject Win32_ComputerSystem).HypervisorPresent'"
level=debug msg="Running '@(Get-Service vmms).Status'"
level=info msg="Checking if Hyper-V service is enabled"
level=debug msg="Running '@(Get-Service vmms).Status'"
level=info msg="Checking if crc-users group exists"
level=debug msg="Running 'Get-LocalGroup -Name crc-users'"
level=info msg="Checking if current user is in crc-users and Hyper-V admins group"
level=debug msg="Running '(Get-LocalGroupMember -Group 'crc-users').Name'"
level=debug msg="Checking current user is in the 'crc-user' group"
level=debug msg="group members: mapt15b00158\\rhqp"
level=debug msg="Running '(Get-LocalGroupMember -SID 'S-1-5-32-578').Name'"
level=debug msg="Checking current user is in the 'Hyper-v Administrators' group"
level=debug msg="group members: mapt15b00158\\rhqp"
level=info msg="Checking if vsock is correctly configured"
level=debug msg="Running 'Get-Item -Path \"HKLM:\\SOFTWARE\\Microsoft\\Windows NT\\CurrentVersion\\Virtualization\\GuestCommunicationServices\\00000400-FACB-11E6-BD58-64006A7986D3\"'"
level=info msg="Checking if the win32 background launcher is installed"
level=debug msg="Running '(Get-Item 'C:\\Program Files\\Red Hat OpenShift Local\\crc-background-launcher.exe').VersionInfo.FileVersion'"
level=debug msg="Found crc-background-launcher.exe version 0.0.0.1"
level=info msg="Checking if the daemon task is installed"
level=debug msg="Running 'Get-ScheduledTask -TaskName crcDaemon'"
level=debug msg="Running '(Get-ScheduledTask -TaskName \"crcDaemon\").Version'"
level=info msg="Checking if the daemon task is running"
level=debug msg="Running '(Get-ScheduledTask -TaskName \"crcDaemon\").State'"
level=info msg="Checking admin helper service is running"
level=debug msg="Running '(Get-Service crcAdminHelper).Status'"
level=info msg="Checking SSH port availability"
level=debug msg="Checking file: C:\\Users\\rhqp\\.crc\\machines\\crc\\.crc-exist"
level=debug msg="Copying 'C:\\Users\\rhqp\\.crc\\cache\\crc_hyperv_4.16.4_amd64\\oc.exe' to 'C:\\Users\\rhqp\\.crc\\bin\\oc\\oc.exe'"
level=debug msg="Copying 'C:\\Users\\rhqp\\.crc\\cache\\crc_hyperv_4.16.4_amd64\\podman.exe' to 'C:\\Users\\rhqp\\.crc\\bin\\podman\\podman.exe'"
level=info msg="Loading bundle: crc_hyperv_4.16.4_amd64..."
level=debug msg="Using secret from configuration"
level=info msg="Creating CRC VM for OpenShift 4.16.4..."
level=debug msg="Running pre-create checks..."
level=debug msg="Running '@(Get-Module -ListAvailable hyper-v).Name | Get-Unique'"
level=debug msg="Running '@([Security.Principal.WindowsPrincipal][Security.Principal.WindowsIdentity]::GetCurrent()).IsInRole(([System.Security.Principal.SecurityIdentifier]::new(\"S-1-5-32-578\")))'"
level=debug msg="Creating machine..."
level=debug msg="Creating VM..."
level=debug msg="Running 'Hyper-V\\New-VM crc -Path 'C:\\Users\\rhqp\\.crc\\machines\\crc' -MemoryStartupBytes 16000MB'"
level=debug msg="Running 'Hyper-V\\Remove-VMNetworkAdapter -VMName crc'"
level=debug msg="Running 'Hyper-V\\Set-VMMemory -VMName crc -DynamicMemoryEnabled $false'"
level=debug msg="Running 'Hyper-V\\Set-VMProcessor crc -Count 4'"
level=debug msg="Running 'Hyper-V\\Set-VM -VMName crc -AutomaticStartAction Nothing -AutomaticStopAction ShutDown -CheckpointType Disabled'"
level=debug msg="Running 'Hyper-V\\Add-VMHardDiskDrive -VMName crc -Path 'C:\\Users\\rhqp\\.crc\\machines\\crc\\crc.vhdx''"
level=debug msg="Running '@(Get-VHD -Path 'C:\\Users\\rhqp\\.crc\\machines\\crc\\crc.vhdx').Size'"
level=debug msg="C:\\Users\\rhqp\\.crc\\machines\\crc\\crc.vhdx is already 33285996544 bytes"
level=info msg="Generating new SSH key pair..."
level=info msg="Generating new password for the kubeadmin user"
level=debug msg="Created C:\\Users\\rhqp\\.crc\\machines\\crc\\.crc-exist"
level=debug msg="Machine successfully created"
level=debug msg="Checking file: C:\\Users\\rhqp\\.crc\\machines\\crc\\.crc-exist"
level=debug msg="Running 'Hyper-V\\Get-VM crc | Select-Object -ExpandProperty State'"
level=debug msg="Copying 'C:\\Users\\rhqp\\.crc\\cache\\crc_hyperv_4.16.4_amd64\\oc.exe' to 'C:\\Users\\rhqp\\.crc\\bin\\oc\\oc.exe'"
level=debug msg="Copying 'C:\\Users\\rhqp\\.crc\\cache\\crc_hyperv_4.16.4_amd64\\podman.exe' to 'C:\\Users\\rhqp\\.crc\\bin\\podman\\podman.exe'"
level=info msg="Starting CRC VM for openshift 4.16.4..."
level=debug msg="Updating CRC VM configuration"
level=debug msg="Running 'Hyper-V\\Start-VM crc'"
level=debug msg="Waiting for machine to be running, this may take a few minutes..."
level=debug msg="retry loop: attempt 0"
level=debug msg="Running 'Hyper-V\\Get-VM crc | Select-Object -ExpandProperty State'"
level=debug msg="Machine is up and running!"
level=debug msg="Running 'Hyper-V\\Get-VM crc | Select-Object -ExpandProperty State'"
level=info msg="CRC instance is running with IP 127.0.0.1"
level=debug msg="Waiting until ssh is available"
level=debug msg="retry loop: attempt 0"
level=debug msg="Running SSH command: exit 0"
level=debug msg="Using ssh private keys: [C:\\Users\\rhqp\\.crc\\machines\\crc\\id_ecdsa C:\\Users\\rhqp\\.crc\\cache\\crc_hyperv_4.16.4_amd64\\id_ecdsa_crc]"
level=debug msg="SSH command results: err: ssh: handshake failed: read tcp 127.0.0.1:50891->127.0.0.1:2222: wsarecv: An existing connection was forcibly closed by the remote host., output: "
level=debug msg="error: Temporary error: ssh command error:\ncommand : exit 0\nerr     : ssh: handshake failed: read tcp 127.0.0.1:50891->127.0.0.1:2222: wsarecv: An existing connection was forcibly closed by the remote host.\n - sleeping 1s"
level=debug msg="retry loop: attempt 1"
level=debug msg="Running SSH command: exit 0"
level=debug msg="Using ssh private keys: [C:\\Users\\rhqp\\.crc\\machines\\crc\\id_ecdsa C:\\Users\\rhqp\\.crc\\cache\\crc_hyperv_4.16.4_amd64\\id_ecdsa_crc]"
level=debug msg="SSH command results: err: ssh: handshake failed: read tcp 127.0.0.1:50892->127.0.0.1:2222: wsarecv: An existing connection was forcibly closed by the remote host., output: "
level=debug msg="error: Temporary error: ssh command error:\ncommand : exit 0\nerr     : ssh: handshake failed: read tcp 127.0.0.1:50892->127.0.0.1:2222: wsarecv: An existing connection was forcibly closed by the remote host.\n - sleeping 1s"
level=debug msg="retry loop: attempt 2"
level=debug msg="Running SSH command: exit 0"
level=debug msg="Using ssh private keys: [C:\\Users\\rhqp\\.crc\\machines\\crc\\id_ecdsa C:\\Users\\rhqp\\.crc\\cache\\crc_hyperv_4.16.4_amd64\\id_ecdsa_crc]"
level=debug msg="SSH command results: err: ssh: handshake failed: read tcp 127.0.0.1:50905->127.0.0.1:2222: wsarecv: An existing connection was forcibly closed by the remote host., output: "
level=debug msg="error: Temporary error: ssh command error:\ncommand : exit 0\nerr     : ssh: handshake failed: read tcp 127.0.0.1:50905->127.0.0.1:2222: wsarecv: An existing connection was forcibly closed by the remote host.\n - sleeping 1s"
level=debug msg="retry loop: attempt 3"
level=debug msg="Running SSH command: exit 0"
level=debug msg="Using ssh private keys: [C:\\Users\\rhqp\\.crc\\machines\\crc\\id_ecdsa C:\\Users\\rhqp\\.crc\\cache\\crc_hyperv_4.16.4_amd64\\id_ecdsa_crc]"
level=debug msg="SSH command results: err: ssh: handshake failed: read tcp 127.0.0.1:50907->127.0.0.1:2222: wsarecv: An existing connection was forcibly closed by the remote host., output: "
level=debug msg="error: Temporary error: ssh command error:\ncommand : exit 0\nerr     : ssh: handshake failed: read tcp 127.0.0.1:50907->127.0.0.1:2222: wsarecv: An existing connection was forcibly closed by the remote host.\n - sleeping 1s"
level=debug msg="retry loop: attempt 4"
level=debug msg="Running SSH command: exit 0"
level=debug msg="Using ssh private keys: [C:\\Users\\rhqp\\.crc\\machines\\crc\\id_ecdsa C:\\Users\\rhqp\\.crc\\cache\\crc_hyperv_4.16.4_amd64\\id_ecdsa_crc]"
level=debug msg="SSH command results: err: <nil>, output: "
level=info msg="CRC VM is running"
level=debug msg="Using root access: disable core user password"
level=debug msg="Running SSH command: sudo passwd --lock core"
level=debug msg="SSH command results: err: <nil>, output: Locking password for user core.\npasswd: Success\n"
level=debug msg="Running SSH command: cat /home/core/.ssh/authorized_keys"
level=debug msg="SSH command results: err: Process exited with status 1, output: "
level=info msg="Updating authorized keys..."
level=debug msg="Creating /home/core/.ssh/authorized_keys with permissions 0644 in the CRC VM"
level=debug msg="Running SSH command: <hidden>"
level=debug msg="SSH command succeeded"
level=debug msg="Running SSH command: rm /home/core/.ssh/authorized_keys.d/ignition"
level=debug msg="SSH command results: err: <nil>, output: "
level=debug msg="Using root access: Get device id"
level=debug msg="Running SSH command: sudo /usr/sbin/blkid -t TYPE=xfs -o device"
level=debug msg="SSH command results: err: <nil>, output: /dev/sda4\n"
level=debug msg="Using root access: Growing /dev/sda4 partition"
level=debug msg="Running SSH command: sudo /usr/bin/growpart /dev/sda 4"
level=debug msg="SSH command results: err: Process exited with status 1, output: NOCHANGE: partition 4 is size 63961055. it cannot be grown\n"
level=debug msg="No free space after /dev/sda4, nothing to do"
level=debug msg="Using root access: make root Podman socket accessible"
level=debug msg="Running SSH command: sudo chmod 777 /run/podman/ /run/podman/podman.sock"
level=debug msg="SSH command results: err: <nil>, output: "
level=debug msg="Running SSH command: systemctl status ovs-configuration.service"
level=debug msg="SSH command results: err: Process exited with status 3, output: ○ ovs-configuration.service - Configures OVS with proper host networking configuration\n     Loaded: loaded (/etc/systemd/system/ovs-configuration.service; enabled; preset: disabled)\n     Active: inactive (dead)\n"
level=debug msg="Using root access: Executing systemctl daemon-reload command"
level=debug msg="Running SSH command: sudo systemctl daemon-reload"
level=debug msg="SSH command results: err: <nil>, output: "
level=debug msg="Using root access: Executing systemctl start ovs-configuration.service"
level=debug msg="Running SSH command: sudo systemctl start ovs-configuration.service"
level=debug msg="SSH command results: err: <nil>, output: "
level=debug msg="Using root access: Update resolv.conf file"
level=debug msg="Running SSH command: sudo nmcli con modify --temporary ovs-if-br-ex ipv4.dns 192.168.127.1 ipv4.dns-search crc.testing"
level=debug msg="SSH command results: err: <nil>, output: "
level=debug msg="Using root access: Executing systemctl daemon-reload command"
level=debug msg="Running SSH command: sudo systemctl daemon-reload"
level=debug msg="SSH command results: err: <nil>, output: "
level=debug msg="Using root access: Executing systemctl restart NetworkManager.service"
level=debug msg="Running SSH command: sudo systemctl restart NetworkManager.service"
level=debug msg="SSH command results: err: <nil>, output: "
level=debug msg="retry loop: attempt 0"
level=debug msg="Running SSH command: host -R 3 foo.apps-crc.testing"
level=debug msg="SSH command results: err: <nil>, output: foo.apps-crc.testing has address 192.168.127.2\n"
level=info msg="Check internal and public DNS query..."
level=debug msg="Running SSH command: curl --head quay.io"
level=debug msg="SSH command results: err: <nil>, output: HTTP/1.1 301 Moved Permanently\r\nServer: awselb/2.0\r\nDate: Wed, 07 Aug 2024 12:37:58 GMT\r\nContent-Type: text/html\r\nContent-Length: 134\r\nConnection: keep-alive\r\nLocation: https://quay.io:443/\r\n\r\n"
level=info msg="Check DNS query from host..."
level=debug msg="api.crc.testing resolved to [127.0.0.1]"
level=info msg="Verifying validity of the kubelet certificates..."
level=debug msg="Running SSH command: date --date=\"$(sudo openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -noout -enddate | cut -d= -f 2)\" --iso-8601=seconds"
level=debug msg="SSH command results: err: <nil>, output: 2025-07-30T10:53:17+00:00\n"
level=debug msg="Running SSH command: date --date=\"$(sudo openssl x509 -in /var/lib/kubelet/pki/kubelet-server-current.pem -noout -enddate | cut -d= -f 2)\" --iso-8601=seconds"
level=debug msg="SSH command results: err: <nil>, output: 2025-07-30T10:54:23+00:00\n"
level=debug msg="Running SSH command: date --date=\"$(sudo openssl x509 -in /etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/aggregator-client/tls.crt -noout -enddate | cut -d= -f 2)\" --iso-8601=seconds"
level=debug msg="SSH command results: err: <nil>, output: 2025-07-30T10:57:10+00:00\n"
level=info msg="Starting kubelet service"
level=debug msg="Using root access: Executing systemctl daemon-reload command"
level=debug msg="Running SSH command: sudo systemctl daemon-reload"
level=debug msg="SSH command results: err: <nil>, output: "
level=debug msg="Using root access: Executing systemctl start kubelet"
level=debug msg="Running SSH command: sudo systemctl start kubelet"
level=debug msg="SSH command results: err: <nil>, output: "
level=info msg="Waiting for kube-apiserver availability... [takes around 2min]"
level=debug msg="retry loop: attempt 0"
level=debug msg="Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig"
level=debug msg="SSH command results: err: Process exited with status 1, output: "
level=debug msg="E0807 12:38:07.714674    3792 memcache.go:265] couldn't get current server API group list: Get \"https://api.crc.testing:6443/api?timeout=32s\": dial tcp 192.168.127.2:6443: connect: connection refused\nE0807 12:38:07.716492    3792 memcache.go:265] couldn't get current server API group list: Get \"https://api.crc.testing:6443/api?timeout=32s\": dial tcp 192.168.127.2:6443: connect: connection refused\nE0807 12:38:07.717463    3792 memcache.go:265] couldn't get current server API group list: Get \"https://api.crc.testing:6443/api?timeout=32s\": dial tcp 192.168.127.2:6443: connect: connection refused\nE0807 12:38:07.718677    3792 memcache.go:265] couldn't get current server API group list: Get \"https://api.crc.testing:6443/api?timeout=32s\": dial tcp 192.168.127.2:6443: connect: connection refused\nE0807 12:38:07.719850    3792 memcache.go:265] couldn't get current server API group list: Get \"https://api.crc.testing:6443/api?timeout=32s\": dial tcp 192.168.127.2:6443: connect: connection refused\nThe connection to the server api.crc.testing:6443 was refused - did you specify the right host or port?\n"
level=debug msg="error: Temporary error: ssh command error:\ncommand : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig\nerr     : Process exited with status 1\n - sleeping 1s"
level=debug msg="retry loop: attempt 1"
level=debug msg="Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig"
level=debug msg="SSH command results: err: Process exited with status 1, output: "
level=debug msg="E0807 12:38:08.914761    4188 memcache.go:265] couldn't get current server API group list: Get \"https://api.crc.testing:6443/api?timeout=32s\": dial tcp 192.168.127.2:6443: connect: connection refused\nE0807 12:38:08.915739    4188 memcache.go:265] couldn't get current server API group list: Get \"https://api.crc.testing:6443/api?timeout=32s\": dial tcp 192.168.127.2:6443: connect: connection refused\nE0807 12:38:08.917402    4188 memcache.go:265] couldn't get current server API group list: Get \"https://api.crc.testing:6443/api?timeout=32s\": dial tcp 192.168.127.2:6443: connect: connection refused\nE0807 12:38:08.918050    4188 memcache.go:265] couldn't get current server API group list: Get \"https://api.crc.testing:6443/api?timeout=32s\": dial tcp 192.168.127.2:6443: connect: connection refused\nE0807 12:38:08.920013    4188 memcache.go:265] couldn't get current server API group list: Get \"https://api.crc.testing:6443/api?timeout=32s\": dial tcp 192.168.127.2:6443: connect: connection refused\nThe connection to the server api.crc.testing:6443 was refused - did you specify the right host or port?\n"
level=debug msg="error: Temporary error: ssh command error:\ncommand : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig\nerr     : Process exited with status 1\n - sleeping 1s"
level=debug msg="retry loop: attempt 2"
level=debug msg="Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig"
level=debug msg="SSH command results: err: Process exited with status 1, output: "
level=debug msg="E0807 12:38:10.082329    4628 memcache.go:265] couldn't get current server API group list: Get \"https://api.crc.testing:6443/api?timeout=32s\": dial tcp 192.168.127.2:6443: connect: connection refused\nE0807 12:38:10.083740    4628 memcache.go:265] couldn't get current server API group list: Get \"https://api.crc.testing:6443/api?timeout=32s\": dial tcp 192.168.127.2:6443: connect: connection refused\nE0807 12:38:10.086183    4628 memcache.go:265] couldn't get current server API group list: Get \"https://api.crc.testing:6443/api?timeout=32s\": dial tcp 192.168.127.2:6443: connect: connection refused\nE0807 12:38:10.087730    4628 memcache.go:265] couldn't get current server API group list: Get \"https://api.crc.testing:6443/api?timeout=32s\": dial tcp 192.168.127.2:6443: connect: connection refused\nE0807 12:38:10.088958    4628 memcache.go:265] couldn't get current server API group list: Get \"https://api.crc.testing:6443/api?timeout=32s\": dial tcp 192.168.127.2:6443: connect: connection refused\nThe connection to the server api.crc.testing:6443 was refused - did you specify the right host or port?\n"
level=debug msg="error: Temporary error: ssh command error:\ncommand : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig\nerr     : Process exited with status 1\n - sleeping 1s"
level=debug msg="retry loop: attempt 3"
level=debug msg="Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig"
level=debug msg="SSH command results: err: Process exited with status 124, output: "
level=debug
level=debug msg="error: Temporary error: ssh command error:\ncommand : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig\nerr     : Process exited with status 124\n - sleeping 1s"
level=debug msg="retry loop: attempt 4"
level=debug msg="Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig"
level=debug msg="SSH command results: err: Process exited with status 124, output: "
level=debug
level=debug msg="error: Temporary error: ssh command error:\ncommand : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig\nerr     : Process exited with status 124\n - sleeping 1s"
level=debug msg="retry loop: attempt 5"
level=debug msg="Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig"
level=debug msg="SSH command results: err: <nil>, output: NAME   STATUS     ROLES                         AGE   VERSION\ncrc    NotReady   control-plane,master,worker   9d    v1.29.6+aba1e8d\n"
level=debug msg="NAME   STATUS     ROLES                         AGE   VERSION\ncrc    NotReady   control-plane,master,worker   9d    v1.29.6+aba1e8d\n"
level=debug msg="Waiting for availability of resource type 'configmap'"
level=debug msg="retry loop: attempt 0"
level=debug msg="Running SSH command: timeout 5s oc get configmap --context admin --cluster crc --kubeconfig /opt/kubeconfig"
level=debug msg="SSH command results: err: <nil>, output: NAME                       DATA   AGE\nkube-root-ca.crt           1      9d\nopenshift-service-ca.crt   1      9d\n"
level=debug msg="NAME                       DATA   AGE\nkube-root-ca.crt           1      9d\nopenshift-service-ca.crt   1      9d\n"
level=debug msg="Running SSH command: timeout 30s oc delete -n openshift-machine-config-operator configmap machine-config-controller --context admin --cluster crc --kubeconfig /opt/kubeconfig"
level=debug msg="SSH command results: err: Process exited with status 1, output: "
level=debug msg="Waiting for availability of resource type 'lease'"
level=debug msg="retry loop: attempt 0"
level=debug msg="Running SSH command: timeout 5s oc get lease --context admin --cluster crc --kubeconfig /opt/kubeconfig"
level=debug msg="SSH command results: err: <nil>, output: "
level=debug
level=debug msg="Running SSH command: timeout 30s oc delete -A lease --all --context admin --cluster crc --kubeconfig /opt/kubeconfig"
level=debug msg="SSH command results: err: <nil>, output: lease.coordination.k8s.io \"crc\" deleted\nlease.coordination.k8s.io \"apiserver-yz2phfmetulqy4pemhbirhdnii\" deleted\nlease.coordination.k8s.io \"cluster-policy-controller-lock\" deleted\nlease.coordination.k8s.io \"kube-controller-manager\" deleted\nlease.coordination.k8s.io \"kube-scheduler\" deleted\nlease.coordination.k8s.io \"openshift-apiserver-operator-lock\" deleted\nlease.coordination.k8s.io \"cluster-authentication-operator-lock\" deleted\nlease.coordination.k8s.io \"cluster-machine-approver-leader\" deleted\nlease.coordination.k8s.io \"version\" deleted\nlease.coordination.k8s.io \"config-operator-lock\" deleted\nlease.coordination.k8s.io \"console-operator-lock\" deleted\nlease.coordination.k8s.io \"openshift-controller-manager-operator-lock\" deleted\nlease.coordination.k8s.io \"openshift-master-controllers\" deleted\nlease.coordination.k8s.io \"openshift-cluster-etcd-operator-lock\" deleted\nlease.coordination.k8s.io \"openshift-master-controllers\" deleted\nlease.coordination.k8s.io \"kube-apiserver-operator-lock\" deleted\nlease.coordination.k8s.io \"cert-regeneration-controller-lock\" deleted\nlease.coordination.k8s.io \"kube-controller-manager-operator-lock\" deleted\nlease.coordination.k8s.io \"cert-recovery-controller-lock\" deleted\nlease.coordination.k8s.io \"cluster-policy-controller-lock\" deleted\nlease.coordination.k8s.io \"openshift-cluster-kube-scheduler-operator-lock\" deleted\nlease.coordination.k8s.io \"cert-recovery-controller-lock\" deleted\nlease.coordination.k8s.io \"kube-scheduler\" deleted\nlease.coordination.k8s.io \"openshift-kube-storage-version-migrator-operator-lock\" deleted\nlease.coordination.k8s.io \"control-plane-machine-set-leader\" deleted\nlease.coordination.k8s.io \"machine-api-operator\" deleted\nlease.coordination.k8s.io \"machine-config\" deleted\nlease.coordination.k8s.io \"machine-config-controller\" deleted\nlease.coordination.k8s.io \"marketplace-operator-lock\" deleted\nlease.coordination.k8s.io \"ovnkube-identity\" deleted\nlease.coordination.k8s.io \"network-operator-lock\" deleted\nlease.coordination.k8s.io \"packageserver-controller-lock\" deleted\nlease.coordination.k8s.io \"ovn-kubernetes-master\" deleted\nlease.coordination.k8s.io \"openshift-route-controllers\" deleted\nlease.coordination.k8s.io \"service-ca-operator-lock\" deleted\nlease.coordination.k8s.io \"service-ca-controller-lock\" deleted\n"
level=debug msg="Waiting for availability of resource type 'secret'"
level=debug msg="retry loop: attempt 0"
level=debug msg="Running SSH command: timeout 5s oc get secret --context admin --cluster crc --kubeconfig /opt/kubeconfig"
level=debug msg="SSH command results: err: <nil>, output: NAME                       TYPE                      DATA   AGE\nbuilder-dockercfg-jt25m    kubernetes.io/dockercfg   1      8d\ndefault-dockercfg-24gr5    kubernetes.io/dockercfg   1      8d\ndeployer-dockercfg-l6r6j   kubernetes.io/dockercfg   1      8d\n"
level=debug msg="NAME                       TYPE                      DATA   AGE\nbuilder-dockercfg-jt25m    kubernetes.io/dockercfg   1      8d\ndefault-dockercfg-24gr5    kubernetes.io/dockercfg   1      8d\ndeployer-dockercfg-l6r6j   kubernetes.io/dockercfg   1      8d\n"
level=debug msg="Running SSH command: <hidden>"
level=debug msg="SSH command succeeded"
level=info msg="Adding user's pull secret to the cluster..."
level=debug msg="Running SSH command: <hidden>"
level=debug msg="SSH command succeeded"
level=debug msg="Waiting for availability of resource type 'machineconfigs'"
level=debug msg="retry loop: attempt 0"
level=debug msg="Running SSH command: timeout 5s oc get machineconfigs --context admin --cluster crc --kubeconfig /opt/kubeconfig"
level=debug msg="SSH command results: err: <nil>, output: NAME                                               GENERATEDBYCONTROLLER                      IGNITIONVERSION   AGE\n00-master                                          d70a17fc8fee26d27c0624522f000504838f4b22   3.4.0             9d\n00-worker                                          d70a17fc8fee26d27c0624522f000504838f4b22   3.4.0             9d\n01-master-container-runtime                        d70a17fc8fee26d27c0624522f000504838f4b22   3.4.0             9d\n01-master-kubelet                                  d70a17fc8fee26d27c0624522f000504838f4b22   3.4.0             9d\n01-worker-container-runtime                        d70a17fc8fee26d27c0624522f000504838f4b22   3.4.0             9d\n01-worker-kubelet                                  d70a17fc8fee26d27c0624522f000504838f4b22   3.4.0             9d\n97-master-generated-kubelet                        d70a17fc8fee26d27c0624522f000504838f4b22   3.4.0             9d\n97-worker-generated-kubelet                        d70a17fc8fee26d27c0624522f000504838f4b22   3.4.0             9d\n98-master-generated-kubelet                        d70a17fc8fee26d27c0624522f000504838f4b22   3.4.0             9d\n98-worker-generated-kubelet                        d70a17fc8fee26d27c0624522f000504838f4b22   3.4.0             9d\n99-master-generated-registries                     d70a17fc8fee26d27c0624522f000504838f4b22   3.4.0             9d\n99-master-ssh                                                                                 3.2.0             9d\n99-node-sizing-for-crc                                                                        3.2.0             9d\n99-openshift-machineconfig-master-dummy-networks                                              3.2.0             9d\n99-worker-generated-registries                     d70a17fc8fee26d27c0624522f000504838f4b22   3.4.0             9d\n99-worker-ssh                                                                                 3.2.0             9d\nrendered-master-bbb3ba43e267fd340b8a9ef009fc8c10   d70a17fc8fee26d27c0624522f000504838f4b22   3.4.0             8d\nrendered-worker-ff77eb300072526cc60d287c6b3b1326   d70a17fc8fee26d27c0624522f000504838f4b22   3.4.0             8d\n"
level=debug msg="NAME                                               GENERATEDBYCONTROLLER                      IGNITIONVERSION   AGE\n00-master                                          d70a17fc8fee26d27c0624522f000504838f4b22   3.4.0             9d\n00-worker                                          d70a17fc8fee26d27c0624522f000504838f4b22   3.4.0             9d\n01-master-container-runtime                        d70a17fc8fee26d27c0624522f000504838f4b22   3.4.0             9d\n01-master-kubelet                                  d70a17fc8fee26d27c0624522f000504838f4b22   3.4.0             9d\n01-worker-container-runtime                        d70a17fc8fee26d27c0624522f000504838f4b22   3.4.0             9d\n01-worker-kubelet                                  d70a17fc8fee26d27c0624522f000504838f4b22   3.4.0             9d\n97-master-generated-kubelet                        d70a17fc8fee26d27c0624522f000504838f4b22   3.4.0             9d\n97-worker-generated-kubelet                        d70a17fc8fee26d27c0624522f000504838f4b22   3.4.0             9d\n98-master-generated-kubelet                        d70a17fc8fee26d27c0624522f000504838f4b22   3.4.0             9d\n98-worker-generated-kubelet                        d70a17fc8fee26d27c0624522f000504838f4b22   3.4.0             9d\n99-master-generated-registries                     d70a17fc8fee26d27c0624522f000504838f4b22   3.4.0             9d\n99-master-ssh                                                                                 3.2.0             9d\n99-node-sizing-for-crc                                                                        3.2.0             9d\n99-openshift-machineconfig-master-dummy-networks                                              3.2.0             9d\n99-worker-generated-registries                     d70a17fc8fee26d27c0624522f000504838f4b22   3.4.0             9d\n99-worker-ssh                                                                                 3.2.0             9d\nrendered-master-bbb3ba43e267fd340b8a9ef009fc8c10   d70a17fc8fee26d27c0624522f000504838f4b22   3.4.0             8d\nrendered-worker-ff77eb300072526cc60d287c6b3b1326   d70a17fc8fee26d27c0624522f000504838f4b22   3.4.0             8d\n"
level=debug msg="Running SSH command: timeout 30s oc get machineconfigs 99-master-ssh -o jsonpath='{.spec.config.passwd.users[0].sshAuthorizedKeys[0]}' --context admin --cluster crc --kubeconfig /opt/kubeconfig"
level=debug msg="SSH command results: err: <nil>, output: ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAFg2/aqXgYip0GM9bo/6Aqm1w72ZVHPnr/gwaP3urcEfqonAOfSTX33l23FDpMA/O5OpsAqA+IimIa77/NgTVXYNADxi0sLQOxIpbLMbS62TKfihC+LfmeUirT1ov4/NVBmq32vkfS4HX/z47BpU05XEQUCVG8fVFaTq1PdeBRiM/+y6w== core"
level=info msg="Updating SSH key to machine config resource..."
level=debug msg="Running SSH command: timeout 30s oc patch machineconfig 99-master-ssh -p '{\"spec\": {\"config\": {\"passwd\": {\"users\": [{\"name\": \"core\", \"sshAuthorizedKeys\": [\"ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAFFfK9nq6RZZa3juuJlSD0ndONyY6Fp6d2srtqvcD7lSqyuozj4/uFFxcs26S5mZfeagxd27kgGscGCDwd1pxh5tQGGpjj5BSowrOQXZYoYb6W2MM+8Pe1UpwrvfbIJCjwkhFUnsBnMlqvnHRzWxRj3m2oup6G2+DxspWEALKklAJ6tbg==\"]}]}}}}' --type merge --context admin --cluster crc --kubeconfig /opt/kubeconfig"
level=debug msg="SSH command results: err: <nil>, output: machineconfig.machineconfiguration.openshift.io/99-master-ssh patched\n"
level=info msg="Waiting until the user's pull secret is written to the instance disk..."
level=debug msg="retry loop: attempt 0"
level=debug msg="Running SSH command: <hidden>"
level=debug msg="SSH command succeeded"
level=debug msg="error: Temporary error: pull secret not updated to disk - sleeping 2s"
level=debug msg="retry loop: attempt 1"
level=debug msg="Running SSH command: <hidden>"
level=debug msg="SSH command succeeded"
level=debug msg="error: Temporary error: pull secret not updated to disk - sleeping 2s"
level=debug msg="retry loop: attempt 2"
level=debug msg="Running SSH command: <hidden>"
level=debug msg="SSH command succeeded"
level=debug msg="error: Temporary error: pull secret not updated to disk - sleeping 2s"
level=debug msg="retry loop: attempt 3"
level=debug msg="Running SSH command: <hidden>"
level=debug msg="SSH command succeeded"
level=debug msg="error: Temporary error: pull secret not updated to disk - sleeping 2s"
level=debug msg="retry loop: attempt 4"
level=debug msg="Running SSH command: <hidden>"
level=debug msg="SSH command succeeded"
level=debug msg="error: Temporary error: pull secret not updated to disk - sleeping 2s"
level=debug msg="retry loop: attempt 5"
level=debug msg="Running SSH command: <hidden>"
level=debug msg="SSH command succeeded"
level=debug msg="error: Temporary error: pull secret not updated to disk - sleeping 2s"
level=debug msg="retry loop: attempt 6"
level=debug msg="Running SSH command: <hidden>"
level=debug msg="SSH command succeeded"
level=debug msg="error: Temporary error: pull secret not updated to disk - sleeping 2s"
level=debug msg="retry loop: attempt 7"
level=debug msg="Running SSH command: <hidden>"
level=debug msg="SSH command succeeded"
level=debug msg="error: Temporary error: pull secret not updated to disk - sleeping 2s"
level=debug msg="retry loop: attempt 8"
level=debug msg="Running SSH command: <hidden>"
level=debug msg="SSH command succeeded"
level=debug msg="error: Temporary error: pull secret not updated to disk - sleeping 2s"
level=debug msg="retry loop: attempt 9"
level=debug msg="Running SSH command: <hidden>"
level=debug msg="SSH command succeeded"
level=debug msg="error: Temporary error: pull secret not updated to disk - sleeping 2s"
level=debug msg="retry loop: attempt 10"
level=debug msg="Running SSH command: <hidden>"
level=debug msg="SSH command succeeded"
level=debug msg="error: Temporary error: pull secret not updated to disk - sleeping 2s"
level=debug msg="retry loop: attempt 11"
level=debug msg="Running SSH command: <hidden>"
level=debug msg="SSH command succeeded"
level=debug msg="error: Temporary error: pull secret not updated to disk - sleeping 2s"
level=debug msg="retry loop: attempt 12"
level=debug msg="Running SSH command: <hidden>"
level=debug msg="SSH command succeeded"
level=debug msg="error: Temporary error: pull secret not updated to disk - sleeping 2s"
level=debug msg="retry loop: attempt 13"
level=debug msg="Running SSH command: <hidden>"
level=debug msg="SSH command succeeded"
level=debug msg="error: Temporary error: pull secret not updated to disk - sleeping 2s"
level=debug msg="retry loop: attempt 14"
level=debug msg="Running SSH command: <hidden>"
level=debug msg="SSH command succeeded"
level=debug msg="error: Temporary error: pull secret not updated to disk - sleeping 2s"
level=debug msg="retry loop: attempt 15"
level=debug msg="Running SSH command: <hidden>"
level=debug msg="SSH command succeeded"
level=debug msg="error: Temporary error: pull secret not updated to disk - sleeping 2s"
level=debug msg="retry loop: attempt 16"
level=debug msg="Running SSH command: <hidden>"
level=debug msg="SSH command succeeded"
level=debug msg="error: Temporary error: pull secret not updated to disk - sleeping 2s"
level=debug msg="retry loop: attempt 17"
level=debug msg="Running SSH command: <hidden>"
level=debug msg="SSH command succeeded"
level=debug msg="error: Temporary error: pull secret not updated to disk - sleeping 2s"
level=debug msg="retry loop: attempt 18"
level=debug msg="Running SSH command: <hidden>"
level=debug msg="SSH command succeeded"
level=debug msg="error: Temporary error: pull secret not updated to disk - sleeping 2s"
level=debug msg="retry loop: attempt 19"
level=debug msg="Running SSH command: <hidden>"
level=debug msg="SSH command succeeded"
level=debug msg="error: Temporary error: pull secret not updated to disk - sleeping 2s"
level=debug msg="retry loop: attempt 20"
level=debug msg="Running SSH command: <hidden>"
level=debug msg="SSH command succeeded"
level=debug msg="error: Temporary error: pull secret not updated to disk - sleeping 2s"
level=debug msg="retry loop: attempt 21"
level=debug msg="Running SSH command: <hidden>"
level=debug msg="SSH command succeeded"
level=debug msg="error: Temporary error: pull secret not updated to disk - sleeping 2s"
level=debug msg="retry loop: attempt 22"
level=debug msg="Running SSH command: <hidden>"
level=debug msg="SSH command succeeded"
level=debug msg="error: Temporary error: pull secret not updated to disk - sleeping 2s"
level=debug msg="retry loop: attempt 23"
level=debug msg="Running SSH command: <hidden>"
level=debug msg="SSH command succeeded"
level=debug msg="error: Temporary error: pull secret not updated to disk - sleeping 2s"
level=debug msg="retry loop: attempt 24"
level=debug msg="Running SSH command: <hidden>"
level=debug msg="SSH command succeeded"
level=debug msg="error: Temporary error: pull secret not updated to disk - sleeping 2s"
level=debug msg="retry loop: attempt 25"
level=debug msg="Running SSH command: <hidden>"
level=debug msg="SSH command succeeded"
level=debug msg="error: Temporary error: pull secret not updated to disk - sleeping 2s"
level=debug msg="retry loop: attempt 26"
level=debug msg="Running SSH command: <hidden>"
level=debug msg="SSH command succeeded"
level=debug msg="error: Temporary error: pull secret not updated to disk - sleeping 2s"
level=debug msg="retry loop: attempt 27"
level=debug msg="Running SSH command: <hidden>"
level=debug msg="SSH command succeeded"
level=debug msg="error: Temporary error: pull secret not updated to disk - sleeping 2s"
level=debug msg="retry loop: attempt 28"
level=debug msg="Running SSH command: <hidden>"
level=debug msg="SSH command succeeded"
level=debug msg="error: Temporary error: pull secret not updated to disk - sleeping 2s"
level=debug msg="retry loop: attempt 29"
level=debug msg="Running SSH command: <hidden>"
level=debug msg="SSH command succeeded"
level=debug msg="error: Temporary error: pull secret not updated to disk - sleeping 2s"
level=debug msg="retry loop: attempt 30"
level=debug msg="Running SSH command: <hidden>"
level=debug msg="SSH command succeeded"
level=debug msg="error: Temporary error: pull secret not updated to disk - sleeping 2s"
level=debug msg="retry loop: attempt 31"
level=debug msg="Running SSH command: <hidden>"
level=debug msg="SSH command succeeded"
level=debug msg="error: Temporary error: pull secret not updated to disk - sleeping 2s"
level=debug msg="retry loop: attempt 32"
level=debug msg="Running SSH command: <hidden>"
level=debug msg="SSH command succeeded"
level=debug msg="error: Temporary error: pull secret not updated to disk - sleeping 2s"
level=debug msg="retry loop: attempt 33"
level=debug msg="Running SSH command: <hidden>"
level=debug msg="SSH command succeeded"
level=debug msg="error: Temporary error: pull secret not updated to disk - sleeping 2s"
level=debug msg="retry loop: attempt 34"
level=debug msg="Running SSH command: <hidden>"
level=debug msg="SSH command succeeded"
level=debug msg="error: Temporary error: pull secret not updated to disk - sleeping 2s"
level=debug msg="retry loop: attempt 35"
level=debug msg="Running SSH command: <hidden>"
level=debug msg="SSH command succeeded"
level=debug msg="error: Temporary error: pull secret not updated to disk - sleeping 2s"
level=debug msg="retry loop: attempt 36"
level=debug msg="Running SSH command: <hidden>"
level=debug msg="SSH command succeeded"
level=debug msg="error: Temporary error: pull secret not updated to disk - sleeping 2s"
level=debug msg="retry loop: attempt 37"
level=debug msg="Running SSH command: <hidden>"
level=debug msg="SSH command succeeded"
level=debug msg="error: Temporary error: pull secret not updated to disk - sleeping 2s"
level=debug msg="retry loop: attempt 38"
level=debug msg="Running SSH command: <hidden>"
level=debug msg="SSH command succeeded"
level=debug msg="error: Temporary error: pull secret not updated to disk - sleeping 2s"
level=debug msg="retry loop: attempt 39"
level=debug msg="Running SSH command: <hidden>"
level=debug msg="SSH command succeeded"
level=debug msg="error: Temporary error: pull secret not updated to disk - sleeping 2s"
level=debug msg="retry loop: attempt 40"
level=debug msg="Running SSH command: <hidden>"
level=debug msg="SSH command succeeded"
level=debug msg="error: Temporary error: pull secret not updated to disk - sleeping 2s"
level=debug msg="retry loop: attempt 41"
level=debug msg="Running SSH command: <hidden>"
level=debug msg="SSH command succeeded"
level=debug msg="error: Temporary error: pull secret not updated to disk - sleeping 2s"
level=debug msg="retry loop: attempt 42"
level=debug msg="Running SSH command: <hidden>"
level=debug msg="SSH command succeeded"
level=debug msg="error: Temporary error: pull secret not updated to disk - sleeping 2s"
level=debug msg="retry loop: attempt 43"
level=debug msg="Running SSH command: <hidden>"
level=debug msg="SSH command succeeded"
level=debug msg="error: Temporary error: pull secret not updated to disk - sleeping 2s"
level=debug msg="retry loop: attempt 44"
level=debug msg="Running SSH command: <hidden>"
level=debug msg="SSH command succeeded"
level=debug msg="error: Temporary error: pull secret not updated to disk - sleeping 2s"
level=debug msg="retry loop: attempt 45"
level=debug msg="Running SSH command: <hidden>"
level=debug msg="SSH command succeeded"
level=debug msg="Waiting for availability of resource type 'secret'"
level=debug msg="retry loop: attempt 0"
level=debug msg="Running SSH command: timeout 5s oc get secret --context admin --cluster crc --kubeconfig /opt/kubeconfig"
level=debug msg="SSH command results: err: <nil>, output: NAME                       TYPE                      DATA   AGE\nbuilder-dockercfg-jt25m    kubernetes.io/dockercfg   1      8d\ndefault-dockercfg-24gr5    kubernetes.io/dockercfg   1      8d\ndeployer-dockercfg-l6r6j   kubernetes.io/dockercfg   1      8d\n"
level=debug msg="NAME                       TYPE                      DATA   AGE\nbuilder-dockercfg-jt25m    kubernetes.io/dockercfg   1      8d\ndefault-dockercfg-24gr5    kubernetes.io/dockercfg   1      8d\ndeployer-dockercfg-l6r6j   kubernetes.io/dockercfg   1      8d\n"
level=debug msg="Running SSH command: <hidden>"
level=debug msg="SSH command succeeded"
level=info msg="Changing the password for the kubeadmin user"
level=debug msg="Running SSH command: <hidden>"
level=debug msg="SSH command succeeded"
level=debug msg="Waiting for availability of resource type 'clusterversion'"
level=debug msg="retry loop: attempt 0"
level=debug msg="Running SSH command: timeout 5s oc get clusterversion --context admin --cluster crc --kubeconfig /opt/kubeconfig"
level=debug msg="SSH command results: err: <nil>, output: NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS\nversion   4.16.4    True        False         9d      Cluster version is 4.16.4\n"
level=debug msg="NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS\nversion   4.16.4    True        False         9d      Cluster version is 4.16.4\n"
level=debug msg="Running SSH command: timeout 30s oc get clusterversion version -o jsonpath=\"{['spec']['clusterID']}\" --context admin --cluster crc --kubeconfig /opt/kubeconfig"
level=debug msg="SSH command results: err: <nil>, output: "
level=info msg="Updating cluster ID..."
level=debug msg="Running SSH command: timeout 30s oc patch clusterversion version -p '{\"spec\":{\"clusterID\":\"b2041f1a-acbe-4cef-ad00-10a4f1a1d2aa\"}}' --type merge --context admin --cluster crc --kubeconfig /opt/kubeconfig"
level=debug msg="SSH command results: err: <nil>, output: clusterversion.config.openshift.io/version patched\n"
level=debug msg="Running SSH command: ls /opt/crc/routes-controller.yaml"
level=debug msg="SSH command results: err: <nil>, output: /opt/crc/routes-controller.yaml\n"
level=debug msg="Running SSH command: timeout 30s oc apply -f /opt/crc/routes-controller.yaml --context admin --cluster crc --kubeconfig /opt/kubeconfig"
level=debug msg="SSH command results: err: Process exited with status 124, output: "
ssh command error:
command : timeout 30s oc apply -f /opt/crc/routes-controller.yaml --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 124