"Error: Failed to push" when launching "meepctl dockerize all" #423

fehmi8 · 2023-01-01T19:06:23Z

fehmi8
Jan 1, 2023

Hi there!

I'm trying to get AdvantEDGE installed by following the instructions here:
https://interdigitalinc.github.io/AdvantEDGE/docs/platform-mgmt/mgmt-workflow/

When I reach the "Containerize core micro-services" step, I get this:

-:~/AdvantEDGE$ meepctl dockerize all
Using repo config file: /home/fehmi/AdvantEDGE/.meepctl-repocfg.yaml
Using meepctl config file: /home/fehmi/.meepctl.yaml
Using docker registry: meep-docker-registry:30001
-- meep-auth-svc --
   + copy docker file
   + copy api files
   + copy docker data
   + dockerize meep-auth-svc
Error: Failed to push  meep-docker-registry:30001/meep-auth-svc  with error:  exit status 1
[...]

-:~/AdvantEDGE$ meepctl dockerize meep-auth-svc -v
[...]
Successfully tagged meep-docker-registry:30001/meep-auth-svc:latest

Cmd: [docker push meep-docker-registry:30001/meep-auth-svc]
Result: Using default tag: latest
The push refers to repository [meep-docker-registry:30001/meep-auth-svc]
Get "https://meep-docker-registry:30001/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

Error: Failed to push  meep-docker-registry:30001/meep-auth-svc  with error:  exit status 1

I guess I messed up the local registry setup... Here's what I have:

========================================
meep.gitdir : /home/fehmi/AdvantEDGE
meep.registry : meep-docker-registry:30001
meep.workdir : /home/fehmi/.meep
node.ip : 10.50.2.72
========================================

Got the node IP above from here:

-:~/AdvantEDGE$ kubectl get nodes -o wide
NAME   STATUS   ROLES           AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION     CONTAINER-RUNTIME
main   Ready    control-plane   3d    v1.26.0   10.50.2.72    <none>        Ubuntu 20.04.1 LTS   5.4.0-53-generic   containerd://1.6.14

/etc/hosts:

127.0.0.1 localhost
10.106.5.249 meep-docker-registry

I got the registry IP address above from here:

-:~/AdvantEDGE$ kubectl get services | grep registry
meep-docker-registry                       NodePort    10.106.5.249     <none>        5000:30001/TCP                  2d2h

Any help troubleshooting this would be appreciated!
Thanks!

Answered by fehmi8

Jan 10, 2023

Thanks to Kevin @dilallkx and his answer here and here (on another thread), this has been solved.

I first changed the IP of meep-docker-registry in my /etc/hosts file to my main node IP. Also:

-:~$ id
uid=1001(fehmi) gid=1001(fehmi) groups=1001(fehmi),27(sudo),998(docker)

My user id is 1001 instead of the default 1000 (root), so the docker-registry folder was owned by 1001:

-:~$ ls -aln ~/.meep/
total 60
drwxrwxr-x 15 1001 1001 4096 Jan  1 18:15 .
drwxr-xr-x 13 1001 1001 4096 Jan 10 02:30 ..
drwxrwxr-x  2 1001 1001 4096 Dec 29 22:51 certs
drwxrwxr-x 17 1001 1001 4096 Jan  1 18:15 codecov
drwxrwxr-x 17 1001 1001 4096 Dec 30 16:38 codecov-bak
drwxr-xr-x  3 5984 5984 4096 Dec 29 21:16 couch…

View full answer

dilallkx · 2023-01-09T14:01:16Z

dilallkx
Jan 9, 2023
Maintainer

Hi @fehmi8,

Thank you for reaching out!

Your issue is most likely caused by the docker registry configuration in /etc/hosts. The IP address that should be used in /etc/hosts is the k8s Master node IP address, which in your case is 10.50.2.72. This address must be used when accessing the docker registry via the k8s NodePort (i.e. 30001) in order to reach the service.

Note that the service IP address (i.e. 10.106.5.249 in your case) can be used as well, however port 5000 must be used instead of the NodePort in this case. Also, the docker image name tag must include the correct port number in order to successfully push the image to the docker registry. I recommend using port 30001 and setting your meep-docker-registry IP address to your Master node IP address.

Also, please note that docker registry issues may also occur when the k8s CA has not been added to the list of trusted CAs. It is important to re-run step 7 of the k8s installation procedure in this case.

I hope this helps resolve your issue.
Best regards!

3 replies

fehmi8 Jan 10, 2023
Author

Hi Kevin,

Thank you so much for taking the time to answer, please bear with me!

meep-docker-registry is now 10.50.2.72 in /etc/hosts, but still no luck:

-:~$ meepctl dockerize meep-auth-svc -v
Using repo config file: /home/fehmi/AdvantEDGE/.meepctl-repocfg.yaml
Using meepctl config file: /home/fehmi/.meepctl.yaml
Dockerize called
[arg]  targets: [meep-auth-svc]
[flag] registry: 
[flag] verbose: true
[flag] time: false
Using docker registry: meep-docker-registry:30001
-- meep-auth-svc --
[...]
Successfully tagged meep-docker-registry:30001/meep-auth-svc:latest

Cmd: [docker push meep-docker-registry:30001/meep-auth-svc]
Result: Using default tag: latest
The push refers to repository [meep-docker-registry:30001/meep-auth-svc]
Get "https://meep-docker-registry:30001/v2/": dial tcp 10.50.2.72:30001: connect: connection refused

Error: Failed to push  meep-docker-registry:30001/meep-auth-svc  with error:  exit status 1

I ran again step 7 of the k8s installation procedure as you suggested.

I then ran again meepctl build all and meepctl deploy dep -f (got an error Error installing component [meep-open-map-tiles]), and tried to dockerize again:

-:~$ meepctl dockerize meep-auth-svc -v
Using repo config file: /home/fehmi/AdvantEDGE/.meepctl-repocfg.yaml
Using meepctl config file: /home/fehmi/.meepctl.yaml
Dockerize called
[arg]  targets: [meep-auth-svc]
[flag] registry: 
[flag] verbose: true
[flag] time: false
Using docker registry: meep-docker-registry:30001
-- meep-auth-svc --
[...]
Successfully tagged meep-docker-registry:30001/meep-auth-svc:latest

Cmd: [docker push meep-docker-registry:30001/meep-auth-svc]

Result: Using default tag: latest
The push refers to repository [meep-docker-registry:30001/meep-auth-svc]
eb800a5d1027: Preparing  
372c2410eb84: Preparing  
c630ed4c140f: Preparing  
c4221f50d319: Preparing  
44ed3803702a: Preparing  
e5ffc89b0b26: Preparing  
b4bbbcb136a9: Preparing  
3c816b4ead84: Preparing  
e5ffc89b0b26: Waiting
b4bbbcb136a9: Waiting
3c816b4ead84: Waiting
c630ed4c140f: Retrying in 5 seconds
c4221f50d319: Retrying in 5 seconds
372c2410eb84: Retrying in 5 seconds
eb800a5d1027: Retrying in 5 seconds
44ed3803702a: Retrying in 5 seconds
[...]
c630ed4c140f: Retrying in 1 second
eb800a5d1027: Retrying in 1 second
c4221f50d319: Retrying in 1 second
44ed3803702a: Retrying in 1 second
372c2410eb84: Retrying in 1 second
received unexpected HTTP status: 500 Internal Server Error

Error: Failed to push  meep-docker-registry:30001/meep-auth-svc  with error:  exit status 1

My pods look like this:

-:~$ kubectl get pods --all-namespaces | grep meep
default       meep-couchdb-couchdb-0                           1/1     Running            0              12m
default       meep-docker-registry-86b6b6b849-282dw            1/1     Running            0              11m
default       meep-grafana-66fff596d-tt89f                     1/1     Running            0              11m
default       meep-influxdb-0                                  1/1     Running            0              10m
default       meep-ingress-controller-4p78j                    1/1     Running            0              10m
default       meep-ingress-defaultbackend-5d8778d54f-8f9fq     1/1     Running            0              10m
default       meep-kube-state-metrics-6c75f9d88c-sft8v         1/1     Running            0              9m59s
default       meep-open-map-tiles-8698cccbfb-hjfvd             1/1     Running            0              9m40s
default       meep-postgis-0                                   2/2     Running            0              9m18s
default       meep-prometheus-operator-7d979b99b-d8vjq         1/1     Running            0              8m31s
default       meep-prometheus-prometheus-node-exporter-xxk2t   1/1     Running            0              8m31s
default       meep-redis-master-0                              2/2     Running            0              8m
default       meep-redis-slave-0                               2/2     Running            0              8m
default       meep-ut-couchdb-couchdb-0                        1/1     Running            0              8d
default       meep-ut-influxdb-0                               1/1     Running            0              8d
default       meep-ut-postgis-postgis-0                        2/2     Running            1 (8d ago)     8d
default       meep-ut-redis-master-0                           2/2     Running            0              8d
default       meep-ut-redis-slave-0                            2/2     Running            0              8d
default       prometheus-meep-prometheus-prometheus-0          2/2     Running            0              8m26s

My main node:

-:~$ kubectl describe node
Name:               main 
Roles:              control-plane
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=main
                    kubernetes.io/os=linux
                    node-role.kubernetes.io/control-plane=
                    node.kubernetes.io/exclude-from-external-load-balancers=
Annotations:        kubeadm.alpha.kubernetes.io/cri-socket: unix:///run/containerd/containerd.sock
                    node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Thu, 29 Dec 2022 19:54:06 +0000
Taints:             <none>
Unschedulable:      false
Lease:
  HolderIdentity:  main  
  AcquireTime:     <unset>
  RenewTime:       Tue, 10 Jan 2023 01:37:13 +0000
Conditions:
  Type                 Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----                 ------  -----------------                 ------------------                ------                       -------
  NetworkUnavailable   False   Thu, 29 Dec 2022 19:56:28 +0000   Thu, 29 Dec 2022 19:56:28 +0000   WeaveIsUp                    Weave pod has set this
  MemoryPressure       False   Tue, 10 Jan 2023 01:36:29 +0000   Thu, 29 Dec 2022 19:54:06 +0000   KubeletHasSufficientMemory   kubelet has sufficient memory availabl
e
  DiskPressure         False   Tue, 10 Jan 2023 01:36:29 +0000   Thu, 29 Dec 2022 19:54:06 +0000   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure          False   Tue, 10 Jan 2023 01:36:29 +0000   Thu, 29 Dec 2022 19:54:06 +0000   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready                True    Tue, 10 Jan 2023 01:36:29 +0000   Thu, 29 Dec 2022 19:56:04 +0000   KubeletReady                 kubelet is posting ready status. AppAr
mor enabled
Addresses:
  InternalIP:  10.50.2.72
  Hostname:    main
Capacity:
  cpu:                12 
  ephemeral-storage:  521155356Ki
  hugepages-1Gi:      0  
  hugepages-2Mi:      0  
  memory:             65867764Ki
  pods:               110
Allocatable:
  cpu:                12 
  ephemeral-storage:  480296775295
  hugepages-1Gi:      0  
  hugepages-2Mi:      0  
  memory:             65765364Ki
  pods:               110
System Info:
  Machine ID:                 e16724b258ae4c1ba2656ad012088c33
  System UUID:                e16724b2-58ae-4c1b-a265-6ad012088c33
  Boot ID:                    1d70adee-5fce-4470-9672-84488769f491
  Kernel Version:             5.4.0-53-generic
  OS Image:                   Ubuntu 20.04.1 LTS
  Operating System:           linux
  Architecture:               amd64
  Container Runtime Version:  containerd://1.6.14
  Kubelet Version:            v1.26.0
  Kube-Proxy Version:         v1.26.0
Non-terminated Pods:          (28 in total)
  Namespace                   Name                                              CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
  ---------                   ----                                              ------------  ----------  ---------------  -------------  ---
  default                     meep-couchdb-couchdb-0                            0 (0%)        0 (0%)      0 (0%)           0 (0%)         9m39s
  default                     meep-docker-registry-86b6b6b849-jsh5l             0 (0%)        0 (0%)      0 (0%)           0 (0%)         9m5s
  default                     meep-grafana-66fff596d-hx6kz                      0 (0%)        0 (0%)      0 (0%)           0 (0%)         8m39s
  default                     meep-influxdb-0                                   0 (0%)        0 (0%)      0 (0%)           0 (0%)         8m6s
  default                     meep-ingress-controller-xjrlt                     100m (0%)     0 (0%)      90Mi (0%)        0 (0%)         7m46s
  default                     meep-ingress-defaultbackend-5d8778d54f-vnqcb      0 (0%)        0 (0%)      0 (0%)           0 (0%)         7m46s
  default                     meep-kube-state-metrics-6c75f9d88c-f2j49          0 (0%)        0 (0%)      0 (0%)           0 (0%)         7m24s
  default                     meep-postgis-0                                    250m (2%)     0 (0%)      256Mi (0%)       0 (0%)         6m46s
  default                     meep-prometheus-operator-7d979b99b-7mmpv          0 (0%)        0 (0%)      0 (0%)           0 (0%)         6m5s
  default                     meep-prometheus-prometheus-node-exporter-qrkvk    0 (0%)        0 (0%)      0 (0%)           0 (0%)         6m5s
  default                     meep-redis-master-0                               0 (0%)        0 (0%)      0 (0%)           0 (0%)         5m16s
  default                     meep-redis-slave-0                                0 (0%)        0 (0%)      0 (0%)           0 (0%)         5m16s
  default                     meep-ut-couchdb-couchdb-0                         0 (0%)        0 (0%)      0 (0%)           0 (0%)         8d
  default                     meep-ut-influxdb-0                                0 (0%)        0 (0%)      0 (0%)           0 (0%)         8d
  default                     meep-ut-postgis-postgis-0                         250m (2%)     0 (0%)      256Mi (0%)       0 (0%)         8d
  default                     meep-ut-redis-master-0                            0 (0%)        0 (0%)      0 (0%)           0 (0%)         8d
  default                     meep-ut-redis-slave-0                             0 (0%)        0 (0%)      0 (0%)           0 (0%)         8d
  default                     node-debugger-main-gwktp                          0 (0%)        0 (0%)      0 (0%)           0 (0%)         3h53m
  default                     node-debugger-main-stcfb                          0 (0%)        0 (0%)      0 (0%)           0 (0%)         3h53m
  default                     prometheus-meep-prometheus-prometheus-0           200m (1%)     200m (1%)   50Mi (0%)        50Mi (0%)      5m57s
  kube-system                 coredns-787d4945fb-4tqwj                          100m (0%)     0 (0%)      70Mi (0%)        170Mi (0%)     11d
  kube-system                 coredns-787d4945fb-dgqbg                          100m (0%)     0 (0%)      70Mi (0%)        170Mi (0%)     11d
  kube-system                 etcd-main                                         100m (0%)     0 (0%)      100Mi (0%)       0 (0%)         11d
  kube-system                 kube-apiserver-main                               250m (2%)     0 (0%)      0 (0%)           0 (0%)         11d
  kube-system                 kube-controller-manager-main                      200m (1%)     0 (0%)      0 (0%)           0 (0%)         11d
  kube-system                 kube-proxy-z9477                                  0 (0%)        0 (0%)      0 (0%)           0 (0%)         11d
  kube-system                 kube-scheduler-main                               100m (0%)     0 (0%)      0 (0%)           0 (0%)         11d
  kube-system                 weave-net-lclft                                   100m (0%)     0 (0%)      0 (0%)           0 (0%)         11d
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests     Limits
  --------           --------     ------
  cpu                1750m (14%)  200m (1%)
  memory             892Mi (1%)   390Mi (0%)
  ephemeral-storage  0 (0%)       0 (0%)
  hugepages-1Gi      0 (0%)       0 (0%)
  hugepages-2Mi      0 (0%)       0 (0%)
Events:              <none>

My meep-docker-registry pod:

-:~$ kubectl describe pods meep-docker-registry
Name:             meep-docker-registry-86b6b6b849-282dw
Namespace:        default
Priority:         0
Service Account:  default
Node:             main/10.50.2.72
Start Time:       Tue, 10 Jan 2023 01:45:29 +0000
Labels:           app=docker-registry
                  pod-template-hash=86b6b6b849
                  release=meep-docker-registry
Annotations:      checksum/config: f31bd9cd948aa400401620ba08b4cc9945c5c20870c263734c69ee18994c2760
Status:           Running
IP:               10.32.0.5
IPs:
  IP:           10.32.0.5
Controlled By:  ReplicaSet/meep-docker-registry-86b6b6b849
Containers:
  docker-registry:
    Container ID:  containerd://64296b1f98940e17fc4723bff033ef30f374284eb65f29801a00234c7e24163e
    Image:         registry:2.7.1
    Image ID:      docker.io/library/registry@sha256:169211e20e2f2d5d115674681eb79d21a217b296b43374b8e39f97fcf866b375
    Port:          5000/TCP
    Host Port:     0/TCP 
    Command:
      /bin/registry
      serve
      /etc/docker/registry/config.yml
    State:          Running
      Started:      Tue, 10 Jan 2023 01:45:36 +0000
    Ready:          True 
    Restart Count:  0
    Liveness:       http-get https://:5000/ delay=0s timeout=1s period=10s #success=1 #failure=3
    Readiness:      http-get https://:5000/ delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:
      REGISTRY_HTTP_SECRET:                       <set to the key 'haSharedSecret' in secret 'meep-docker-registry-secret'>  Optional: false
      REGISTRY_HTTP_TLS_CERTIFICATE:              /etc/ssl/docker/tls.crt
      REGISTRY_HTTP_TLS_KEY:                      /etc/ssl/docker/tls.key
      REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY:  /var/lib/registry
    Mounts:
      /etc/docker/registry from meep-docker-registry-config (rw)
      /etc/ssl/docker from tls-cert (ro)
      /var/lib/registry/ from data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-w5fs4 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  meep-docker-registry
    ReadOnly:   false
  meep-docker-registry-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      meep-docker-registry-config
    Optional:  false
  tls-cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  meep-docker-registry
    Optional:    false
  kube-api-access-w5fs4:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  13m   default-scheduler  Successfully assigned default/meep-docker-registry-86b6b6b849-282dw to main
  Normal  Pulled     13m   kubelet            Container image "registry:2.7.1" already present on machine
  Normal  Created    13m   kubelet            Created container docker-registry
  Normal  Started    13m   kubelet            Started container docker-registry

Any other idea?

fehmi8 Jan 10, 2023
Author

Success!

Your comment in another thread made the trick!

dilallkx Jan 10, 2023
Maintainer

Great to hear it is now working!
Let us know if you have any other questions.
Regards!

fehmi8 · 2023-01-10T02:48:06Z

fehmi8
Jan 10, 2023
Author

Thanks to Kevin @dilallkx and his answer here and here (on another thread), this has been solved.

I first changed the IP of meep-docker-registry in my /etc/hosts file to my main node IP. Also:

-:~$ id
uid=1001(fehmi) gid=1001(fehmi) groups=1001(fehmi),27(sudo),998(docker)

My user id is 1001 instead of the default 1000 (root), so the docker-registry folder was owned by 1001:

-:~$ ls -aln ~/.meep/
total 60
drwxrwxr-x 15 1001 1001 4096 Jan  1 18:15 .
drwxr-xr-x 13 1001 1001 4096 Jan 10 02:30 ..
drwxrwxr-x  2 1001 1001 4096 Dec 29 22:51 certs
drwxrwxr-x 17 1001 1001 4096 Jan  1 18:15 codecov
drwxrwxr-x 17 1001 1001 4096 Dec 30 16:38 codecov-bak
drwxr-xr-x  3 5984 5984 4096 Dec 29 21:16 couchdb
drwxrwxr-x  3 1001 1001 4096 Jan 10 02:25 docker-registry
drwxrwxr-x  5  472  472 4096 Dec 29 21:18 grafana
drwxrwxr-x  5 1001 1001 4096 Dec 29 21:17 influxdb
drwxrwxr-x  2 1001 1001 4096 Dec 29 21:16 omt
drwxrwxr-x  3 1001 1001 4096 Dec 29 21:17 postgis
drwxrwxr-x  4 1001 1001 4096 Dec 29 21:16 prometheus
drwxrwxr-x  2 1001 1001 4096 Dec 29 21:16 tmp
drwxrwxr-x  3 1001 1001 4096 Dec 29 21:16 user
drwxrwxr-x  3 1001 1001 4096 Dec 29 21:16 virt-engine

I ran the following command to change its ownership:

-:~$sudo chown -R 1000:1000 ~/.meep/docker-registry/

I had already changed the uid and gid to 1001 in the AdvantEDGE/.meepctl-repocfg.yaml file:

repo:
  name: AdvantEDGE

  #------------------------------
  #  Deployment
  #------------------------------
  deployment:
    # permissions
    permissions:
      # user id
      uid: 1001
      # group id
      gid: 1001

Thanks again, Kevin!

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

"Error: Failed to push" when launching "meepctl dockerize all" #423

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 2 comments 3 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

Select a reply

"Error: Failed to push" when launching "meepctl dockerize all" #423

fehmi8 Jan 1, 2023

Replies: 2 comments · 3 replies

dilallkx Jan 9, 2023 Maintainer

fehmi8 Jan 10, 2023 Author

fehmi8 Jan 10, 2023 Author

dilallkx Jan 10, 2023 Maintainer

fehmi8 Jan 10, 2023 Author

fehmi8
Jan 1, 2023

Replies: 2 comments 3 replies

dilallkx
Jan 9, 2023
Maintainer

fehmi8 Jan 10, 2023
Author

fehmi8 Jan 10, 2023
Author

dilallkx Jan 10, 2023
Maintainer

fehmi8
Jan 10, 2023
Author