The pods are not in Running state

[jvaidyan]

I deployed the latest connaisseur helm chart in an openshift cluster. The Pods are not in running state.
The describe pod doesnt show much useful information other than the Readiness probe failed.
The logs also shows only the probe information. How can I troubleshoot to see the underlying issue?

Logs
[2021-09-27 18:20:09,666] INFO: 10.131.0.1 - - [27/Sep/2021 18:20:09] "GET /ready HTTP/1.1" 500 -
[2021-09-27 18:20:09,877] INFO: 10.128.2.1 - - [27/Sep/2021 18:20:09] "GET /health HTTP/1.1" 200 -
[2021-09-27 18:20:10,721] INFO: 10.128.2.1 - - [27/Sep/2021 18:20:10] "GET /ready HTTP/1.1" 500 -
[2021-09-27 18:20:11,712] INFO: 10.129.2.1 - - [27/Sep/2021 18:20:11] "GET /ready HTTP/1.1" 500 -

$ kubectl get all -n connaisseur
NAME READY STATUS RESTARTS AGE
pod/connaisseur-deployment-5ff859845f-2rmwp 0/1 Running 0 105m
pod/connaisseur-deployment-5ff859845f-k8ndj 0/1 Running 0 105m
pod/connaisseur-deployment-5ff859845f-ths7h 0/1 Running 0 105m
pod/demo-595648d5f4-c7gtr 1/1 Running 0 10m

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/connaisseur-svc ClusterIP 172.30.158.139 443/TCP 105m

NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/connaisseur-deployment 0/3 3 0 105m
deployment.apps/demo 1/1 1 1 10m

NAME DESIRED CURRENT READY AGE
replicaset.apps/connaisseur-deployment-5ff859845f 3 3 0 105m
replicaset.apps/demo-595648d5f4 1 1 1 10m

[jvaidyan]

The helm chart uses the default values. No overrides for the Notary server.

[jvaidyan]

Sure . Will checkout the 2.2.0 version

[xopham]

Awesome. Will let you know

[xopham]

@jvaidyan
We have implemented some major improvements in the upcoming release v2.2.0 that also aim to improve compatibility and resolve some of the installation issues. We still need to fix some minor problems that were found during testing already, but hope to release it within the next 2 weeks 🥳

I prepared a release candidate for you that is not the final version, but would help us see whether it resolves your issues. To test it, please follow the steps below:

1. Switch to the test/release-candidate-v2.2.0 branch.
2. Make sure to remove all prior Connaisseur resources, e.g. make annihilate or follow the docs on deleting.
3. Depending on your Kubernetes version (e.g. <v1.19) or flavor (e.g. Openshift), adjust the securityContext in helm/values.yaml. The fields in questions contain comments on needed changes.
4. Install connaisseur, e.g. make install or follow the docs on deploying. Connaisseur should come up much quicker than before.
5. Once Connaisseur has come up, kubectl get all -n connaisseur should show the resources as described in the docs on checking.
6. Test Connaisseur using the following test images (notary v1 and cosign):

# cosign-based signatures
kubectl run cosigned --image=xoph/demo:signed   # is accepted
kubectl run counsigned --image=xoph/demo:unsigned   # is denied

# notary-based signatures
kubectl run signed --image=docker.io/securesystemsengineering/testimage:signed  # is accepted
kubectl run unsigned --image=docker.io/securesystemsengineering/testimage:unsigned  # is denied

Please let me know, if that works for you or whether you encounter any issues during installation. If something fails, please share your logs via kubectl describe and/or kubectl logs of failing components.

Hope this helps 🙂

[mrjoelkamp]

@xopham the test/release-candidate-v2.2.0 branch fixed installation errors for me when running on a local Kind cluster. Prior to this version, installation would fail.

[xopham]

@mrjoelkamp awesome. Thanks for letting us know 🙏
We are currently merging the last few fixes and will probably release tomorrow 🚀