How to deploy Quay with RHOCS backend on OCP 4.6

Introduction

Quay is a secure, private container registry that builds, analyzes and distributes container images. In this post we will show how we deploy it on top of OCP 4 and how we use OCS Multi-Cloud Object Gateway object service to provide a storage backend to Quay. We will also enable Clair container vulnerability analysis service.

Lab Virtual Env Overview

OCP 4.6 UPI baremetal deployment
All oc commands are issued from a CentOS 8 bastion node
3 Masters (16Go of RAM) and 3 Workers (32Go of RAM) VMs

[root@bastion ~] oc get nodes
NAME                        STATUS   ROLES    AGE   VERSION
master0.mycluster.lab.local   Ready    master   10h   v1.19.0+7070803
master1.mycluster.lab.local   Ready    master   10h   v1.19.0+7070803
master2.mycluster.lab.local   Ready    master   10h   v1.19.0+7070803
worker0.mycluster.lab.local   Ready    worker   10h   v1.19.0+7070803
worker1.mycluster.lab.local   Ready    worker   10h   v1.19.0+7070803
worker2.mycluster.lab.local   Ready    worker   10h   v1.19.0+7070803

OCS 4.6 has been deployed

[root@bastion ~] oc get cephcluster -n openshift-storage
NAME                             DATADIRHOSTPATH   MONCOUNT   AGE     PHASE   MESSAGE                        HEALTH
ocs-storagecluster-cephcluster   /var/lib/rook     3          5h17m   Ready   Cluster created successfully   HEALTH_OK

Simple deployment

Let's Create a dedicated Namespace and deploy Quay Operator

cat << EOF | oc apply -f -
apiVersion: v1
kind: Namespace
metadata:
  labels:
    openshift.io/cluster-monitoring: "true"
  name: local-quay
---
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: local-quay
  namespace: local-quay
spec:
  targetNamespaces:
  - local-quay
---
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: quay-operator
  namespace: local-quay
spec:
  channel: quay-v3.4
  installPlanApproval: Automatic
  name: quay-operator
  source: redhat-operators
  sourceNamespace: openshift-marketplace
EOF

We wait for the operator to be ready

[root@bastion ~] oc get pod -n local-quay
NAME                                                    READY   STATUS    RESTARTS   AGE
quay-operator-7cb56c7c5f-7dgwk                          1/1     Running   0          6s

Create an OpenShift secret to be able to pull the required container images

oc create -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: redhat-pull-secret
data:
  .dockerconfigjson:  <Quay Key>
type: kubernetes.io/dockerconfigjson
EOF
Get the key here https://access.redhat.com/solutions/3533201

RHOCS Object storage service provides an S3 API and is based on NooBaa project.

To interact with the service, we need to:

Install Noobaa CLI on our Bastion

OS="linux"
VERSION=$(curl -s https://api.github.com/repos/noobaa/noobaa-operator/releases/latest | jq -r '.name')
curl -LO https://github.com/noobaa/noobaa-operator/releases/download/$VERSION/noobaa-$OS-$VERSION
chmod +x noobaa-$OS-$VERSION
mv noobaa-$OS-$VERSION /usr/local/bin/noobaa

Full documentation can be found here: https://github.com/noobaa/noobaa-operator

Retrieve S3 API credentials and endpoint

[root@bastion ~] noobaa status -n openshift-storage
INFO[0000] CLI version: 2.3.0                           
INFO[0000] noobaa-image: noobaa/noobaa-core:5.5.0       
INFO[0000] operator-image: noobaa/noobaa-operator:2.3.0
INFO[0000] Namespace: openshift-storage                 
 .
 .
 .
#----------------#
#- S3 Addresses -#
#----------------#
ExternalDNS : [https://s3-openshift-storage.mycluster.lab.local]
ExternalIP  : []
NodePorts   : [https://10.226.115.10:30795]
InternalDNS : [https://s3.openshift-storage.svc:443]
InternalIP  : [https://172.30.20.203:443]
PodPorts    : [https://10.131.0.90:6443]
#------------------#
#- S3 Credentials -#
#------------------#
AWS_ACCESS_KEY_ID     : FSfXsNiWnva1lmDnzZ1c
AWS_SECRET_ACCESS_KEY : r+RpYkH8Es81ZipMjngCmHUfMvIa4TLYOlTJwLgE
#------------------#
#- Backing Stores -#
#------------------#
NAME                           TYPE            TARGET-BUCKET                             PHASE   AGE         
noobaa-default-backing-store   s3-compatible   nb.1609358827719.mycluster.lab.local   Ready   94h48m18s   
#------------------#
#- Bucket Classes -#
#------------------#
NAME                          PLACEMENT                                                             PHASE   AGE         
noobaa-default-bucket-class   {Tiers:[{Placement: BackingStores:[noobaa-default-backing-store]}]}   Ready   94h48m18s   
#-----------------#
#- Bucket Claims -#
#-----------------#
No OBCs found.

Take good notes of ExternalDNS, InternalDNS, AWS_ACCESS_KEY_ID and AWS_ACCESS_KEY_ID as we will need it to create our Quay Echosystem

Install s3cmd and create .s3cmd config (This is not mandatory but makes manipulation of buckets and objects easier)

[root@bastion ~] dnf install s3cmd -y
[root@bastion ~] cat .s3cfg
[default]
access_key = FSfXsNiWnva1lmDnzZ1c
secret_key = r+RpYkH8Es81ZipMjngCmHUfMvIa4TLYOlTJwLgE
host_base = s3-openshift-storage.mycluster.lab.local  ##We use ExternalDNS retrieved with nooba command
host_bucket = s3-openshift-storage.mycluster.lab.local
check_ssl_certificate = False
check_ssl_hostname = False
use_http_expect = False
use_https = True
signature_v2 = True
signurl_use_https = True

Create a bucket for Quay

[root@bastion ~]# s3cmd mb  s3://quay1
Bucket 's3://quay1/' created

We can now create a QuayEcosystem with the following command:

oc create  -f - <<EOF
apiVersion: redhatcop.redhat.io/v1alpha1
kind: QuayEcosystem
metadata:
  name: quayecosystem1
spec:
  clair:
    enabled: true
    imagePullSecretName: redhat-pull-secret
  quay:
    imagePullSecretName: redhat-pull-secret
    registryBackends:
      - name: rhocs
        rhocs:
          hostname: s3.openshift-storage.svc.cluster.local:443
          accessKey: FSfXsNiWnva1lmDnzZ1c
          secretKey: r+RpYkH8Es81ZipMjngCmHUfMvIa4TLYOlTJwLgE
          bucketName: quay1
EOF

Let's take a closer look :

First we create a QuayEcossytem object and name it quayecosystem1

apiVersion: redhatcop.redhat.io/v1alpha1
kind: QuayEcosystem
metadata:
  name: quayecosystem1

We specify we want to enable Clair and we use redhat-pull-secret so we can pull the necessary image

  spec:
    clair:
      enabled: true
      imagePullSecretName: redhat-pull-secret

This is the most important part (see comments)

quay:
  imagePullSecretName: redhat-pull-secret
  registryBackends:
    - name: rhocs
      rhocs:    # We specify Quay we are using rhcos as storage backend
        hostname: s3.openshift-storage.svc.cluster.local:443 #Make sure InternalDNS and Port are specified here
        accessKey: FSfXsNiWnva1lmDnzZ1c
        secretKey: r+RpYkH8Es81ZipMjngCmHUfMvIa4TLYOlTJwLgE
        bucketName: quay1 # This is the bucket we created previously

hostname field needs to be specified with the port otherwise you'll get the infamous error:

 'Failed to Validate Component: registry-storage Validation Failed: Invalid storage configuration: rhocs: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:727)'    


* After a few minutes we check all pods are up and running
```bash
[root@bastion ~] oc get pod
NAME                                               READY   STATUS    RESTARTS   AGE
quay-operator-7cb56c7c5f-7dgwk                     1/1     Running   0          4d5h
quayecosystem1-clair-77f7844f4f-75lh8              1/1     Running   0          3h7m
quayecosystem1-clair-postgresql-5c69478878-bfmtn   1/1     Running   0          3h7m
quayecosystem1-quay-84cbc66cbd-mdw4j               1/1     Running   0          3h8m
quayecosystem1-quay-config-5bf565888c-dtvpv        1/1     Running   0          3h9m
quayecosystem1-quay-postgresql-c7645f945-nzhrr     1/1     Running   0          3h9m
quayecosystem1-redis-68945f56b7-xk67g              1/1     Running   0          3h10m

We can now find out the route and access Quay GUI at https://quayecosystem1-quay-local-quay.mycluster.lab.local (login: quay pass: password)

[root@esxi-bastion ~] oc get route
NAME                         HOST/PORT                                                      PATH   SERVICES                     PORT   TERMINATION            WILDCARD
quayecosystem1-quay          quayecosystem1-quay-local-quay.mycluster.lab.local                 quayecosystem1-quay          8443   passthrough/Redirect   None
quayecosystem1-quay-config   quayecosystem1-quay-config-local-quay.mycluster.lab.local          quayecosystem1-quay-config   8443   passthrough/Redirect   None

## To be continued

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

README.md

README.md

How to deploy Quay with RHOCS backend on OCP 4.6

Introduction

Lab Virtual Env Overview

Simple deployment

Files

README.md

Latest commit

History

README.md

File metadata and controls

How to deploy Quay with RHOCS backend on OCP 4.6

Introduction

Lab Virtual Env Overview

Simple deployment