How to set OPAL-client as a sidecar with Istio EnvoyFilter

[WellyHong]

Hi,

We have built microservice with k8s & istio , then follow this opa-istio tutorial to add OPA be an authorization solution and it works well.

Now we would like to choose OPAL to solve policy & data update realtime issue.

After deploying OPAL to k8s, connection between both server & client is well.

Next step I wanna replace opa sidecar by opal-client, is there an example or guidance? Any advice would be appreciated.

[orweis]

Hi @WellyHong !
Sounds like a plan :)

In short you can run the OPAL client sidecar, it would run OPA as part of it by default for you.

I'd suggest starting from here: https://docs.opal.ac/getting-started/get_started_with_opal_docker_compose_tutorial

and here:
https://youtu.be/A5adHlkmdC0

An then taking a look at the helm charts repo:
https://github.com/permitio/opal-helm-chart

And of course joining the slack community:
bit.ly/permit-slack

[orweis]

It seems like trying to connect/write to OPA fails.

Please check it responds on its port, and that the data and policy you provide are valid Rego and Json.

@asafc - got more ideas?

[WellyHong]

I can confirm policy rego & policy data are both correct, services have worked well for a long while with istio envoy-opa solution.
I didn't modify any port setting, is it the crux of problem?

[orweis]

We need some more information here to help.

• Can you share the config you're using for the OPAL client, and OPA?

• Can you share your log from OPA?

• Are you using the OPA run by OPAL-client as a subprocess, or are you running it separately?

• Did you try pushing the data to OPA directly to see that it works well

[asafc]

from the log, it looks like writing to OPA from OPAL works, and OPAL only terminates because it got a SIGTERM (ctrl+c).

My guess is that istio expects to find OPA at another port (you mentioned gRPC on port 9191) but OPAL client will run OPA by default on port 8181 and expose the REST API.

OPAL client has configs in place to run OPA with different settings (port, etc) but i will need to know exactly how you want OPA to run and then we can help with configuration.

[WellyHong]

The repeatedly shutdown/restart problem is due to the wrong livenessProbe/readinessProbe configuration, we have figure it out, and OPAL client can fetch policy data from our server correctly.

Thank you . 🙂

[WellyHong]

Hi @orweis

We have deployed opal server/client to k8s via using_docker and set relative config via env variable.

In some cases we want to set opal-client config with .env file, what path should it mounts to container(/.env ?), and content should like

OPAL_CLIENT_TOKEN=eyJ0eXAiOiJKV1QiLCJhbGci.OiJSUzI1NiJ9.do6LZp2b
OPAL_SERVER_URL=http://host-domain/

is it right?

[orweis]

Hi @WellyHong ,
OPAL's support for main files is based on python's Decouple module - you can see the formats there.

Your .env file format seems correct :)

The .env file needs to go at the root of OPAL's folder (server or client), if I'm not mistaken that means next to the main.py file.

@asafc @orishavit Can you doublecheck my statement here ?

[orishavit]

Hey @WellyHong

You can mount a .env file to /usr/.env, with the format you used above.

Another option to do this, if you're using K8S ConfigMaps/Secrets is to use envFrom in the Pod specification:
See: https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/#configure-all-key-value-pairs-in-a-configmap-as-container-environment-variables)

[WellyHong]

in the end, we put it to /opal/.env, now it works as expected, thanks for your advice. 👍

[WellyHong]

Hi @orishavit , @orweis

I upgrade from opal-client-standalone:0.1.21 to 0.7.0, the .env file cannot be loaded correctly.
With 0.1.21, I mount /opal volume and put .env under it,
but 0.7.0 does not work normally.
I searched the document and found this topic, it doesn't instruct the .env path.
Could you give me a guidance to mount .env correctly with 0.7.0?

[orweis]

Hi @WellyHong I couldn't find a change affecting this; I'll ask @orishavit to take a closer look next week.

[orishavit]

It appears that the search path for .env files has changed at some point, I will open a ticket to restore /opal/.env.

In the mean time, you can place your env file in /usr/.env.

[WellyHong]

Hi @orishavit ,

thanks for your reply.

more information,
I write OPAL_CLIENT_TOKEN & OPAL_DATA_TOPICS into .env file
0.1.21 runs normally
0.7.0, errors take place as below

[1] [INFO] Starting gunicorn 20.1.0
[1] [INFO] Listening at: http://0.0.0.0:7000 (1)
[1] [INFO] Using worker: uvicorn.workers.UvicornWorker
[8] [INFO] Booting worker with pid: 8
opal_client.policy_store.opa_client     | INFO  | Authentication mode for policy store: PolicyStoreAuth.NONE
opal_common.fetcher.fetcher_register    | INFO  | Loading FetcherProvider 'FastApiRpcFetchProvider' found at: <class 'opal_common.fetcher.providers.fastapi_rpc_fetch_provider.FastApiRpcFetchProvider'>
opal_common.fetcher.fetcher_register    | INFO  | Loading FetcherProvider 'HttpFetchProvider' found at: <class 'opal_common.fetcher.providers.http_fetch_provider.HttpFetchProvider'>
opal_common.fetcher.fetcher_register    | INFO  | Fetcher Register loaded
opal_client.callbacks.register          | INFO  | Callbacks register loaded
... omit ...
fastapi_websocket_pubsub.pub_sub_client | INFO  | Trying to connect to Pub/Sub server - ws://opal-server-sandbox.opal-test.svc.cluster.local/ws
fastapi_websocket_rpc.websocket_rpc_c...| INFO  | Trying server - ws://opal-server-sandbox.opal-test.svc.cluster.local/ws
fastapi_websocket_rpc.websocket_rpc_c...| INFO  | RPC Websocket failed - with invalid status code 403
fastapi_websocket_pubsub.pub_sub_client | ERROR  | Connection failed with an invalid status code - 403 -- Won't try to reconnect.
Traceback (most recent call last):
... omit ...
websockets.exceptions.InvalidStatusCode: server rejected WebSocket connection: HTTP 403
fastapi_websocket_rpc.websocket_rpc_c...| INFO  | RPC Websocket failed - with invalid status code 403
fastapi_websocket_pubsub.pub_sub_client | ERROR  | Connection failed with an invalid status code - 403 -- Won't try to reconnect.
Traceback (most recent call last):

[WellyHong]

below is the k8s Deployment setting

apiVersion: apps/v1
kind: Deployment
metadata:
  name: opal-client
  namespace: opal
spec:
  replicas: 1
  selector:
    matchLabels:
      app: opal-client
  template:
    metadata:
      labels:
        app: opal-client
      annotations:
        rollme: {{ randAlphaNum 5 | quote }}
    spec:
      containers:
      - name: opal-client
        image: permitio/opal-client-standalone:0.7.0
        imagePullPolicy: Always
        securityContext:
          runAsUser: 1000
          runAsNonRoot: true
          allowPrivilegeEscalation: false
        volumeMounts:
        - mountPath: /opal/.env
          subPath: ".env"
          name: opal-volume
      volumes:
      - name: opal-volume
        emptyDir: {}

[orishavit]

Looks good, just set mountPath: /usr/.env

[WellyHong]

Hi @orishavit, @orweis

Cause container run with runAsNonRoot: true & runAsUser: 1000(opal)

Create /usr/.env shows permission denied error, it needs root privilege.

Do you have any updates for this issue? Hope it can mount /opal/.env as before.

[rahulaga01]

When we enable the istio on the namespace level, it automatically injects the sidecar. How can we use opal-server with Invoy-proxy?

Please share some simple steps for this.

[orweis]

HI @rahulaga01 - you can connect the OPA that is being managed by the OPAL-client to Envoy.
Use this OPA version: https://github.com/open-policy-agent/opa-envoy-plugin and run it standalone and configure the OPAL client to connect to it: https://docs.opal.ac/getting-started/running-opal/run-opal-client/standalone-opa-uri

[rahulaga01]

We deployed the opa-envoy plugin as a standalone, here is the config.yaml:

---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: opa-envoy
  namespace: opal
  labels:
    app: opa-envoy
spec:
  replicas: 1
  selector:
    matchLabels:
      app: opa-envoy
  template:
    metadata:
      labels:
        app: opa-envoy
    spec:
      containers:
        - name: opa-envoy
          securityContext:
            capabilities:
              drop: ["ALL"]
            runAsNonRoot: true
            allowPrivilegeEscalation: false
            seccompProfile:
              type: "RuntimeDefault"
          image: openpolicyagent/opa:0.55.0-envoy-2-rootless
          ports:
            - name: http
              containerPort: 8181
          args:
            - "run"
            - "--server"
            - "--addr=0.0.0.0:8181"
            - "--diagnostic-addr=0.0.0.0:8282"
            - "--set=plugins.envoy_ext_authz_grpc.addr=:9191"
            - "--set=plugins.envoy_ext_authz_grpc.path=envoy/authz/allow"
            - "--set=decision_logs.console=true"
            - "--set=status.console=true"
            - "--ignore=.*"
          livenessProbe:
            httpGet:
              path: /health?plugins
              scheme: HTTP
              port: 8282
            initialDelaySeconds: 5
            periodSeconds: 5
          readinessProbe:
            httpGet:
              path: /health?plugins
              scheme: HTTP
              port: 8282
            initialDelaySeconds: 5
            periodSeconds: 5
---
kind: Service
apiVersion: v1
metadata:
  name: opa-envoy
  labels:
    app: opa-envoy
  namespace: opal
spec:
  selector:
    app: opa-envoy
  ports:
  - name: http
    protocol: TCP
    port: 8181
    targetPort: 8181

---

We configure this in opal-client:

OPAL_POLICY_STORE_URL: http://opa-envoy:8181/

Pods are running fine, and now we are to test envoy-proxy with opal. Can you please describe the next step how can we test this one?

We mainly following this article:
https://raw.githubusercontent.com/open-policy-agent/opa-envoy-plugin/main/quick_start.yaml

[orweis]

You simply run the OPAL client and point it to the OPA instance you deployed -as described in the docs link I've shared with you