KeycloakRealm reconciliation not triggered at control loop interval frequency

[y3lousso]

Describe the bug

KeycloakRealm CR does not reconcile in a loop

To Reproduce

Set the spec.interval to 15s, check the KeycloakRealm Events

apiVersion: keycloak.infra.doodle.com/v1beta1
kind: KeycloakRealm
metadata:
  name: master
  namespace: keycloak
spec:
  interval: 15s
  ...

Check the KeycloakRealm Events

I do not have the Suspended property set.
Basically, the loop stops. If I change some config via the UI, it never gets overriden.

Expected behavior

At interval=15, I would expect 4 reconciliation events per minutes, in this screenshot 4min44 ~ 18 events, but we only get 2.

Environment

• controller version: 2.2.0
• keycloak version: 24.0.1
• kubernetes version: 1.27.7

[raffis]

You really don't want an interval that low.
Also please post your entire .status as well as controller logs.

[y3lousso]

I did a complete wipe of controller & crds, then reinstall and got the following

Controller pod full logs:

│                                                                                                                                                 Autoscroll:On      FullScreen:Off     Timestamps:Off     Wrap:Off                                                                                                                                                  │
│ kube-rbac-proxy W0320 08:29:15.062093       1 options.go:164]                                                                                                                                                                                                                                                                                                      │
│ kube-rbac-proxy ==== Deprecation Warning ======================                                                                                                                                                                                                                                                                                                    │
│ kube-rbac-proxy                                                                                                                                                                                                                                                                                                                                                    │
│ kube-rbac-proxy Insecure listen address will be removed.                                                                                                                                                                                                                                                                                                           │
│ kube-rbac-proxy Using --insecure-listen-address won't be possible!                                                                                                                                                                                                                                                                                                 │
│ kube-rbac-proxy                                                                                                                                                                                                                                                                                                                                                    │
│ kube-rbac-proxy The ability to run kube-rbac-proxy without TLS certificates will be removed.                                                                                                                                                                                                                                                                       │
│ kube-rbac-proxy Not using --tls-cert-file and --tls-private-key-file won't be possible!                                                                                                                                                                                                                                                                            │
│ kube-rbac-proxy                                                                                                                                                                                                                                                                                                                                                    │
│ kube-rbac-proxy For more information, please go to https://github.com/brancz/kube-rbac-proxy/issues/187                                                                                                                                                                                                                                                            │
│ kube-rbac-proxy                                                                                                                                                                                                                                                                                                                                                    │
│ keycloak-controller {"level":"info","ts":"2024-03-20T08:29:14.926Z","logger":"controller-runtime.metrics","msg":"Metrics server is starting to listen","addr":"127.0.0.1:9556"}                                                                                                                                                                                    │
│ kube-rbac-proxy ===============================================                                                                                                                                                                                                                                                                                                    │
│ kube-rbac-proxy                                                                                                                                                                                                                                                                                                                                                    │
│ kube-rbac-proxy                                                                                                                                                                                                                                                                                                                                                    │
│ keycloak-controller {"level":"info","ts":"2024-03-20T08:29:14.960Z","logger":"setup","msg":"starting manager"}                                                                                                                                                                                                                                                     │
│ keycloak-controller {"level":"info","ts":"2024-03-20T08:29:14.961Z","msg":"Starting server","kind":"health probe","addr":":9557"}                                                                                                                                                                                                                                  │
│ keycloak-controller {"level":"info","ts":"2024-03-20T08:29:15.064Z","msg":"starting server","path":"/metrics","kind":"metrics","addr":"127.0.0.1:9556"}                                                                                                                                                                                                            │
│ kube-rbac-proxy W0320 08:29:15.062146       1 options.go:215]                                                                                                                                                                                                                                                                                                      │
│ kube-rbac-proxy ==== Removed Flag Warning ======================                                                                                                                                                                                                                                                                                                   │
│ kube-rbac-proxy                                                                                                                                                                                                                                                                                                                                                    │
│ kube-rbac-proxy logtostderr is removed in the k8s upstream and has no effect any more.                                                                                                                                                                                                                                                                             │
│ keycloak-controller {"level":"info","ts":"2024-03-20T08:29:15.065Z","msg":"Starting EventSource","controller":"keycloakrealm","controllerGroup":"keycloak.infra.doodle.com","controllerKind":"KeycloakRealm","source":"kind source: *v1beta1.KeycloakRealm"}                                                                                                       │
│ keycloak-controller {"level":"info","ts":"2024-03-20T08:29:15.065Z","msg":"Starting EventSource","controller":"keycloakrealm","controllerGroup":"keycloak.infra.doodle.com","controllerKind":"KeycloakRealm","source":"kind source: *v1.Secret"}                                                                                                                   │
│ keycloak-controller {"level":"info","ts":"2024-03-20T08:29:15.065Z","msg":"Starting EventSource","controller":"keycloakrealm","controllerGroup":"keycloak.infra.doodle.com","controllerKind":"KeycloakRealm","source":"kind source: *v1beta1.KeycloakClient"}                                                                                                      │
│ keycloak-controller {"level":"info","ts":"2024-03-20T08:29:15.065Z","msg":"Starting EventSource","controller":"keycloakrealm","controllerGroup":"keycloak.infra.doodle.com","controllerKind":"KeycloakRealm","source":"kind source: *v1beta1.KeycloakUser"}                                                                                                        │
│ kube-rbac-proxy                                                                                                                                                                                                                                                                                                                                                    │
│ kube-rbac-proxy ===============================================                                                                                                                                                                                                                                                                                                    │
│ keycloak-controller {"level":"info","ts":"2024-03-20T08:29:15.065Z","msg":"Starting EventSource","controller":"keycloakrealm","controllerGroup":"keycloak.infra.doodle.com","controllerKind":"KeycloakRealm","source":"kind source: *v1.Pod"}                                                                                                                      │
│ keycloak-controller {"level":"info","ts":"2024-03-20T08:29:15.065Z","msg":"Starting Controller","controller":"keycloakrealm","controllerGroup":"keycloak.infra.doodle.com","controllerKind":"KeycloakRealm"}                                                                                                                                                       │
│ keycloak-controller {"level":"info","ts":"2024-03-20T08:29:15.371Z","msg":"Starting workers","controller":"keycloakrealm","controllerGroup":"keycloak.infra.doodle.com","controllerKind":"KeycloakRealm","worker count":4}                                                                                                                                         │
│ keycloak-controller {"level":"info","ts":"2024-03-20T08:29:16.513Z","logger":"controllers.KeycloakRealm","msg":"reconciling KeycloakRealm","namespace":"keycloak","name":{"name":"master","namespace":"keycloak"}}                                                                                                                                                 │
│ keycloak-controller {"level":"info","ts":"2024-03-20T08:29:16.514Z","logger":"controllers.KeycloakRealm","msg":"reconciler","template":null}                                                                                                                                                                                                                       │
│ keycloak-controller {"level":"info","ts":"2024-03-20T08:29:16.514Z","logger":"controllers.KeycloakRealm","msg":"create new reconciler pod","pod":"keycloakrealm-master-xsmtz","previous":""}                                                                                                                                                                       │
│ keycloak-controller {"level":"info","ts":"2024-03-20T08:29:16.536Z","logger":"controllers.KeycloakRealm","msg":"creating new realm secret","secret":"keycloakrealm-master-xsmtz"}                                                                                                                                                                                  │
│ keycloak-controller {"level":"info","ts":"2024-03-20T08:29:16.568Z","logger":"controllers.KeycloakRealm","msg":"reconciling KeycloakRealm","namespace":"keycloak","name":{"name":"master","namespace":"keycloak"}}                                                                                                                                                 │
│ keycloak-controller {"level":"info","ts":"2024-03-20T08:29:16.590Z","logger":"controllers.KeycloakRealm","msg":"reconciling KeycloakRealm","namespace":"keycloak","name":{"name":"master","namespace":"keycloak"}}                                                                                                                                                 │
│ keycloak-controller {"level":"info","ts":"2024-03-20T08:29:16.605Z","logger":"controllers.KeycloakRealm","msg":"reconciling KeycloakRealm","namespace":"keycloak","name":{"name":"master","namespace":"keycloak"}}                                                                                                                                                 │
│ keycloak-controller {"level":"info","ts":"2024-03-20T08:29:16.621Z","logger":"controllers.KeycloakRealm","msg":"reconciling KeycloakRealm","namespace":"keycloak","name":{"name":"master","namespace":"keycloak"}}                                                                                                                                                 │
│ keycloak-controller {"level":"info","ts":"2024-03-20T08:29:18.066Z","logger":"controllers.KeycloakRealm","msg":"reconciling KeycloakRealm","namespace":"keycloak","name":{"name":"master","namespace":"keycloak"}}                                                                                                                                                 │
│ keycloak-controller {"level":"info","ts":"2024-03-20T08:29:36.225Z","logger":"controllers.KeycloakRealm","msg":"reconciling KeycloakRealm","namespace":"keycloak","name":{"name":"master","namespace":"keycloak"}}                                                                                                                                                 │
│ kube-rbac-proxy                                                                                                                                                                                                                                                                                                                                                    │
│ kube-rbac-proxy I0320 08:29:15.062427       1 kube-rbac-proxy.go:225] Valid token audiences:                                                                                                                                                                                                                                                                       │
│ kube-rbac-proxy I0320 08:29:15.062462       1 kube-rbac-proxy.go:319] Generating self signed cert as no cert is provided                                                                                                                                                                                                                                           │
│ kube-rbac-proxy I0320 08:29:16.284405       1 kube-rbac-proxy.go:383] Starting TCP socket on 0.0.0.0:8443                                                                                                                                                                                                                                                          │
│ kube-rbac-proxy I0320 08:29:16.284726       1 kube-rbac-proxy.go:390] Listening securely on 0.0.0.0:8443                                                                                                                                                                                                                                                           │
│ keycloak-controller {"level":"info","ts":"2024-03-20T08:29:36.225Z","logger":"controllers.KeycloakRealm","msg":"reconciler pod succeeded"}                                                                                                                                                                                                                         │
│ keycloak-controller {"level":"info","ts":"2024-03-20T08:29:36.259Z","logger":"controllers.KeycloakRealm","msg":"reconciling KeycloakRealm","namespace":"keycloak","name":{"name":"master","namespace":"keycloak"}}                                                                                                                                                 │
│ keycloak-controller {"level":"info","ts":"2024-03-20T08:29:37.240Z","logger":"controllers.KeycloakRealm","msg":"reconciling KeycloakRealm","namespace":"keycloak","name":{"name":"master","namespace":"keycloak"}}                                                                                                                                                 │
│ keycloak-controller {"level":"info","ts":"2024-03-20T08:29:37.703Z","logger":"controllers.KeycloakRealm","msg":"reconciling KeycloakRealm","namespace":"keycloak","name":{"name":"master","namespace":"keycloak"}}                                                                                                                                                 │
│ keycloak-controller {"level":"info","ts":"2024-03-20T08:29:38.241Z","logger":"controllers.KeycloakRealm","msg":"reconciling KeycloakRealm","namespace":"keycloak","name":{"name":"master","namespace":"keycloak"}}                                                                                                                                                 │
│ keycloak-controller {"level":"info","ts":"2024-03-20T08:29:38.269Z","logger":"controllers.KeycloakRealm","msg":"reconciling KeycloakRealm","namespace":"keycloak","name":{"name":"master","namespace":"keycloak"}}                                                                                                                                                 │
│ keycloak-controller {"level":"info","ts":"2024-03-20T08:29:38.284Z","logger":"controllers.KeycloakRealm","msg":"reconciling KeycloakRealm","namespace":"keycloak","name":{"name":"master","namespace":"keycloak"}}                                                                                                                                                 │
│ keycloak-controller {"level":"info","ts":"2024-03-20T08:30:18.081Z","logger":"controllers.KeycloakRealm","msg":"reconciling KeycloakRealm","namespace":"keycloak","name":{"name":"master","namespace":"keycloak"}}          

KeycloakRealm CR description:

│ Name:         master                                                                                                                                                                                                                                                                                                                                         │
│ Namespace:    keycloak                                                                                                                                                                                                                                                                                                                                       │
│ Labels:       <none>                                                                                                                                                                                                                                                                                                                                         │
│ Annotations:  <none>                                                                                                                                                                                                                                                                                                                                         │
│ API Version:  keycloak.infra.doodle.com/v1beta1                                                                                                                                                                                                                                                                                                              │
│ Kind:         KeycloakRealm                                                                                                                                                                                                                                                                                                                                  │
│ Metadata:                                                                                                                                                                                                                                                                                                                                                    │
│   Creation Timestamp:  2024-03-20T08:29:16Z                                                                                                                                                                                                                                                                                                                  │
│   Generation:          1                                                                                                                                                                                                                                                                                                                                     │
│   Resource Version:    15475393                                                                                                                                                                                                                                                                                                                              │
│   UID:                 1d943bd8-676e-4520-94ea-c5526e66ff86                                                                                                                                                                                                                                                                                                  │
│ Spec:                                                                                                                                                                                                                                                                                                                                                        │
│   Address:  http://keycloak-service.keycloak:8080/auth                                                                                                                                                                                                                                                                                                       │
│   Auth Secret: ... # hidden on purpose                                                                                                                                                                                                                                                                                                                                   │
│   Interval:          1m                                                                                                                                                                                                                                                                                                                                      │
│   Realm:                                                                                                                                                                                                                                                                                                                                                     │
│     Account Theme:  keycloak                                                                                                                                                                                                                                                                                                                                 │
│     Attributes:                                                                                                                                                                                                                                                                                                                                              │
│       Ciba Auth Requested User Hint:         login_hint                                                                                                                                                                                                                                                                                                      │
│       Ciba Backchannel Token Delivery Mode:  poll                                                                                                                                                                                                                                                                                                            │
│       Ciba Expires In:                       120                                                                                                                                                                                                                                                                                                             │
│       Ciba Interval:                         5                                                                                                                                                                                                                                                                                                               │
│       Client Offline Session Idle Timeout:   0                                                                                                                                                                                                                                                                                                               │
│       Client Offline Session Max Lifespan:   0                                                                                                                                                                                                                                                                                                               │
│       Client Session Idle Timeout:           0                                                                                                                                                                                                                                                                                                               │
│       Client Session Max Lifespan:           0                                                                                                                                                                                                                                                                                                               │
│       Frontend URL:                                                                                                                                                                                                                                                                                                                                          │
│       Par Request Uri Lifespan:              60                                                                                                                                                                                                                                                                                                              │
│       Realm Reusable Otp Code:               false                                                                                                                                                                                                                                                                                                           │
│     Display Name:                            Keycloak                                                                                                                                                                                                                                                                                                        │
│     Display Name Html:                       <div class="kc-logo-text"><span>Keycloak</span></div>                                                                                                                                                                                                                                                           │
│     Groups: ... # hidden on purpose                                                                                                                                                                                                                                                                                                                                │
│     Realm:   master                                                                                                                                                                                                                                                                                                                                          │
│   Reconciler Template:                                                                                                                                                                                                                                                                                                                                       │
│     Spec:                                                                                                                                                                                                                                                                                                                                                    │
│       Containers:                                                                                                                                                                                                                                                                                                                                            │
│         Env:                                                                                                                                                                                                                                                                                                                                                 │
│           Name:   LOGGING_LEVEL_ROOT                                                                                                                                                                                                                                                                                                                         │
│           Value:  debug                                                                                                                                                                                                                                                                                                                                      │
│         Name:     keycloak-config-cli                                                                                                                                                                                                                                                                                                                        │
│   Resource Selector:                                                                                                                                                                                                                                                                                                                                         │
│     Match Labels:                                                                                                                                                                                                                                                                                                                                            │
│       Realm:  master                                                                                                                                                                                                                                                                                                                                         │
│   Version:    24.0.1                                                                                                                                                                                                                                                                                                                                         │
│ Status:                                                                                                                                                                                                                                                                                                                                                      │
│   Conditions:                                                                                                                                                                                                                                                                                                                                                │
│     Last Transition Time:  2024-03-20T08:29:36Z                                                                                                                                                                                                                                                                                                              │
│     Message:                                                                                                                                                                                                                                                                                                                                                 │
│     Observed Generation:   1                                                                                                                                                                                                                                                                                                                                 │
│     Reason:                ReconciliationSucceeded                                                                                                                                                                                                                                                                                                           │
│     Status:                True                                                                                                                                                                                                                                                                                                                              │
│     Type:                  Ready                                                                                                                                                                                                                                                                                                                             │
│   Observed Generation:     1                                                                                                                                                                                                                                                                                                                                 │
│   observedSHA256:          5f5b8c46384518f79b71f62f42c630a1e014713f7d3fd0f53d1a8c6b6622d18e                                                                                                                                                                                                                                                                  │
│ Events:                                                                                                                                                                                                                                                                                                                                                      │
│   Type    Reason  Age    From           Message                                                                                                                                                                                                                                                                                                              │
│   ----    ------  ----   ----           -------                                                                                                                                                                                                                                                                                                              │
│   Normal  info    2m38s  KeycloakRealm  reconcile realm progressing                                                                                                                                                                                                                                                                                          │
│   Normal  info    2m18s  KeycloakRealm  Realm successfully reconciled

[raffis]

You may try with v2.3.0 which might fix your issue.

[y3lousso]

It partially fixed the issue, now I can see the CRD KeycloakRealm doing it's reconciliation.

But I am still facing the following issue:

• Set a realm displayNameHtml to "abc"
• Apply the KeycloakRealm CRD, if you check the displayNameHtml in keycloak UI, it's set to "abc"
• Modify the displayNameHtml manually via keycloak UI to "def"
• Wait until CRD reconciliation
• Check displayNameHtml in keycloak UI

Expected value: "abc"
Actual value: "def"

Issue: The CRD is not overriding manual changes as I would expect

Current workaround: reapply the CRD once in a while to ensure no drift as happened

[raffis]

Readding the resource is really not the intention of this controller. I assume by reapplying the CRD you mean the cr and not the schema.

However the underlying "problem" is probably the conflg client which caches the realm spec in keycloak itself https://github.com/adorsys/keycloak-config-cli?tab=readme-ov-file#import-options.

We actually have this disabled, you can specify a custom reconciler template, see https://github.com/DoodleScheduling/keycloak-controller#reconciler-template
and set the env:

     - name: IMPORT_CACHE_ENABLED
       value: "false"

But taken from this this should really be documented here and I will think of making this the default behaviour aka overriding the default of the keycloak-config-cli.