Dependencies and Hooks

[anders-swanson]

Hi, this is a general question on a user story for carvel kapp. Is there a way to define dependencies or hooks on an app? E.g.,

• Run custom logic before the kapp-controller processing the YAML and deploys the app
• Run custom logic after the kapp-controller marks the app as healthy
• Define a list of selectors (i.e., matchLabels) for apps. All apps matching the selectors must be healthy before kapp-controller deploys the app.

I did not see any similar definition while inspecting the go types for app.

[cppforlife]

(issued transfered to kapp-controller repo since it seems like question was kapp-controller centric.)

@anders-swanson would you mind providing some concrete examples of what kind of actions you may want to take at those times. i may be able to provide recommended solutions for some and for the rest it could develop into a feature request.

[anders-swanson]

Hi @cppforlife, thanks for triaging this issue.

Hooks

In many use cases your app may have some pre/post logic that needs to be reconciled, not present in helm chart. I will give some specific examples of apps that may require this. Because hooks are intrinsically user and environment specific, it would be best if the user has full control of how to define them.

Pre-Processing Hook

An App may have some specific logic that should execute before the App is reconciled. For example, bootstrapping initial app data for default users, such as populating a secret with a default username and randomly generated password

Post-Healthy Hook

Imagine an App which represents a Keycloak server deployment. After Keycloak installs, to make it usable I need to configure Users/Realms/etc. within Keycloak. In a standard installation, the human operator may do this manually via the Keycloak UI, or automate it with some external script. Because the Keycloak configuration is a part of the App (it is required to get Keycloak running in some environment), it would be ideal if the App could call out to some extensible/modular addon to do all this configuration as part of the reconcile.

Dependencies

In some situations you may want to backoff the deployment of an App until the health status of dependent Apps is met. For example, if my App depends on some CRDs deployed by another App, it will fail to deploy if the CRD-deploying App is not present.
Another example of Dependencies: My App is integrated with an ingress-controller + external DNS. I need these to be present before deploying the App.

[anders-swanson]

I guess, I am wanting to orchestrate the deployment of many apps simultaneously as a cohesive and functional unit. There is some wiring from Hooks and safety checks with Dependencies that are desirable in such a scenario.

[cppforlife]

thanks for providing more detail here. really useful. ill try to pick at concrete examples with suggestions:

bootstrapping initial app data for default users

general pattern we have recommended is to use kapp ordering rules (https://carvel.dev/kapp/docs/v0.49.0/apply-ordering/). as long as you can identify before or after particular resource change(s) you want to do something, you can order creation/update of a k8s Job that can perform arbitrary action. example on the docs page, shows how to run a db migration Job before starting off your Deployment. keep in mind that Jobs should be written idempotently since they may be executed multiple times.

such as populating a secret with a default username and randomly generated password

i pulled this one out in its own item since i typically recommend use of tools like https://github.com/vmware-tanzu/carvel-secretgen-controller to generate secrets. (other tools exist as well that are able to pull down secrets into environment etc).

Because the Keycloak configuration is a part of the App (it is required to get Keycloak running in some environment), it would be ideal if the App could call out to some extensible/modular addon to do all this configuration as part of the reconcile.

this one i think will also fall into a bucket of -- add a Job that gets applied after successful reconciliation of Deployment resource for example. Job can communicate with necessary APIs etc to set everything up.

if my App depends on some CRDs deployed by another App, it will fail to deploy if the CRD-deploying App is not present.
My App is integrated with an ingress-controller + external DNS. I need these to be present before deploying the App.

what've recommended (and used ourselves) is use of ordering rules as well. a common pattern might be to wrap collection of Apps/PackageInstalls into higher level App and within that higher level App one could apply order rules between those resources.

we also have another feature that allows to do this in a different way -- exists annotation in kapp (https://carvel.dev/blog/migrate-existing-resources-to-a-new-kapp-app/). though right now it's limited to being used with non-kapp deployed resources (we are planning to address this in the next verison of kapp).

i could see adding App CR API feature that allows you to "order" the resources though we would want to collect a bit more info where above solutions dont fit well.

[anders-swanson]

Thanks, great suggestions. I am getting an idea of how this work would for an App collection that represents a platform.

Re: Pre Hook

I am hoping any pre-hook logic is simple enough that it could be coerced into a helm chart or applied via some other controller like you mention for secrets generation. In the case that it does not fit into this model, it's unclear how to proceed.

Re: Post Hook Job

A job would work (and probably the only way to currently implement this using kapp-controller), but I don't know if it is a best-fit for post-reconcile configuration. I have concerns about the resiliency of doing configuration in a job, since we would need to ensure that the job needs to run every time the App reconciles, in case of data loss or corruption (this is a bit of a stretch, but for the sake of example imagine the KeyCloak MySQL DB gets corrupted - we would want the next reconcile to regenerate the Keycloak bootstrap state).
Perhaps in this case a custom controller developed by the platform owner (external to kapp controller and even carvel) is needed that watches the App resource, and applies the user-defined custom post-reconcile logic.

Re: Dependencies

I am trying to avoid tightly coupling Apps into higher-order components to keep them composable and easy to independently manage their lifecycle (install/upgrade/etc.). Wrapping Apps with App does allow for ordering but requires them to ship together - consider that one user may want to install a subset of available Apps.

In short, it seems like layering custom controllers on kapp-controller or using jobs judiciously may be the way to go.

[cppforlife]

In the case that it does not fit into this model, it's unclear how to proceed.

are there cases of in pre-hook category that you feel cannot be addressed by a Job + order rules?

A job would work (and probably the only way to currently implement this using kapp-controller), but I don't know if it is a best-fit for post-reconcile configuration.
...since we would need to ensure that the job needs to run every time the App reconciles...

if you absolutely want this Job to run every time reconciliation happens (vs only when Job definition changes), then you can use kapp.k14s.io/nonce annotation on a Job. it will force Job to "change" every single deploy. (this note is also relevant: https://carvel.dev/kapp/docs/v0.49.0/faq/#jobbatch-is-invalid--specselector-required-value-error)

Perhaps in this case a custom controller developed by the platform owner (external to kapp controller and even carvel) is needed that watches the App resource, and applies the user-defined custom post-reconcile logic.

you'll have to find a balance between assembling lower level primitives quickly (e.g. add a Job) vs deciding when to invest a bit in building specialized features (e.g. creating a Pod that watches some resources, creating a dedicated CR). a good example might be, if you want your Keycloak installation to have some special logic based on the state of Keycloak i would recommend having a tiny controller (does not have to expose a CR), but rather just sits alongside Keycloak and observes it and potentially makes changes to Keycloak to keep it running. such system is not coupled to Carvel or App CRs (should probably watch Keycloak API or Keycloak deployment) and provides independent value for Keycloak users.

[voor]

One of the mechanisms that I've been playing around with lately is the kapp.k14s.io/exists, it effectively tells a package something else is going to create this CRD or object, and you should hold your horses until it's there. I realize it was primarily created to handle use cases like Gatekeeper or Dex where the operator makes its own CRD, but it has been helpful to handle situations where one App CR might contain a combination of loose custom resources + a Package Install that creates those custom resources.

If there was some kind of exposure mechanism, where a Package could optionally expose some kind of exists-annotationed things, in particular its CRDs, maybe another Package could consume those. Not entirely sure how you would surface that, though. Maybe in the status of an App CR.

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: certificates.cert-manager.io
  annotations:
    kapp.k14s.io/exposed-exists: ""
spec:
  group: cert-manager.io
  names:
    categories:
    - cert-manager
    kind: Certificate
    listKind: CertificateList
    plural: certificates
    shortNames:
    - cert
    - certs
    singular: certificate
  scope: Namespaced
---
apiVersion: kappctrl.k14s.io/v1alpha1
kind: App
metadata:
  name: cert-manager
...
status:
  exists:
    items:
    - apiVersion: apiextensions.k8s.io/v1
      kind: CustomResourceDefinition
      metadata:
        name: certificates.cert-manager.io
      spec:
        group: cert-manager.io
---
apiVersion: packaging.carvel.dev/v1alpha1
kind: PackageInstall
metadata:
  name: external-dns
  annotations:
    kapp.k14s.io/change-rule: "app cert-manager exists before upserting"

[neil-hickey]

Leaving this open for discussion sake, nothing has been added for a while. So we can close it out soon if this doesn't need to be open anymore

[voor]

By the way 😉 kapp-controller (and so I'm assuming this was surfaced by kapp) does have something very similar to what I asked for in my comment:

status:
  conditions:
  - status: "True"
    type: ReconcileSucceeded
  consecutiveReconcileSuccesses: 209
  deploy:
    exitCode: 0
    finished: true
    kapp:
      associatedResources:
        groupKinds:
        - group: apiextensions.k8s.io
          kind: CustomResourceDefinition

What's missing there is just a tiny bit more information about those CRDs that could take on the form of something kapp-controller could morph into an kapp.k14s.io/exists

[artem-nefedov]

I'm also interested in seeing this implemented.
I would very much like to replace helm with kapp for better developer experience.
(helm is currently used in GitOps way with flux)

However, there are certain features that would be required:

• ability to run pre-delete/post-delete hooks (jobs) for custom cleanup logic (absolutely crucial)
• ability to run different post-install hook based on whether it's a clean install or an upgrade (not as important, can be worked around if not available)

Both of these things are possible in helm, and both are currently not possible in kapp without doing something stupid like installing an App from a helm chart just to keep hooks there.

[praveenrewar]

Hi @artem-nefedov !
Regarding pre-delete/post-delete hooks, wanted to confirm if you have gone through Dmitriy's comments from above regarding using jobs and change-group and change-rules to order stuff. Would that be helpful for your use case?

[artem-nefedov]

@praveenrewar I'm not sure I fully understand, but from what I could gather based on documentation, you're suggesting I do something like this (correct me if I'm wrong):

---
kind: Job
metadata:
  name: pre-delete-hook
  annotations:
    kapp.k14s.io/change-rule: "upsert before deleting apps.big.co/deployment"
---
kind: Job
metadata:
  name: post-delete-hook
  annotations:
    kapp.k14s.io/change-rule: "upsert after deleting apps.big.co/deployment"

Naturally, besides Jobs there could be other resources such as RBAC, etc.

If I did get it properly, I have a couple of questions:

• Does this guarantee that the jobs will only be created during deletion? (the word "change" is confusing here)
• How/when those jobs themselves will be deleted?