Kubernetes Validations / CEL

[baloo42]

We have recently added support for Kubernetes Validations to the CRDGenerator:
https://github.com/fabric8io/kubernetes-client/blob/main/doc/CRD-generator.md#iofabric8generatorannotationvalidationrule

Now we want to discuss how we can add further support for Kubernetes Validation / CEL rules.

Some java-implementations for CEL already exists:

• https://github.com/projectnessie/cel-java
• https://github.com/google/cel-java

I'll start with some Ideas:

• The CRDGenerator could verify during CRD generation if a given ValidationRule is syntactically correct.
• The CRDGenerator could verify during CRD generation if a given ValidationRule exceeds cost limits.
• A CLI tool could verify if a YAML file is valid. kubernetes-client could provide the base for such a CLI tool.
  See also https://github.com/istio/istio/blob/master/pkg/config/crd/validator.go

What ideas do you have?

[andreaTP]

Thanks for starting this conversation @baloo42 !
I haven't dug enough into CEL and Validation Rules enough to understand one detail:
should validation be performed on the CRD(this looks more like "static analysis")? or do we need to have concrete values for the fields and it applies on the CR(i.e. runtime checks)?
or both? 🙂

Everything you are saying about the CRDGenerator applies to the java-generator too, in this case, validation should be implemented independently on the models so that it will benefit all of the remaining toolchains.

[baloo42]

Let's start from the definition / behaviour of Kubernetes Validation Rules:

Kubernetes Validation Rules written in CEL are useful to add additional validations directly to a CRD where OpenAPIv3 value validations / JSON Schema validations (maxLength, maxItems, maxProperties, required, enum, minimum, maximum, format, ...) are not enough.

The expressions of Kubernetes Validation Rules are compiled and type checked against a CRD's schema "ahead-of-time" (when CRDs are created and updated) allowing them to be evaluated efficiently and safely at "runtime" (when custom resources are validated aka when custom resources are created or updated). Both workloads, the "ahead-of-time" compilation and the "runtime" validation, are implemented and executed by the Kubernetes API Server itself. This allows a controller to rely on Kubernetes so that the controller doesn't have to implement validation on its own.

To prevent CEL evaluation from consuming excessive compute resources, validation rules impose some limits. Those limits are based on CEL cost units, a platform and machine independent measure of execution cost. As a result, the limits are the same regardless of where they are enforced.

The CRD-Generator and Java-Generator are tools for developers to build controllers/operators. Those tools help developers to correlate the source code to the CRD or vice-versa the CRD to the code. Let's call this phase "development time".

As such both tools should support the developer as much they can, but they should not restrict the developer because Kubernetes itself already implements the heavy lifting regarding CR validation.

should validation be performed on the CRD(this looks more like "static analysis")?

In my opinion, some kind of "static anylisis" would be useful: It allows the developer to fail fast during development time.
We do this already e.g. by detecting multiple "stored" versions and it makes sense to me to enhance this behaviour by checking the syntax and the cost limits.

Everything you are saying about the CRDGenerator applies to the java-generator too, in this case, validation should be implemented independently on the models so that it will benefit all of the remaining toolchains.

Yes, you are right. This makes sense to me, too.

or do we need to have concrete values for the fields and it applies on the CR(i.e. runtime checks)?

It doesn't make sense to me to build some kind of evaluation of CEL rules into the Java-Generator/CRD-Generator because both tools are not relevant during runtime.

But there might be other use cases where kubernetes-client (or a subproject) could help.
Let's imagine someone wants to build a tool like kubeconform, kubeval or kubectl-validate which allows validation of Custom Resources outside of the target Kubernetes cluster e.g. in a pipeline for a GitOps project.
Now imagine he wants to use Java and the kubernetes-client project.
In this case the kubernetes-client project (or a subproject) could provide the base integration for evaluating Kubernetes CEL rules, because Kubernetes uses a specific set of functionalities:

• https://kubernetes.io/blog/2022/09/23/crd-validation-rules-beta/#using-the-functions-libraries
• https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#validation-rules

But I think this requires a lot efforts and that's why I suggest to start with some kind of static analysis first (and only as opt-in / by printing warnings):

1. The CRDGenerator/JavaGenerator could verify during generation (during the development phase) if a given Kubernetes Validation Rule is syntactically correct.
2. The CRDGenerator/JavaGenerator could verify during generation (during the development phase) if a given Kubernetes Validation Rule exceeds cost limits.

[metacosm]

The Quarkus extension for the Java Operator SDK could compile the CEL schema at build time and provide dev mode commands to validate custom resources.
If we wanted to go crazy, we could imagine validation being built into the generated builders but this might indeed prove to be an effort that's not worth the return if not widely used.