Tekton PipelineRun Works Directly But Not Using the Kubernetes-Client

[kjq]

We have two pipelines, one using a PVC and another not. Using the latest Tekton operator (v.0.67.0), both work in minikube or when applied maually/directly to the EKS cluster using the PipelineRun.

When we go to trigger the pipeline using a PVC using the Quarkus Kubernetes-Tekton Client we get this error when a PVC is involved:

Failure executing: POST at: [https://172.20.0.1:443/apis/tekton.dev/v1beta1/namespaces/pipeline/pipelineruns](https://172.20.0.1/apis/tekton.dev/v1beta1/namespaces/pipeline/pipelineruns). Message: admission webhook \"webhook.pipeline.tekton.dev\" denied the request: mutation failed: cannot decode incoming new object: json: unknown field \"claims\". Received status: Status(apiVersion=v1, code=400, details=null, kind=Status, message=admission webhook \"webhook.pipeline.tekton.dev\" denied the request: mutation failed: cannot decode incoming new object: json: unknown field \"claims\", metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=BadRequest, status=Failure, additionalProperties={}).

Is there anything we are missing? We are not running this in OpenShift.

We saw this ticket which seems similar: openshift-pipelines/pipelines-as-code#750

[shawkins]

It seems like a claims field would only be coming from a nested ResourceRequirements under the PVC. Is is possile that the admission webhook doesn't know the full definition of what's possible via ResourceRequirements? Have you tried using tekton v1 resources?

[kjq]

No, we are using the v1Beta both from the Kubernetes client/Tekton and Tekton itself (when applied directly). I would guess the v1Beta in the client matches up to what Tekton provides. Again, applying this outside of the Kubernetes client using the same PipelineRun works with v1Beta1. We are using the latest operator v0.67 and are about to downgrade to v.0.66.

[shawkins]

We are using the latest operator v0.67 and are about to downgrade to v.0.66.

Are you referring to fabric8 client versions?

The fabric8 client added support for tekton v0.47.0 in 6.7: 1ae7d9a#diff-58498cae415a055b99fd4fcafc6b4f8fd98b9efc19c37e12e9e4bbf0ad30fe6a

I don't see any changes for PipelineRunSpec in that commit. However PipelineTaskRunSpec is new and contains ResourceRequirements. There is also tektoncd/pipeline@c82f201 which seems like they needed to address some compatibility issues related to resources.

Is the webhook based upon tekton v0.47.0 or later?

And are you able to obtain a trace with the payload from the kubernetes client, just to verify the resource you are trying to create and compare to what is actually created in minikube.

[kjq]

We are using the Quarkus extension for the Tekton client which looked to be using older versions of the fabric8 tekton models:

[INFO] +- io.quarkiverse.tektonclient:quarkus-tekton-client:jar:1.0.1:compile
[INFO] |  \- io.fabric8:tekton-client:jar:6.6.2:compile
[INFO] |     +- io.fabric8:tekton-model-v1alpha1:jar:6.6.2:compile
[INFO] |     |  \- io.fabric8:knative-model:jar:6.6.2:compile
[INFO] |     +- io.fabric8:tekton-model-v1beta1:jar:6.6.2:compile
[INFO] |     +- io.fabric8:tekton-model-triggers-v1alpha1:jar:6.6.2:compile
[INFO] |     \- io.fabric8:tekton-model-triggers-v1beta1:jar:6.6.2:compile

We are checking to see if we can override those dependencies to the later version as such now in the hope it syncs up:

[INFO] +- io.quarkiverse.tektonclient:quarkus-tekton-client:jar:1.0.1:compile
[INFO] +- io.fabric8:tekton-client:jar:6.7.2:compile
[INFO] |  \- io.fabric8:tekton-model-v1:jar:6.7.2:compile
[INFO] +- io.fabric8:tekton-model-v1alpha1:jar:6.7.2:compile
[INFO] |  +- io.fabric8:kubernetes-model-core:jar:6.6.2:compile
[INFO] |  |  +- io.fabric8:kubernetes-model-common:jar:6.6.2:compile
[INFO] |  |  \- com.fasterxml.jackson.core:jackson-annotations:jar:2.15.2:compile
[INFO] |  \- io.fabric8:knative-model:jar:6.6.2:compile
[INFO] +- io.fabric8:tekton-model-v1beta1:jar:6.7.2:compile
[INFO] +- io.fabric8:tekton-model-triggers-v1alpha1:jar:6.7.2:compile
[INFO] +- io.fabric8:tekton-model-triggers-v1beta1:jar:6.7.2:compile

We are not sure though if the PVC spec being referred to is a part of the k8s client models or if the Tekton models can be overriden only. We dont even know if this will work its just a try ;)

[kjq]

We would prefer not to have to downgrade especially since it works outside of the Quarkus Tekton extension/client, and we dont even know if that will solve the PVC issue, but we will try that next and if we still have the issue we believe a fallback, for now, would be to combine the pipeline steps into a single task so that we can drop the PVC. Again, just spitballing at the moment.

[kjq]

For reference, we think we worked it out:

https://groups.google.com/g/quarkus-dev/c/VCM0zxVYkJI

[shawkins]

What looks a little odd is that the serialization of claims is supposed to omit empty https://github.com/fabric8io/kubernetes-client/blob/master/kubernetes-model-generator/kubernetes-model-core/src/generated/java/io/fabric8/kubernetes/api/model/ResourceRequirements.java#L44

Are you saying that the annotation is not being followed in native mode?

[kjq]

That is our "initial" thought. The same service works perfectly fine as a JVM instance, it's when we deploy the native instance that only the PVC does not work. We threw in that quick hack to null out the value and it works. We have seen similar issues before with Jackson serialization but could control it with @RegisterForReflection.

Also, it is important to note, the tekton pipelines work applied directly (i.e not using the Kubernetes + Tekton client) or run as JVM instances pointing to the cluster/tekton.

The only outlier at this point is the native piece. Whether or not that is the real cause, the "nulling" of the value fixes it - we never even made it into Tekton before, now the pipelines are running and using the PVC.

Just to be clear, we did not patch anything in either the Kubernetes Client, Tekton Extension, or Quakus Tekton Extension itself. We hacked out code to forcibly null the value for the claim prior to sending it to Tekton using the Quarkus Tekton Extension. Without the hack the same exact code works as-is using the quarkus extension.

[shawkins]

cc @manusa @metacosm @geoand