Error: unexpected new value in metadata.annotations: default computed_fields should cover it

[thomas-riccardi]

Terraform Version, Provider Version and Kubernetes Version

Terraform version: v1.1.4
Kubernetes provider version: v2.7.1
Kubernetes version: 1.21.5-gke.1302

Affected Resource(s)

• kubernetes_resource

Terraform Configuration Files

resource "kubernetes_manifest" "foo" {
  manifest = {
    apiVersion = "apps/v1"
    kind       = "DaemonSet"
    ...
  }
}

Steps to Reproduce

1. terraform apply
2. edit the spec of the manifest
3. terraform apply

Expected Behavior

The terraform apply works

Actual Behavior

The terraform apply fails with the following error:

Error: Provider produced inconsistent result after apply

When applying changes to kubernetes_manifest.foo, provider "provider[\"registry.terraform.io/hashicorp/kubernetes\"]" produced an unexpected new value: .object.metadata.annotations["deprecated.daemonset.template.generation"]: was cty.StringVal("1"), but now cty.StringVal("2"). This is a bug in the provider, which should be reported in the provider's own issue tracker.

Important Factoids

• I read about computed_fields. If I understand correctly, the default value includes the metadata.annotations, so it should cover my scenario, shouldn't it?
• A subsequent apply does work
• An explicit computed_fields = ["metadata.annotations","metadata.labels"] doesn't change anything

References

• deployment.kubernetes.io/revision annotation causes problems when successively applying kubernetes_manifest #1370 is the closest issue I could find. maybe it was before the computed_fields though? (and thus fixed by it?)

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[ErikCarlson-RGA]

I too am having this same issue. It is happening when I spawn an environment without declaring spot usage. Once I modify it to include spot declaration, thus modifying the end result manifest annotations, I get the same exact error. It also errors out on other sections that are modified, even with them added inside computed_fields.

I created computed_fields for spec.template.spec.tolerations, but not for metadata.annotations due to the documentation says I should be able to ignore adding that one.

│ Error: Provider produced inconsistent result after apply
│
│ When applying changes to
│ module.k8-proxy.kubernetes_manifest.proxy-deployment, provider
│ "provider[\"registry.terraform.io/hashicorp/kubernetes\"]" produced an
│ unexpected new value: .object.spec.template.spec.tolerations: was null, but
│ now
│ cty.ListVal([]cty.Value{cty.ObjectVal(map[string]cty.Value{"effect":cty.StringVal("NoSchedule"),
│ "key":cty.StringVal("cloud.google.com/gke-spot"),
│ "operator":cty.StringVal("Equal"),
│ "tolerationSeconds":cty.NullVal(cty.Number),
│ "value":cty.StringVal("true")})}).
│
│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.

Versions of providers and CLI used:

- hashicorp/google-beta: v4.8.0
- hashicorp/kubernetes: v2.12.1
- Terraform: v1.2.2

[jbg]

I created computed_fields for spec.template.spec.tolerations, but not for metadata.annotations due to the documentation says I should be able to ignore adding that one.

The documentation says that the default is ["metadata.labels", "metadata.annotations"]. If you set computed_fields yourself, you need to list those if you want them.

Unfortunately the docs are not as clear as they could be about this, but the part I've bolded below makes it clear:

IMPORTANT: By default, metadata.labels and metadata.annotations are already included in the list. You don't have to set them explicitly in the computed_fields list. To turn off these defaults, set the value of computed_fields to an empty list or a concrete list of other fields. For example computed_fields = [].

You've set it to a concrete list of other fields, so the defaults are overridden, and you need to include them explicitly if you want them.

[Fodoj]

I am having the same issue as @thomas-riccardi - regardless if I omit computed_fields compltely, or if I set it to computed_fields = ["metadata.annotations","metadata.labels"], change in annotation makes Terraform throw errors at me.

[chamini2]

I don't know if you have to be more specific to avoid it from raising, but I changed to

    "metadata.labels",
    "metadata.annotations",
    "metadata.annotations[\"deprecated.daemonset.template.generation\"]",

and it passed

EDIT

OK, it seems that did not fix it. It just happens that we apply the changes twice and the second time it does not fail. (?)

[rigman24]

Is there any resolution for this? This bug means you essentially cannot use a replicaset with this provider, which seems... very wrong. I have tried the above workarounds, which according to the documentation should work, but not having any luck.

example:

When applying changes to │ module.deployments["(redacted)"].kubernetes_manifest.kubernetes-manifest["templates/deployment.yaml"], provider │ "provider[\"registry.terraform.io/hashicorp/kubernetes\"]" produced an unexpected new value: │ .object.metadata.annotations["deployment.kubernetes.io/revision"]: was cty.StringVal("10"), but now cty.StringVal("11").

my manifest computed_fields looks like this:

computed_fields = ["spec.ingress", "metadata", "metadata.annotations", "metadata.annotations[\"deployment. kubernetes.io/revision\"]"]

I have tried without the escaped quotes, but it will not plan because it expects a string. It seems like it is interpreting the escaped sequences literally, but I cannot get around this issue.

edit: my particular issue appears to be a bug with a wait block with rollout = true set for the deployment. When I remove this block, I do not see this behavior anymore, no matter what computed_fields is set to.

[sbocinec]

I confirm, we do experience exactly the same issue as @rigman24 - when wait { rollout = true } is provided for a Daemonset in the kubernetes_manifest resource, the apply of the configuration consistently fail wit the above error that requires a retry for the error to disappear.

It's currently very incovenient to manage daemonsets using this resource and there are no many other options as the gavinbunney/kubectl provider is no more maintained...

EDIT: So I have troubleshooted further, and I'm experiencing the error even when I remove the wait block from the resource.

I plan to check if migrating the DaemonSet from kubernetes_manifest to kubernetes_daemon_set could address this issue.

[sbocinec]

I've tried to use kubernetes_daemon_set_v1 resource - it does not throw Error: Provider produced inconsistent result after apply error, but it does not respect wait_for_rollout = true and is always applied immediately, as you can read in #2092. It's a magic circle, there is no way how to correctly manage DaemonSets in kubernetes provider right now.

[johan3141592]

I'm using kubernetes_manifest and an AWS EKS cluster and run into the same issue as @thomas-riccardi.

Terraform v1.9.5
on darwin_arm64
+ provider registry.terraform.io/hashicorp/kubernetes v2.32.0

@alexsomesan it looks like #2432 (created by @BBBmau and marked as closing this issue) is assigned to you, do you have an estimate of when this will be merged?

[JulesClaussen]

We have the same issue with deployments as well, where each update of the deployments triggers the error:
... produced an unexpected new value: .object.metadata.annotations["deployment.kubernetes.io/revision"]: was cty.StringVal("14"), but now cty.StringVal("15").

This has been ongoing for quite a while, does anyone have a workaround?

[gnuletik]

To sum up the issue here, when applying a Deployment through kubernetes_manifest with wait { rollout = true }, this error always occurs.

#2432 seems to fix the issue but the PR is awaiting review since April 2024.
@iBrandyJackson @alexsomesan could you have a look at this PR?

Thanks!

[jan-g]

I hate to add a simple "me too", but we're seeing the same thing with the deployment.kubernetes.io/revision being spuriously carped about.

There seemed to be some energy being spent on this - has it gone off the boil?