ArgoCD keeps trying to update unchanged DNSRecordSets

[peter-tar]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

ArgoCD tries updating DNSRecordSet (config-connector) resources (and keeps failing) even though there are no changes to the resource.

The fact that it fails to sync in case of a genuine change is not unexpected. Similar problems have been discussed here: #5882 and I think this PR will be useful as a workaround argoproj/gitops-engine#526 .

However the real problem is that these DNSRecordSets aren't being changed. We don't change them. Argocd sees them as synced and green. There is nothing in the diff. If I force replace the resource (it makes the problem go away for a few days), there are no differences between the old and new resource except for a few computed fields (metadata.creationTimestamp, metadata.resourceVersion, metadata.uid).
We've been experiencing this problem for at least a couple of years.

To Reproduce

1. Create a DNSRecordSet resource using the following format:

apiVersion: dns.cnrm.cloud.google.com/v1beta1
kind: DNSRecordSet
metadata:
  annotations:
    argocd.argoproj.io/sync-options: Replace=true # This has been added as a potential workaround/remedy, but it's not helping. We had the same problem without
  labels:
    app.kubernetes.io/instance: <omitted>
    app.kubernetes.io/name: <omitted>
    app.kubernetes.io/part-of: <omitted>
  name: dns-record-1
  namespace: development
spec:
  managedZoneRef:
    external: <omitted>
  name: <omitted>
  rrdatas:
    - <omitted>
  ttl: 300
  type: A

2. Let Argo apply it
3. Either sync your resource/Application manually or wait for the next automated sync
4. Find your Argo Application failing to sync with the following error message

error when replacing "/dev/shm/3513087848": admission webhook "deny-immutable-field-updates.cnrm.cloud.google.com" denied the request: annotation cnrm.cloud.google.com/state-into-spec is immutable (retried 5 times).

5. Force replace the resource hoping that it solves your problem
6. Try syncing it again and watch the sync fail

Expected behavior

I would expect ArgoCD to not attempt updating an unchanged resource.

Screenshots

Version

argocd-server: v2.7.3+e7891b8.dirty
  BuildDate: 2023-05-24T15:52:17Z
  GitCommit: e7891b899a35dca06ae94965ea5ae2a86b344848
  GitTreeState: dirty
  GoVersion: go1.20.4
  Compiler: gc
  Platform: darwin/arm64
  Kustomize Version: v5.0.3 2023-05-09T20:05:13Z
  Helm Version: v3.12.0+gc9f554d
  Kubectl Version: v0.24.2
  Jsonnet Version: v0.19.1

Logs

error when replacing "/dev/shm/3513087848": admission webhook "deny-immutable-field-updates.cnrm.cloud.google.com" denied the request: annotation cnrm.cloud.google.com/state-into-spec is immutable (retried 5 times).

[peter-tar]

I tried a couple more things.

a) I noticed that the rrdatas field has been deprecated. I switched to rrdatasRefs but it made no difference with regards to this bug.

b) Even though I could not see any unexpected or automatically added fields, I tried changing the config-connector annotation state-into-spec to absent.
https://cloud.google.com/config-connector/docs/concepts/ignore-unspecified-fields#resolve_fighting_between_config_management_tools_and
This changed the error message for the sync failure to:

one or more objects failed to apply, reason: error when replacing "/dev/shm/952324908": admission webhook "deny-immutable-field-updates.cnrm.cloud.google.com" denied the request: error validating container annotations: cannot make changes to container annotation cnrm.cloud.google.com/project-id

Please note, that the project id was not changed, in fact, nothing was.

It also turns out that the sync fails immediately on the first try after successfully creating the resource. (Maybe we just haven't noticed it in the past.)

[peter-tar]

Got some more information from the kind people over at k8s-config-connector. So apparently config-connector sets a few resource annotations after deployment and this is what's causing the sync problem.

I added the following three annotations to the resource definition and ArgoCD no longer fails to sync it:

    cnrm.cloud.google.com/management-conflict-prevention-policy: none
    cnrm.cloud.google.com/project-id: <project-id>
    cnrm.cloud.google.com/state-into-spec: merge

However I still think there must be something odd going on with this resource type. We have lots of different config-connector resources in the cluster (buckets, bigquery datasets and tables, pubsub topics and schemas, etc.) and most if not all of them set at least some of the above annotations. There are resources with the exact same ones. Yet we only ever experienced this problem with the DNSRecordSet resource. I wonder what's different?

Also, on a successful ArgoCD sync other resources are marked as unchanged:

pubsubtopic.pubsub.cnrm.cloud.google.com/topic-name unchanged

While these DNSRecordSets are marked as replaced:

dnsrecordset.dns.cnrm.cloud.google.com/record-name replaced

But the DNSRecordSet resource in the cluster is in-fact untouched.