Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] postgresql cluster restart error #3966

Closed
JashBook opened this issue Jun 26, 2023 · 0 comments · Fixed by #4024
Closed

[BUG] postgresql cluster restart error #3966

JashBook opened this issue Jun 26, 2023 · 0 comments · Fixed by #4024
Assignees
@Y-Rookie
Labels
bug kind/bug Something isn't working
Milestone
Release 0.6.0

Comments

@JashBook
Copy link
Collaborator

Describe the bug
postgresql cluster restart error.

kbcli version
Kubernetes: v1.25.10-eks-c12679a
KubeBlocks: 0.6.0-alpha.23
kbcli: v0.6.0-alpha.23

To Reproduce
Steps to reproduce the behavior:

  1. create cluster
  2. restart
  3. See error
kubectl get ops,pod,cluster,sts,pvc -l app.kubernetes.io/instance=cluster-hkjsxx
NAME                                                         TYPE      CLUSTER          STATUS    PROGRESS   AGE
opsrequest.apps.kubeblocks.io/cluster-hkjsxx-restart-rg6z4   Restart   cluster-hkjsxx   Running   0/3        13m

NAME                              READY   STATUS    RESTARTS   AGE
pod/cluster-hkjsxx-postgresql-0   4/5     Running   0          13m
pod/cluster-hkjsxx-postgresql-1   4/5     Running   0          12m
pod/cluster-hkjsxx-postgresql-2   4/5     Running   0          12m

NAME                                        CLUSTER-DEFINITION   VERSION              TERMINATION-POLICY   STATUS     AGE
cluster.apps.kubeblocks.io/cluster-hkjsxx   postgresql           postgresql-12.15.0   DoNotTerminate       Updating   22m

NAME                                         READY   AGE
statefulset.apps/cluster-hkjsxx-postgresql   0/3     22m

describe cluster

kubectl describe cluster cluster-hkjsxx
Name:         cluster-hkjsxx
Namespace:    default
Labels:       app.kubernetes.io/instance=cluster-hkjsxx
              clusterdefinition.kubeblocks.io/name=postgresql
              clusterversion.kubeblocks.io/name=postgresql-12.15.0
Annotations:  kubeblocks.io/ops-request: [{"name":"cluster-hkjsxx-restart-rg6z4","type":"Restart"}]
              kubeblocks.io/reconcile: 2023-06-26T13:31:44.807853849Z
API Version:  apps.kubeblocks.io/v1alpha1
Kind:         Cluster
Metadata:
  Creation Timestamp:  2023-06-26T13:12:01Z
  Finalizers:
    cluster.kubeblocks.io/finalizer
  Generation:        7
  Resource Version:  115732
  UID:               8ded0b09-4416-495d-898d-138e3f61d145
Spec:
  Affinity:
    Pod Anti Affinity:     Preferred
    Tenancy:               SharedNode
  Cluster Definition Ref:  postgresql
  Cluster Version Ref:     postgresql-12.15.0
  Component Specs:
    Class Def Ref:
      Class:            
    Component Def Ref:  postgresql
    Monitor:            false
    Name:               postgresql
    No Create PDB:      false
    Replicas:           3
    Resources:
      Limits:
        Cpu:     300m
        Memory:  966367641600m
      Requests:
        Cpu:               300m
        Memory:            966367641600m
    Service Account Name:  kb-sa-cluster-hkjsxx
    Switch Policy:
      Type:  Noop
    Volume Claim Templates:
      Name:  data
      Spec:
        Access Modes:
          ReadWriteOnce
        Resources:
          Requests:
            Storage:         15Gi
        Storage Class Name:  kb-default-sc
  Termination Policy:        DoNotTerminate
Status:
  Cluster Def Generation:  2
  Components:
    Postgresql:
      Phase:       Updating
      Pods Ready:  false
      Replication Set Status:
        Primary:
          Pod:  cluster-hkjsxx-postgresql-0
        Secondaries:
          Pod:  cluster-hkjsxx-postgresql-1
          Pod:  cluster-hkjsxx-postgresql-2
  Conditions:
    Last Transition Time:  2023-06-26T13:20:35Z
    Message:               Restart opsRequest: cluster-hkjsxx-restart-rg6z4 is processing
    Reason:                Restarting
    Status:                False
    Type:                  LatestOpsRequestProcessed
    Last Transition Time:  2023-06-26T13:12:01Z
    Message:               The operator has started the provisioning of Cluster: cluster-hkjsxx
    Observed Generation:   7
    Reason:                PreCheckSucceed
    Status:                True
    Type:                  ProvisioningStarted
    Last Transition Time:  2023-06-26T13:20:05Z
    Message:               Successfully applied for resources
    Observed Generation:   7
    Reason:                ApplyResourcesSucceed
    Status:                True
    Type:                  ApplyResources
    Last Transition Time:  2023-06-26T13:20:05Z
    Message:               pods are not ready in Components: [postgresql], refer to related component message in Cluster.status.components
    Reason:                ReplicasNotReady
    Status:                False
    Type:                  ReplicasReady
    Last Transition Time:  2023-06-26T13:20:05Z
    Message:               pods are unavailable in Components: [postgresql], refer to related component message in Cluster.status.components
    Reason:                ComponentsNotReady
    Status:                False
    Type:                  Ready
  Observed Generation:     7
  Phase:                   Updating
Events:
  Type     Reason                    Age                 From                       Message
  ----     ------                    ----                ----                       -------
  Normal   ComponentPhaseTransition  20m                 cluster-controller         Create a new component
  Normal   AllReplicasReady          19m                 cluster-controller         all pods of components are ready, waiting for the probe detection successful
  Normal   ClusterReady              19m                 cluster-controller         Cluster: cluster-hkjsxx is ready, current phase is Running
  Normal   SysAcctCreate             19m                 system-account-controller  Created accounts for cluster: cluster-hkjsxx, component: postgresql, accounts: kbmonitoring
  Normal   SysAcctCreate             19m                 system-account-controller  Created accounts for cluster: cluster-hkjsxx, component: postgresql, accounts: kbreplicator
  Normal   SysAcctCreate             19m                 system-account-controller  Created accounts for cluster: cluster-hkjsxx, component: postgresql, accounts: kbadmin
  Normal   SysAcctCreate             19m                 system-account-controller  Created accounts for cluster: cluster-hkjsxx, component: postgresql, accounts: kbprobe
  Normal   SysAcctCreate             19m                 system-account-controller  Created accounts for cluster: cluster-hkjsxx, component: postgresql, accounts: kbdataprotection
  Normal   Reconfiguring             18m                 ops-request-controller     Start to process the Reconfiguring opsRequest "cluster-hkjsxx-reconfiguring-jcptq" in Cluster: cluster-hkjsxx
  Normal   Running                   18m (x2 over 19m)   cluster-controller         Cluster: cluster-hkjsxx is ready, current phase is Running
  Normal   Processed                 18m                 cluster-controller         Reconfiguring opsRequest: cluster-hkjsxx-reconfiguring-jcptq has been processed
  Normal   ComponentPhaseTransition  18m (x2 over 18m)   cluster-controller         Component workload updated
  Warning  VerticalScaling           18m                 cluster-controller         VerticalScaling opsRequest: cluster-hkjsxx-verticalscaling-qgldj is processing
  Normal   VerticalScaling           18m                 ops-request-controller     Start to process the VerticalScaling opsRequest "cluster-hkjsxx-verticalscaling-qgldj" in Cluster: cluster-hkjsxx
  Normal   PreCheckSucceed           18m (x2 over 20m)   cluster-controller         The operator has started the provisioning of Cluster: cluster-hkjsxx
  Warning  ComponentsNotReady        18m                 cluster-controller         pods are unavailable in Components: [postgresql], refer to related component message in Cluster.status.components
  Normal   WaitingForProbeSuccess    18m (x5 over 18m)   cluster-controller         Waiting for probe success
  Warning  ReplicasNotReady          18m                 cluster-controller         pods are not ready in Components: [postgresql], refer to related component message in Cluster.status.components
  Warning  ApplyResourcesFailed      17m                 cluster-controller         Operation cannot be fulfilled on statefulsets.apps "cluster-hkjsxx-postgresql": the object has been modified; please apply your changes to the latest version and try again
  Normal   ApplyResourcesSucceed     17m (x3 over 20m)   cluster-controller         Successfully applied for resources
  Normal   VolumeExpansion           17m                 ops-request-controller     Start to process the VolumeExpansion opsRequest "cluster-hkjsxx-volumeexpansion-fxpt8" in Cluster: cluster-hkjsxx
  Normal   Stop                      16m                 ops-request-controller     Start to process the Stop opsRequest "cluster-hkjsxx-stop-g25md" in Cluster: cluster-hkjsxx
  Normal   Start                     15m                 ops-request-controller     Start to process the Start opsRequest "cluster-hkjsxx-start-6425x" in Cluster: cluster-hkjsxx
  Normal   ComponentPhaseTransition  15m (x5 over 19m)   cluster-controller         Running: true, PodsReady: true, PodsTimedout: false
  Normal   Reconfiguring             15m                 ops-request-controller     Start to process the Reconfiguring opsRequest "cluster-hkjsxx-reconfiguring-wgrrd" in Cluster: cluster-hkjsxx
  Normal   HorizontalScaling         13m                 ops-request-controller     Start to process the HorizontalScaling opsRequest "cluster-hkjsxx-horizontalscaling-xf9hj" in Cluster: cluster-hkjsxx
  Normal   Restart                   12m                 ops-request-controller     Start to process the Restart opsRequest "cluster-hkjsxx-restart-rg6z4" in Cluster: cluster-hkjsxx
  Warning  Unhealthy                 106s (x4 over 11m)  event-controller           Pod cluster-hkjsxx-postgresql-0: Readiness probe failed: 127.0.0.1:5432 - no response
  Warning  Unhealthy                 96s (x5 over 19m)   event-controller           Pod cluster-hkjsxx-postgresql-1: Readiness probe failed: 127.0.0.1:5432 - no response
  Warning  Unhealthy                 66s (x6 over 10m)   event-controller           Pod cluster-hkjsxx-postgresql-2: Readiness probe failed: 127.0.0.1:5432 - no response

logs pod

kubectl logs cluster-hkjsxx-postgresql-0 
Defaulted container "postgresql" out of: postgresql, pgbouncer, metrics, kb-checkrole, config-manager, pg-init-container (init)
+ attempt=1
+ max_attempts=5
+ '[' 1 -le 5 ']'
++ cat /kb-podinfo/primary-pod
+ '[' -z '' ']'
+ sleep 5
+ attempt=2
+ '[' 2 -le 5 ']'
++ cat /kb-podinfo/primary-pod
+ '[' -z cluster-hkjsxx-postgresql-1 ']'
++ cat /kb-podinfo/primary-pod
+ primary=cluster-hkjsxx-postgresql-1
+ echo 'DownwardAPI get primary=cluster-hkjsxx-postgresql-1'
+ echo KB_POD_NAME=cluster-hkjsxx-postgresql-0
+ '[' -z cluster-hkjsxx-postgresql-1 ']'
+ '[' cluster-hkjsxx-postgresql-1 '!=' cluster-hkjsxx-postgresql-0 ']'
+ primary_fqdn=cluster-hkjsxx-postgresql-1.cluster-hkjsxx-postgresql-headless.default.svc
+ echo primary_fqdn=cluster-hkjsxx-postgresql-1.cluster-hkjsxx-postgresql-headless.default.svc
+ pg_isready -U postgres -h cluster-hkjsxx-postgresql-1.cluster-hkjsxx-postgresql-headless.default.svc -p 5432
cluster-hkjsxx-postgresql-1.cluster-hkjsxx-postgresql-headless.default.svc:5432 - no response
+ sleep 5
+ pg_isready -U postgres -h cluster-hkjsxx-postgresql-1.cluster-hkjsxx-postgresql-headless.default.svc -p 5432
cluster-hkjsxx-postgresql-1.cluster-hkjsxx-postgresql-headless.default.svc:5432 - no response
+ sleep 5
+ pg_isready -U postgres -h cluster-hkjsxx-postgresql-1.cluster-hkjsxx-postgresql-headless.default.svc -p 5432
cluster-hkjsxx-postgresql-1.cluster-hkjsxx-postgresql-headless.default.svc:5432 - no response
+ sleep 5
+ pg_isready -U postgres -h cluster-hkjsxx-postgresql-1.cluster-hkjsxx-postgresql-headless.default.svc -p 5432
cluster-hkjsxx-postgresql-1.cluster-hkjsxx-postgresql-headless.default.svc:5432 - no response
...
+ pg_isready -U postgres -h cluster-hkjsxx-postgresql-1.cluster-hkjsxx-postgresql-headless.default.svc -p 5432
cluster-hkjsxx-postgresql-1.cluster-hkjsxx-postgresql-headless.default.svc:5432 - no response
+ sleep 5
+ pg_isready -U postgres -h cluster-hkjsxx-postgresql-1.cluster-hkjsxx-postgresql-headless.default.svc -p 5432
cluster-hkjsxx-postgresql-1.cluster-hkjsxx-postgresql-headless.default.svc:5432 - no response
+ sleep 5
+ pg_isready -U postgres -h cluster-hkjsxx-postgresql-1.cluster-hkjsxx-postgresql-headless.default.svc -p 5432
cluster-hkjsxx-postgresql-1.cluster-hkjsxx-postgresql-headless.default.svc:5432 - no response
+ sleep 5

➜  kubectl logs cluster-hkjsxx-postgresql-1 
Defaulted container "postgresql" out of: postgresql, pgbouncer, metrics, kb-checkrole, config-manager, pg-init-container (init)
+ attempt=1
+ max_attempts=5
+ '[' 1 -le 5 ']'
++ cat /kb-podinfo/primary-pod
+ '[' -z cluster-hkjsxx-postgresql-0 ']'
++ cat /kb-podinfo/primary-pod
+ primary=cluster-hkjsxx-postgresql-0
+ echo 'DownwardAPI get primary=cluster-hkjsxx-postgresql-0'
+ echo KB_POD_NAME=cluster-hkjsxx-postgresql-1
+ '[' -z cluster-hkjsxx-postgresql-0 ']'
+ '[' cluster-hkjsxx-postgresql-0 '!=' cluster-hkjsxx-postgresql-1 ']'
+ primary_fqdn=cluster-hkjsxx-postgresql-0.cluster-hkjsxx-postgresql-headless.default.svc
+ echo primary_fqdn=cluster-hkjsxx-postgresql-0.cluster-hkjsxx-postgresql-headless.default.svc
+ pg_isready -U postgres -h cluster-hkjsxx-postgresql-0.cluster-hkjsxx-postgresql-headless.default.svc -p 5432
cluster-hkjsxx-postgresql-0.cluster-hkjsxx-postgresql-headless.default.svc:5432 - no response
+ sleep 5
+ pg_isready -U postgres -h cluster-hkjsxx-postgresql-0.cluster-hkjsxx-postgresql-headless.default.svc -p 5432
cluster-hkjsxx-postgresql-0.cluster-hkjsxx-postgresql-headless.default.svc:5432 - no response
+ sleep 5
+ pg_isready -U postgres -h cluster-hkjsxx-postgresql-0.cluster-hkjsxx-postgresql-headless.default.svc -p 5432
cluster-hkjsxx-postgresql-0.cluster-hkjsxx-postgresql-headless.default.svc:5432 - no response
+ sleep 5
+ pg_isready -U postgres -h cluster-hkjsxx-postgresql-0.cluster-hkjsxx-postgresql-headless.default.svc -p 5432
cluster-hkjsxx-postgresql-0.cluster-hkjsxx-postgresql-headless.default.svc:5432 - no response
+ sleep 5
+ pg_isready -U postgres -h cluster-hkjsxx-postgresql-0.cluster-hkjsxx-postgresql-headless.default.svc -p 5432
cluster-hkjsxx-postgresql-0.cluster-hkjsxx-postgresql-headless.default.svc:5432 - no response
+ sleep 5
+ pg_isready -U postgres -h cluster-hkjsxx-postgresql-0.cluster-hkjsxx-postgresql-headless.default.svc -p 5432
cluster-hkjsxx-postgresql-0.cluster-hkjsxx-postgresql-headless.default.svc:5432 - no response
+ sleep 5
...
+ pg_isready -U postgres -h cluster-hkjsxx-postgresql-0.cluster-hkjsxx-postgresql-headless.default.svc -p 5432
cluster-hkjsxx-postgresql-0.cluster-hkjsxx-postgresql-headless.default.svc:5432 - no response
+ sleep 5
+ pg_isready -U postgres -h cluster-hkjsxx-postgresql-0.cluster-hkjsxx-postgresql-headless.default.svc -p 5432
cluster-hkjsxx-postgresql-0.cluster-hkjsxx-postgresql-headless.default.svc:5432 - no response
+ sleep 5
+ pg_isready -U postgres -h cluster-hkjsxx-postgresql-0.cluster-hkjsxx-postgresql-headless.default.svc -p 5432
cluster-hkjsxx-postgresql-0.cluster-hkjsxx-postgresql-headless.default.svc:5432 - no response
+ sleep 5
+ pg_isready -U postgres -h cluster-hkjsxx-postgresql-0.cluster-hkjsxx-postgresql-headless.default.svc -p 5432
cluster-hkjsxx-postgresql-0.cluster-hkjsxx-postgresql-headless.default.svc:5432 - no response
+ sleep 5
+ pg_isready -U postgres -h cluster-hkjsxx-postgresql-0.cluster-hkjsxx-postgresql-headless.default.svc -p 5432
cluster-hkjsxx-postgresql-0.cluster-hkjsxx-postgresql-headless.default.svc:5432 - no response
+ sleep 5
+ pg_isready -U postgres -h cluster-hkjsxx-postgresql-0.cluster-hkjsxx-postgresql-headless.default.svc -p 5432
cluster-hkjsxx-postgresql-0.cluster-hkjsxx-postgresql-headless.default.svc:5432 - no response
+ sleep 5
+ pg_isready -U postgres -h cluster-hkjsxx-postgresql-0.cluster-hkjsxx-postgresql-headless.default.svc -p 5432
cluster-hkjsxx-postgresql-0.cluster-hkjsxx-postgresql-headless.default.svc:5432 - no response
+ sleep 5
+ pg_isready -U postgres -h cluster-hkjsxx-postgresql-0.cluster-hkjsxx-postgresql-headless.default.svc -p 5432
cluster-hkjsxx-postgresql-0.cluster-hkjsxx-postgresql-headless.default.svc:5432 - no response
+ sleep 5

➜ kubectl logs cluster-hkjsxx-postgresql-2
Defaulted container "postgresql" out of: postgresql, pgbouncer, metrics, kb-checkrole, config-manager, pg-init-container (init)
+ attempt=1
+ max_attempts=5
+ '[' 1 -le 5 ']'
++ cat /kb-podinfo/primary-pod
+ '[' -z cluster-hkjsxx-postgresql-0 ']'
++ cat /kb-podinfo/primary-pod
+ primary=cluster-hkjsxx-postgresql-0
+ echo 'DownwardAPI get primary=cluster-hkjsxx-postgresql-0'
+ echo KB_POD_NAME=cluster-hkjsxx-postgresql-2
+ '[' -z cluster-hkjsxx-postgresql-0 ']'
+ '[' cluster-hkjsxx-postgresql-0 '!=' cluster-hkjsxx-postgresql-2 ']'
+ primary_fqdn=cluster-hkjsxx-postgresql-0.cluster-hkjsxx-postgresql-headless.default.svc
+ echo primary_fqdn=cluster-hkjsxx-postgresql-0.cluster-hkjsxx-postgresql-headless.default.svc
+ pg_isready -U postgres -h cluster-hkjsxx-postgresql-0.cluster-hkjsxx-postgresql-headless.default.svc -p 5432
cluster-hkjsxx-postgresql-0.cluster-hkjsxx-postgresql-headless.default.svc:5432 - no response
+ sleep 5
+ pg_isready -U postgres -h cluster-hkjsxx-postgresql-0.cluster-hkjsxx-postgresql-headless.default.svc -p 5432
cluster-hkjsxx-postgresql-0.cluster-hkjsxx-postgresql-headless.default.svc:5432 - no response
+ sleep 5
+ pg_isready -U postgres -h cluster-hkjsxx-postgresql-0.cluster-hkjsxx-postgresql-headless.default.svc -p 5432
cluster-hkjsxx-postgresql-0.cluster-hkjsxx-postgresql-headless.default.svc:5432 - no response
+ sleep 5
+ pg_isready -U postgres -h cluster-hkjsxx-postgresql-0.cluster-hkjsxx-postgresql-headless.default.svc -p 5432
cluster-hkjsxx-postgresql-0.cluster-hkjsxx-postgresql-headless.default.svc:5432 - no response
+ sleep 5
+ pg_isready -U postgres -h cluster-hkjsxx-postgresql-0.cluster-hkjsxx-postgresql-headless.default.svc -p 5432
cluster-hkjsxx-postgresql-0.cluster-hkjsxx-postgresql-headless.default.svc:5432 - no response
+ sleep 5
+ pg_isready -U postgres -h cluster-hkjsxx-postgresql-0.cluster-hkjsxx-postgresql-headless.default.svc -p 5432
cluster-hkjsxx-postgresql-0.cluster-hkjsxx-postgresql-headless.default.svc:5432 - no response
+ sleep 5
...
+ pg_isready -U postgres -h cluster-hkjsxx-postgresql-0.cluster-hkjsxx-postgresql-headless.default.svc -p 5432
cluster-hkjsxx-postgresql-0.cluster-hkjsxx-postgresql-headless.default.svc:5432 - no response
+ sleep 5
+ pg_isready -U postgres -h cluster-hkjsxx-postgresql-0.cluster-hkjsxx-postgresql-headless.default.svc -p 5432
cluster-hkjsxx-postgresql-0.cluster-hkjsxx-postgresql-headless.default.svc:5432 - no response
+ sleep 5
+ pg_isready -U postgres -h cluster-hkjsxx-postgresql-0.cluster-hkjsxx-postgresql-headless.default.svc -p 5432
cluster-hkjsxx-postgresql-0.cluster-hkjsxx-postgresql-headless.default.svc:5432 - no response
+ sleep 5

logs kubeblocks

kubectl logs kubeblocks-976666857-4pt6w|grep cluster-hkjsxx

2023-06-26T13:13:22.058Z	INFO	eventhandler.EnqueueRequestForAncestor	cloud not find ancestor object	{"source object gvk": "/, Kind=", "name": "sysacc-cluster-hkjsxx-postgresql-kbadmin-2frr6-6b5d4", "up to level": 2}
2023-06-26T13:13:22.058Z	INFO	eventhandler.EnqueueRequestForAncestor	cloud not find ancestor object	{"source object gvk": "/, Kind=", "name": "sysacc-cluster-hkjsxx-postgresql-kbreplicator-jc9nk-64wtp", "up to level": 2}
2023-06-26T13:14:14.969Z	INFO	update cluster status after applying resources 	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"cluster-hkjsxx","namespace":"default"}, "namespace": "default", "name": "cluster-hkjsxx", "reconcileID": "93cbddcc-5b4d-48a7-a20c-dc9720ff72bf", "cluster": "default/cluster-hkjsxx"}
2023-06-26T13:14:15.453Z	INFO	replicationSet new role is invalid, please check	{"controller": "event", "controllerGroup": "", "controllerKind": "Event", "Event": {"name":"cluster-hkjsxx-postgresql-0.tkx4hbc2vrx6h2m5","namespace":"default"}, "namespace": "default", "name": "cluster-hkjsxx-postgresql-0.tkx4hbc2vrx6h2m5", "reconcileID": "e33a4d78-b983-4557-9000-4a2c35ba1cd2", "event": "default/cluster-hkjsxx-postgresql-0.tkx4hbc2vrx6h2m5", "new role": ""}
2023-06-26T13:14:21.426Z	INFO	replicationSet new role is invalid, please check	{"controller": "event", "controllerGroup": "", "controllerKind": "Event", "Event": {"name":"cluster-hkjsxx-postgresql-1.cbvfdgtcb9kpkxs5","namespace":"default"}, "namespace": "default", "name": "cluster-hkjsxx-postgresql-1.cbvfdgtcb9kpkxs5", "reconcileID": "b72d8dec-b8eb-4c78-a2da-31240526b3b7", "event": "default/cluster-hkjsxx-postgresql-1.cbvfdgtcb9kpkxs5", "new role": ""}
2023-06-26T13:14:33.041Z	INFO	update cluster status after applying resources 	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"cluster-hkjsxx","namespace":"default"}, "namespace": "default", "name": "cluster-hkjsxx", "reconcileID": "9764f88c-c510-471b-8fce-09c0950d4dd0", "cluster": "default/cluster-hkjsxx"}
2023-06-26T13:15:04.626Z	ERROR	UPDATE *v1.StatefulSet error	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"cluster-hkjsxx","namespace":"default"}, "namespace": "default", "name": "cluster-hkjsxx", "reconcileID": "c66956e1-b5b8-4cda-9a13-39b3af17817b", "cluster": "default/cluster-hkjsxx", "error": "Operation cannot be fulfilled on statefulsets.apps \"cluster-hkjsxx-postgresql\": the object has been modified; please apply your changes to the latest version and try again"}
2023-06-26T13:15:04.646Z	INFO	Operation cannot be fulfilled on statefulsets.apps "cluster-hkjsxx-postgresql": the object has been modified; please apply your changes to the latest version and try again	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"cluster-hkjsxx","namespace":"default"}, "namespace": "default", "name": "cluster-hkjsxx", "reconcileID": "c66956e1-b5b8-4cda-9a13-39b3af17817b", "cluster": "default/cluster-hkjsxx"}
2023-06-26T13:15:04.646Z	ERROR	Reconciler error	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"cluster-hkjsxx","namespace":"default"}, "namespace": "default", "name": "cluster-hkjsxx", "reconcileID": "c66956e1-b5b8-4cda-9a13-39b3af17817b", "error": "Operation cannot be fulfilled on statefulsets.apps \"cluster-hkjsxx-postgresql\": the object has been modified; please apply your changes to the latest version and try again"}
2023-06-26T13:15:17.149Z	INFO	replicationSet new role is invalid, please check	{"controller": "event", "controllerGroup": "", "controllerKind": "Event", "Event": {"name":"cluster-hkjsxx-postgresql-1.54mxcqvlmgt7vz5j","namespace":"default"}, "namespace": "default", "name": "cluster-hkjsxx-postgresql-1.54mxcqvlmgt7vz5j", "reconcileID": "90fe04c2-9457-4b57-bc2e-656b2be5db63", "event": "default/cluster-hkjsxx-postgresql-1.54mxcqvlmgt7vz5j", "new role": ""}
2023-06-26T13:15:21.800Z	INFO	pod current role label equals to new role, ignore it	{"controller": "event", "controllerGroup": "", "controllerKind": "Event", "Event": {"name":"cluster-hkjsxx-postgresql-0.f7fxmwpmqmltxbcl","namespace":"default"}, "namespace": "default", "name": "cluster-hkjsxx-postgresql-0.f7fxmwpmqmltxbcl", "reconcileID": "7f1e7ca9-c521-4cdb-ae28-c846ddd48a3c", "event": "default/cluster-hkjsxx-postgresql-0.f7fxmwpmqmltxbcl", "new role": "primary"}
2023-06-26T13:15:39.110Z	INFO	pod current role label equals to new role, ignore it	{"controller": "event", "controllerGroup": "", "controllerKind": "Event", "Event": {"name":"cluster-hkjsxx-postgresql-1.mxzlngfm8ljgr5jg","namespace":"default"}, "namespace": "default", "name": "cluster-hkjsxx-postgresql-1.mxzlngfm8ljgr5jg", "reconcileID": "7a55c7e0-fbae-4fa2-b597-49ec748af949", "event": "default/cluster-hkjsxx-postgresql-1.mxzlngfm8ljgr5jg", "new role": "secondary"}
2023-06-26T13:15:48.440Z	INFO	update cluster status after applying resources 	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"cluster-hkjsxx","namespace":"default"}, "namespace": "default", "name": "cluster-hkjsxx", "reconcileID": "0eb9ce4b-72a9-4abf-9936-d1a3521c3db4", "cluster": "default/cluster-hkjsxx"}
2023-06-26T13:16:20.016Z	INFO	update cluster status after applying resources 	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"cluster-hkjsxx","namespace":"default"}, "namespace": "default", "name": "cluster-hkjsxx", "reconcileID": "2c0d1aec-944e-49d9-a906-2b682adab7c4", "cluster": "default/cluster-hkjsxx"}
2023-06-26T13:16:20.188Z	ERROR	UPDATE *v1.Pod error	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"cluster-hkjsxx","namespace":"default"}, "namespace": "default", "name": "cluster-hkjsxx", "reconcileID": "2c0d1aec-944e-49d9-a906-2b682adab7c4", "cluster": "default/cluster-hkjsxx", "error": "Operation cannot be fulfilled on pods \"cluster-hkjsxx-postgresql-1\": the object has been modified; please apply your changes to the latest version and try again"}
2023-06-26T13:16:20.205Z	INFO	Operation cannot be fulfilled on pods "cluster-hkjsxx-postgresql-1": the object has been modified; please apply your changes to the latest version and try again	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"cluster-hkjsxx","namespace":"default"}, "namespace": "default", "name": "cluster-hkjsxx", "reconcileID": "2c0d1aec-944e-49d9-a906-2b682adab7c4", "cluster": "default/cluster-hkjsxx"}
2023-06-26T13:16:20.205Z	ERROR	Reconciler error	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"cluster-hkjsxx","namespace":"default"}, "namespace": "default", "name": "cluster-hkjsxx", "reconcileID": "2c0d1aec-944e-49d9-a906-2b682adab7c4", "error": "Operation cannot be fulfilled on pods \"cluster-hkjsxx-postgresql-1\": the object has been modified; please apply your changes to the latest version and try again"}
2023-06-26T13:16:20.434Z	INFO	update cluster status after applying resources 	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"cluster-hkjsxx","namespace":"default"}, "namespace": "default", "name": "cluster-hkjsxx", "reconcileID": "4656e737-e78a-422b-99f9-784c8f0e85d9", "cluster": "default/cluster-hkjsxx"}
2023-06-26T13:16:51.576Z	ERROR	UPDATE *v1.StatefulSet error	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"cluster-hkjsxx","namespace":"default"}, "namespace": "default", "name": "cluster-hkjsxx", "reconcileID": "3078ad76-19d5-4e79-bf25-095f686cf36f", "cluster": "default/cluster-hkjsxx", "error": "Operation cannot be fulfilled on statefulsets.apps \"cluster-hkjsxx-postgresql\": the object has been modified; please apply your changes to the latest version and try again"}
2023-06-26T13:16:51.593Z	INFO	Operation cannot be fulfilled on statefulsets.apps "cluster-hkjsxx-postgresql": the object has been modified; please apply your changes to the latest version and try again	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"cluster-hkjsxx","namespace":"default"}, "namespace": "default", "name": "cluster-hkjsxx", "reconcileID": "3078ad76-19d5-4e79-bf25-095f686cf36f", "cluster": "default/cluster-hkjsxx"}
2023-06-26T13:16:51.593Z	ERROR	Reconciler error	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"cluster-hkjsxx","namespace":"default"}, "namespace": "default", "name": "cluster-hkjsxx", "reconcileID": "3078ad76-19d5-4e79-bf25-095f686cf36f", "error": "Operation cannot be fulfilled on statefulsets.apps \"cluster-hkjsxx-postgresql\": the object has been modified; please apply your changes to the latest version and try again"}
2023-06-26T13:17:00.893Z	INFO	update cluster status after applying resources 	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"cluster-hkjsxx","namespace":"default"}, "namespace": "default", "name": "cluster-hkjsxx", "reconcileID": "4005aa64-1304-42ca-b26f-008d6c77d511", "cluster": "default/cluster-hkjsxx"}
2023-06-26T13:17:14.932Z	INFO	pod current role label equals to new role, ignore it	{"controller": "event", "controllerGroup": "", "controllerKind": "Event", "Event": {"name":"cluster-hkjsxx-postgresql-0.k9bhfsnd5b5tcc6k","namespace":"default"}, "namespace": "default", "name": "cluster-hkjsxx-postgresql-0.k9bhfsnd5b5tcc6k", "reconcileID": "2500281b-c18b-4431-9ab2-82b7c6efbcc0", "event": "default/cluster-hkjsxx-postgresql-0.k9bhfsnd5b5tcc6k", "new role": "primary"}
2023-06-26T13:17:19.982Z	INFO	replicationSet new role is invalid, please check	{"controller": "event", "controllerGroup": "", "controllerKind": "Event", "Event": {"name":"cluster-hkjsxx-postgresql-1.ffhw4sfp8qn9kts4","namespace":"default"}, "namespace": "default", "name": "cluster-hkjsxx-postgresql-1.ffhw4sfp8qn9kts4", "reconcileID": "f995604b-675f-4c46-ab94-409743fa5bb4", "event": "default/cluster-hkjsxx-postgresql-1.ffhw4sfp8qn9kts4", "new role": ""}
2023-06-26T13:17:32.033Z	INFO	pod current role label equals to new role, ignore it	{"controller": "event", "controllerGroup": "", "controllerKind": "Event", "Event": {"name":"cluster-hkjsxx-postgresql-1.lwslf8h2fzcjqs4g","namespace":"default"}, "namespace": "default", "name": "cluster-hkjsxx-postgresql-1.lwslf8h2fzcjqs4g", "reconcileID": "4056caca-86b1-46f9-9eaf-391ce9cfd471", "event": "default/cluster-hkjsxx-postgresql-1.lwslf8h2fzcjqs4g", "new role": "secondary"}
2023-06-26T13:18:15.916Z	ERROR	UPDATE *v1.Pod error	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"cluster-hkjsxx","namespace":"default"}, "namespace": "default", "name": "cluster-hkjsxx", "reconcileID": "46e41492-2f50-401a-bf90-a39ac932ca2a", "cluster": "default/cluster-hkjsxx", "error": "Operation cannot be fulfilled on pods \"cluster-hkjsxx-postgresql-1\": the object has been modified; please apply your changes to the latest version and try again"}
2023-06-26T13:18:15.940Z	INFO	Operation cannot be fulfilled on pods "cluster-hkjsxx-postgresql-1": the object has been modified; please apply your changes to the latest version and try again	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"cluster-hkjsxx","namespace":"default"}, "namespace": "default", "name": "cluster-hkjsxx", "reconcileID": "46e41492-2f50-401a-bf90-a39ac932ca2a", "cluster": "default/cluster-hkjsxx"}
2023-06-26T13:18:15.940Z	ERROR	Reconciler error	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"cluster-hkjsxx","namespace":"default"}, "namespace": "default", "name": "cluster-hkjsxx", "reconcileID": "46e41492-2f50-401a-bf90-a39ac932ca2a", "error": "Operation cannot be fulfilled on pods \"cluster-hkjsxx-postgresql-1\": the object has been modified; please apply your changes to the latest version and try again"}
2023-06-26T13:18:31.417Z	INFO	replicationSet new role is invalid, please check	{"controller": "event", "controllerGroup": "", "controllerKind": "Event", "Event": {"name":"cluster-hkjsxx-postgresql-1.kj6xqqf7bdtwcfqg","namespace":"default"}, "namespace": "default", "name": "cluster-hkjsxx-postgresql-1.kj6xqqf7bdtwcfqg", "reconcileID": "f2bc5bfe-17f5-4bca-840c-7450877086d8", "event": "default/cluster-hkjsxx-postgresql-1.kj6xqqf7bdtwcfqg", "new role": ""}
2023-06-26T13:18:36.340Z	INFO	pod is old primary and new role is secondary, do not deal with this situation	{"controller": "event", "controllerGroup": "", "controllerKind": "Event", "Event": {"name":"cluster-hkjsxx-postgresql-0.bmnrtht4h28n8rq6","namespace":"default"}, "namespace": "default", "name": "cluster-hkjsxx-postgresql-0.bmnrtht4h28n8rq6", "reconcileID": "7ba44841-d72b-4bb5-a13f-db7b6b96f7e4", "event": "default/cluster-hkjsxx-postgresql-0.bmnrtht4h28n8rq6", "podName": "cluster-hkjsxx-postgresql-0", "newRole": "secondary"}
2023-06-26T13:18:37.128Z	INFO	pod current role label equals to new role, ignore it	{"controller": "event", "controllerGroup": "", "controllerKind": "Event", "Event": {"name":"cluster-hkjsxx-postgresql-0.smcspklwlgvtxf8j","namespace":"default"}, "namespace": "default", "name": "cluster-hkjsxx-postgresql-0.smcspklwlgvtxf8j", "reconcileID": "8297d1bc-75e0-45d1-bd03-cd3a56fdcc15", "event": "default/cluster-hkjsxx-postgresql-0.smcspklwlgvtxf8j", "new role": "primary"}
2023-06-26T13:18:57.443Z	INFO	pod current role label equals to new role, ignore it	{"controller": "event", "controllerGroup": "", "controllerKind": "Event", "Event": {"name":"cluster-hkjsxx-postgresql-1.58t2xdmlq2th9zvd","namespace":"default"}, "namespace": "default", "name": "cluster-hkjsxx-postgresql-1.58t2xdmlq2th9zvd", "reconcileID": "6bc4b203-7319-421c-b81f-2a37a7df1b9a", "event": "default/cluster-hkjsxx-postgresql-1.58t2xdmlq2th9zvd", "new role": "secondary"}
2023-06-26T13:19:07.925Z	INFO	update cluster status after applying resources 	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"cluster-hkjsxx","namespace":"default"}, "namespace": "default", "name": "cluster-hkjsxx", "reconcileID": "1d832d76-870f-4f97-9cde-360ceca259f5", "cluster": "default/cluster-hkjsxx"}
2023-06-26T13:19:08.128Z	ERROR	UPDATE *v1.Pod error	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"cluster-hkjsxx","namespace":"default"}, "namespace": "default", "name": "cluster-hkjsxx", "reconcileID": "1d832d76-870f-4f97-9cde-360ceca259f5", "cluster": "default/cluster-hkjsxx", "error": "Operation cannot be fulfilled on pods \"cluster-hkjsxx-postgresql-1\": the object has been modified; please apply your changes to the latest version and try again"}
2023-06-26T13:19:08.152Z	INFO	Operation cannot be fulfilled on pods "cluster-hkjsxx-postgresql-1": the object has been modified; please apply your changes to the latest version and try again	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"cluster-hkjsxx","namespace":"default"}, "namespace": "default", "name": "cluster-hkjsxx", "reconcileID": "1d832d76-870f-4f97-9cde-360ceca259f5", "cluster": "default/cluster-hkjsxx"}
2023-06-26T13:19:08.152Z	ERROR	Reconciler error	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"cluster-hkjsxx","namespace":"default"}, "namespace": "default", "name": "cluster-hkjsxx", "reconcileID": "1d832d76-870f-4f97-9cde-360ceca259f5", "error": "Operation cannot be fulfilled on pods \"cluster-hkjsxx-postgresql-1\": the object has been modified; please apply your changes to the latest version and try again"}
2023-06-26T13:19:08.460Z	INFO	update cluster status after applying resources 	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"cluster-hkjsxx","namespace":"default"}, "namespace": "default", "name": "cluster-hkjsxx", "reconcileID": "7ddd1448-5a5e-4cdf-b56f-b6e59d2bd767", "cluster": "default/cluster-hkjsxx"}
2023-06-26T13:19:08.572Z	ERROR	UPDATE *v1.PersistentVolumeClaim error	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"cluster-hkjsxx","namespace":"default"}, "namespace": "default", "name": "cluster-hkjsxx", "reconcileID": "7ddd1448-5a5e-4cdf-b56f-b6e59d2bd767", "cluster": "default/cluster-hkjsxx", "error": "PersistentVolumeClaim \"data-cluster-hkjsxx-postgresql-2\" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests for bound claims\n  core.PersistentVolumeClaimSpec{\n  \tAccessModes: {\"ReadWriteOnce\"},\n  \tSelector:    nil,\n  \tResources: core.ResourceRequirements{\n  \t\tLimits: nil,\n- \t\tRequests: core.ResourceList{\n- \t\t\ts\"storage\": {i: resource.int64Amount{value: 11811160064}, s: \"11Gi\", Format: \"BinarySI\"},\n- \t\t},\n+ \t\tRequests: core.ResourceList{\n+ \t\t\ts\"storage\": {i: resource.int64Amount{value: 16106127360}, s: \"15Gi\", Format: \"BinarySI\"},\n+ \t\t},\n  \t},\n  \tVolumeName:       \"\",\n  \tStorageClassName: &\"kb-default-sc\",\n  \t... // 3 identical fields\n  }\n"}
2023-06-26T13:19:08.588Z	INFO	PersistentVolumeClaim "data-cluster-hkjsxx-postgresql-2" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests for bound claims
	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"cluster-hkjsxx","namespace":"default"}, "namespace": "default", "name": "cluster-hkjsxx", "reconcileID": "7ddd1448-5a5e-4cdf-b56f-b6e59d2bd767", "cluster": "default/cluster-hkjsxx"}
2023-06-26T13:19:08.589Z	ERROR	Reconciler error	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"cluster-hkjsxx","namespace":"default"}, "namespace": "default", "name": "cluster-hkjsxx", "reconcileID": "7ddd1448-5a5e-4cdf-b56f-b6e59d2bd767", "error": "PersistentVolumeClaim \"data-cluster-hkjsxx-postgresql-2\" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests for bound claims\n  core.PersistentVolumeClaimSpec{\n  \tAccessModes: {\"ReadWriteOnce\"},\n  \tSelector:    nil,\n  \tResources: core.ResourceRequirements{\n  \t\tLimits: nil,\n- \t\tRequests: core.ResourceList{\n- \t\t\ts\"storage\": {i: resource.int64Amount{value: 11811160064}, s: \"11Gi\", Format: \"BinarySI\"},\n- \t\t},\n+ \t\tRequests: core.ResourceList{\n+ \t\t\ts\"storage\": {i: resource.int64Amount{value: 16106127360}, s: \"15Gi\", Format: \"BinarySI\"},\n+ \t\t},\n  \t},\n  \tVolumeName:       \"\",\n  \tStorageClassName: &\"kb-default-sc\",\n  \t... // 3 identical fields\n  }\n"}
2023-06-26T13:19:08.779Z	INFO	update cluster status after applying resources 	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"cluster-hkjsxx","namespace":"default"}, "namespace": "default", "name": "cluster-hkjsxx", "reconcileID": "e6cba08e-1298-42c8-8b34-20c85a8b0bb9", "cluster": "default/cluster-hkjsxx"}
2023-06-26T13:19:08.847Z	ERROR	UPDATE *v1.PersistentVolumeClaim error	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"cluster-hkjsxx","namespace":"default"}, "namespace": "default", "name": "cluster-hkjsxx", "reconcileID": "e6cba08e-1298-42c8-8b34-20c85a8b0bb9", "cluster": "default/cluster-hkjsxx", "error": "PersistentVolumeClaim \"data-cluster-hkjsxx-postgresql-2\" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests for bound claims\n  core.PersistentVolumeClaimSpec{\n  \tAccessModes: {\"ReadWriteOnce\"},\n  \tSelector:    nil,\n  \tResources: core.ResourceRequirements{\n  \t\tLimits: nil,\n- \t\tRequests: core.ResourceList{\n- \t\t\ts\"storage\": {i: resource.int64Amount{value: 11811160064}, s: \"11Gi\", Format: \"BinarySI\"},\n- \t\t},\n+ \t\tRequests: core.ResourceList{\n+ \t\t\ts\"storage\": {i: resource.int64Amount{value: 16106127360}, s: \"15Gi\", Format: \"BinarySI\"},\n+ \t\t},\n  \t},\n  \tVolumeName:       \"\",\n  \tStorageClassName: &\"kb-default-sc\",\n  \t... // 3 identical fields\n  }\n"}
2023-06-26T13:19:08.858Z	INFO	PersistentVolumeClaim "data-cluster-hkjsxx-postgresql-2" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests for bound claims
	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"cluster-hkjsxx","namespace":"default"}, "namespace": "default", "name": "cluster-hkjsxx", "reconcileID": "e6cba08e-1298-42c8-8b34-20c85a8b0bb9", "cluster": "default/cluster-hkjsxx"}
2023-06-26T13:19:08.858Z	ERROR	Reconciler error	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"cluster-hkjsxx","namespace":"default"}, "namespace": "default", "name": "cluster-hkjsxx", "reconcileID": "e6cba08e-1298-42c8-8b34-20c85a8b0bb9", "error": "PersistentVolumeClaim \"data-cluster-hkjsxx-postgresql-2\" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests for bound claims\n  core.PersistentVolumeClaimSpec{\n  \tAccessModes: {\"ReadWriteOnce\"},\n  \tSelector:    nil,\n  \tResources: core.ResourceRequirements{\n  \t\tLimits: nil,\n- \t\tRequests: core.ResourceList{\n- \t\t\ts\"storage\": {i: resource.int64Amount{value: 11811160064}, s: \"11Gi\", Format: \"BinarySI\"},\n- \t\t},\n+ \t\tRequests: core.ResourceList{\n+ \t\t\ts\"storage\": {i: resource.int64Amount{value: 16106127360}, s: \"15Gi\", Format: \"BinarySI\"},\n+ \t\t},\n  \t},\n  \tVolumeName:       \"\",\n  \tStorageClassName: &\"kb-default-sc\",\n  \t... // 3 identical fields\n  }\n"}
2023-06-26T13:19:09.113Z	INFO	update cluster status after applying resources 	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"cluster-hkjsxx","namespace":"default"}, "namespace": "default", "name": "cluster-hkjsxx", "reconcileID": "9ba3420f-7e31-4ccd-bbae-b5acd21fe7ff", "cluster": "default/cluster-hkjsxx"}
2023-06-26T13:19:09.230Z	ERROR	UPDATE *v1.PersistentVolumeClaim error	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"cluster-hkjsxx","namespace":"default"}, "namespace": "default", "name": "cluster-hkjsxx", "reconcileID": "9ba3420f-7e31-4ccd-bbae-b5acd21fe7ff", "cluster": "default/cluster-hkjsxx", "error": "PersistentVolumeClaim \"data-cluster-hkjsxx-postgresql-2\" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests for bound claims\n  core.PersistentVolumeClaimSpec{\n  \tAccessModes: {\"ReadWriteOnce\"},\n  \tSelector:    nil,\n  \tResources: core.ResourceRequirements{\n  \t\tLimits: nil,\n- \t\tRequests: core.ResourceList{\n- \t\t\ts\"storage\": {i: resource.int64Amount{value: 11811160064}, s: \"11Gi\", Format: \"BinarySI\"},\n- \t\t},\n+ \t\tRequests: core.ResourceList{\n+ \t\t\ts\"storage\": {i: resource.int64Amount{value: 16106127360}, s: \"15Gi\", Format: \"BinarySI\"},\n+ \t\t},\n  \t},\n  \tVolumeName:       \"\",\n  \tStorageClassName: &\"kb-default-sc\",\n  \t... // 3 identical fields\n  }\n"}
2023-06-26T13:19:09.251Z	INFO	PersistentVolumeClaim "data-cluster-hkjsxx-postgresql-2" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests for bound claims
	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"cluster-hkjsxx","namespace":"default"}, "namespace": "default", "name": "cluster-hkjsxx", "reconcileID": "9ba3420f-7e31-4ccd-bbae-b5acd21fe7ff", "cluster": "default/cluster-hkjsxx"}
2023-06-26T13:19:09.251Z	ERROR	Reconciler error	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"cluster-hkjsxx","namespace":"default"}, "namespace": "default", "name": "cluster-hkjsxx", "reconcileID": "9ba3420f-7e31-4ccd-bbae-b5acd21fe7ff", "error": "PersistentVolumeClaim \"data-cluster-hkjsxx-postgresql-2\" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests for bound claims\n  core.PersistentVolumeClaimSpec{\n  \tAccessModes: {\"ReadWriteOnce\"},\n  \tSelector:    nil,\n  \tResources: core.ResourceRequirements{\n  \t\tLimits: nil,\n- \t\tRequests: core.ResourceList{\n- \t\t\ts\"storage\": {i: resource.int64Amount{value: 11811160064}, s: \"11Gi\", Format: \"BinarySI\"},\n- \t\t},\n+ \t\tRequests: core.ResourceList{\n+ \t\t\ts\"storage\": {i: resource.int64Amount{value: 16106127360}, s: \"15Gi\", Format: \"BinarySI\"},\n+ \t\t},\n  \t},\n  \tVolumeName:       \"\",\n  \tStorageClassName: &\"kb-default-sc\",\n  \t... // 3 identical fields\n  }\n"}
2023-06-26T13:19:09.433Z	INFO	update cluster status after applying resources 	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"cluster-hkjsxx","namespace":"default"}, "namespace": "default", "name": "cluster-hkjsxx", "reconcileID": "8fcd3928-c2b8-45b5-a753-54f895f09973", "cluster": "default/cluster-hkjsxx"}
2023-06-26T13:19:09.508Z	ERROR	UPDATE *v1.PersistentVolumeClaim error	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"cluster-hkjsxx","namespace":"default"}, "namespace": "default", "name": "cluster-hkjsxx", "reconcileID": "8fcd3928-c2b8-45b5-a753-54f895f09973", "cluster": "default/cluster-hkjsxx", "error": "PersistentVolumeClaim \"data-cluster-hkjsxx-postgresql-2\" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests for bound claims\n  core.PersistentVolumeClaimSpec{\n  \tAccessModes: {\"ReadWriteOnce\"},\n  \tSelector:    nil,\n  \tResources: core.ResourceRequirements{\n  \t\tLimits: nil,\n- \t\tRequests: core.ResourceList{\n- \t\t\ts\"storage\": {i: resource.int64Amount{value: 11811160064}, s: \"11Gi\", Format: \"BinarySI\"},\n- \t\t},\n+ \t\tRequests: core.ResourceList{\n+ \t\t\ts\"storage\": {i: resource.int64Amount{value: 16106127360}, s: \"15Gi\", Format: \"BinarySI\"},\n+ \t\t},\n  \t},\n  \tVolumeName:       \"\",\n  \tStorageClassName: &\"kb-default-sc\",\n  \t... // 3 identical fields\n  }\n"}
2023-06-26T13:19:09.519Z	INFO	PersistentVolumeClaim "data-cluster-hkjsxx-postgresql-2" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests for bound claims
	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"cluster-hkjsxx","namespace":"default"}, "namespace": "default", "name": "cluster-hkjsxx", "reconcileID": "8fcd3928-c2b8-45b5-a753-54f895f09973", "cluster": "default/cluster-hkjsxx"}
2023-06-26T13:19:09.519Z	ERROR	Reconciler error	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"cluster-hkjsxx","namespace":"default"}, "namespace": "default", "name": "cluster-hkjsxx", "reconcileID": "8fcd3928-c2b8-45b5-a753-54f895f09973", "error": "PersistentVolumeClaim \"data-cluster-hkjsxx-postgresql-2\" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests for bound claims\n  core.PersistentVolumeClaimSpec{\n  \tAccessModes: {\"ReadWriteOnce\"},\n  \tSelector:    nil,\n  \tResources: core.ResourceRequirements{\n  \t\tLimits: nil,\n- \t\tRequests: core.ResourceList{\n- \t\t\ts\"storage\": {i: resource.int64Amount{value: 11811160064}, s: \"11Gi\", Format: \"BinarySI\"},\n- \t\t},\n+ \t\tRequests: core.ResourceList{\n+ \t\t\ts\"storage\": {i: resource.int64Amount{value: 16106127360}, s: \"15Gi\", Format: \"BinarySI\"},\n+ \t\t},\n  \t},\n  \tVolumeName:       \"\",\n  \tStorageClassName: &\"kb-default-sc\",\n  \t... // 3 identical fields\n  }\n"}
2023-06-26T13:19:10.033Z	INFO	update cluster status after applying resources 	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"cluster-hkjsxx","namespace":"default"}, "namespace": "default", "name": "cluster-hkjsxx", "reconcileID": "8dfae461-530a-4c8f-8970-799664a8cb16", "cluster": "default/cluster-hkjsxx"}
2023-06-26T13:19:10.099Z	ERROR	UPDATE *v1.PersistentVolumeClaim error	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"cluster-hkjsxx","namespace":"default"}, "namespace": "default", "name": "cluster-hkjsxx", "reconcileID": "8dfae461-530a-4c8f-8970-799664a8cb16", "cluster": "default/cluster-hkjsxx", "error": "PersistentVolumeClaim \"data-cluster-hkjsxx-postgresql-2\" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests for bound claims\n  core.PersistentVolumeClaimSpec{\n  \tAccessModes: {\"ReadWriteOnce\"},\n  \tSelector:    nil,\n  \tResources: core.ResourceRequirements{\n  \t\tLimits: nil,\n- \t\tRequests: core.ResourceList{\n- \t\t\ts\"storage\": {i: resource.int64Amount{value: 11811160064}, s: \"11Gi\", Format: \"BinarySI\"},\n- \t\t},\n+ \t\tRequests: core.ResourceList{\n+ \t\t\ts\"storage\": {i: resource.int64Amount{value: 16106127360}, s: \"15Gi\", Format: \"BinarySI\"},\n+ \t\t},\n  \t},\n  \tVolumeName:       \"\",\n  \tStorageClassName: &\"kb-default-sc\",\n  \t... // 3 identical fields\n  }\n"}
2023-06-26T13:19:10.110Z	INFO	PersistentVolumeClaim "data-cluster-hkjsxx-postgresql-2" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests for bound claims
	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"cluster-hkjsxx","namespace":"default"}, "namespace": "default", "name": "cluster-hkjsxx", "reconcileID": "8dfae461-530a-4c8f-8970-799664a8cb16", "cluster": "default/cluster-hkjsxx"}
2023-06-26T13:19:10.110Z	ERROR	Reconciler error	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"cluster-hkjsxx","namespace":"default"}, "namespace": "default", "name": "cluster-hkjsxx", "reconcileID": "8dfae461-530a-4c8f-8970-799664a8cb16", "error": "PersistentVolumeClaim \"data-cluster-hkjsxx-postgresql-2\" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests for bound claims\n  core.PersistentVolumeClaimSpec{\n  \tAccessModes: {\"ReadWriteOnce\"},\n  \tSelector:    nil,\n  \tResources: core.ResourceRequirements{\n  \t\tLimits: nil,\n- \t\tRequests: core.ResourceList{\n- \t\t\ts\"storage\": {i: resource.int64Amount{value: 11811160064}, s: \"11Gi\", Format: \"BinarySI\"},\n- \t\t},\n+ \t\tRequests: core.ResourceList{\n+ \t\t\ts\"storage\": {i: resource.int64Amount{value: 16106127360}, s: \"15Gi\", Format: \"BinarySI\"},\n+ \t\t},\n  \t},\n  \tVolumeName:       \"\",\n  \tStorageClassName: &\"kb-default-sc\",\n  \t... // 3 identical fields\n  }\n"}
2023-06-26T13:19:10.638Z	INFO	update cluster status after applying resources 	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"cluster-hkjsxx","namespace":"default"}, "namespace": "default", "name": "cluster-hkjsxx", "reconcileID": "acee102e-b065-4dbd-8eb1-62d50bbc754d", "cluster": "default/cluster-hkjsxx"}
2023-06-26T13:19:10.675Z	ERROR	UPDATE *v1.PersistentVolumeClaim error	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"cluster-hkjsxx","namespace":"default"}, "namespace": "default", "name": "cluster-hkjsxx", "reconcileID": "acee102e-b065-4dbd-8eb1-62d50bbc754d", "cluster": "default/cluster-hkjsxx", "error": "PersistentVolumeClaim \"data-cluster-hkjsxx-postgresql-2\" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests for bound claims\n  core.PersistentVolumeClaimSpec{\n  \tAccessModes: {\"ReadWriteOnce\"},\n  \tSelector:    nil,\n  \tResources: core.ResourceRequirements{\n  \t\tLimits: nil,\n- \t\tRequests: core.ResourceList{\n- \t\t\ts\"storage\": {i: resource.int64Amount{value: 11811160064}, s: \"11Gi\", Format: \"BinarySI\"},\n- \t\t},\n+ \t\tRequests: core.ResourceList{\n+ \t\t\ts\"storage\": {i: resource.int64Amount{value: 16106127360}, s: \"15Gi\", Format: \"BinarySI\"},\n+ \t\t},\n  \t},\n  \tVolumeName:       \"\",\n  \tStorageClassName: &\"kb-default-sc\",\n  \t... // 3 identical fields\n  }\n"}
2023-06-26T13:19:10.690Z	INFO	PersistentVolumeClaim "data-cluster-hkjsxx-postgresql-2" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests for bound claims
	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"cluster-hkjsxx","namespace":"default"}, "namespace": "default", "name": "cluster-hkjsxx", "reconcileID": "acee102e-b065-4dbd-8eb1-62d50bbc754d", "cluster": "default/cluster-hkjsxx"}
2023-06-26T13:19:10.690Z	ERROR	Reconciler error	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"cluster-hkjsxx","namespace":"default"}, "namespace": "default", "name": "cluster-hkjsxx", "reconcileID": "acee102e-b065-4dbd-8eb1-62d50bbc754d", "error": "PersistentVolumeClaim \"data-cluster-hkjsxx-postgresql-2\" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests for bound claims\n  core.PersistentVolumeClaimSpec{\n  \tAccessModes: {\"ReadWriteOnce\"},\n  \tSelector:    nil,\n  \tResources: core.ResourceRequirements{\n  \t\tLimits: nil,\n- \t\tRequests: core.ResourceList{\n- \t\t\ts\"storage\": {i: resource.int64Amount{value: 11811160064}, s: \"11Gi\", Format: \"BinarySI\"},\n- \t\t},\n+ \t\tRequests: core.ResourceList{\n+ \t\t\ts\"storage\": {i: resource.int64Amount{value: 16106127360}, s: \"15Gi\", Format: \"BinarySI\"},\n+ \t\t},\n  \t},\n  \tVolumeName:       \"\",\n  \tStorageClassName: &\"kb-default-sc\",\n  \t... // 3 identical fields\n  }\n"}
2023-06-26T13:19:11.188Z	INFO	update cluster status after applying resources 	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"cluster-hkjsxx","namespace":"default"}, "namespace": "default", "name": "cluster-hkjsxx", "reconcileID": "01bf8c3a-9f0e-43d4-a74e-adeab68feb45", "cluster": "default/cluster-hkjsxx"}
2023-06-26T13:19:11.231Z	ERROR	UPDATE *v1.PersistentVolumeClaim error	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"cluster-hkjsxx","namespace":"default"}, "namespace": "default", "name": "cluster-hkjsxx", "reconcileID": "01bf8c3a-9f0e-43d4-a74e-adeab68feb45", "cluster": "default/cluster-hkjsxx", "error": "PersistentVolumeClaim \"data-cluster-hkjsxx-postgresql-2\" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests for bound claims\n  core.PersistentVolumeClaimSpec{\n  \tAccessModes: {\"ReadWriteOnce\"},\n  \tSelector:    nil,\n  \tResources: core.ResourceRequirements{\n  \t\tLimits: nil,\n- \t\tRequests: core.ResourceList{\n- \t\t\ts\"storage\": {i: resource.int64Amount{value: 11811160064}, s: \"11Gi\", Format: \"BinarySI\"},\n- \t\t},\n+ \t\tRequests: core.ResourceList{\n+ \t\t\ts\"storage\": {i: resource.int64Amount{value: 16106127360}, s: \"15Gi\", Format: \"BinarySI\"},\n+ \t\t},\n  \t},\n  \tVolumeName:       \"\",\n  \tStorageClassName: &\"kb-default-sc\",\n  \t... // 3 identical fields\n  }\n"}
2023-06-26T13:19:11.243Z	INFO	PersistentVolumeClaim "data-cluster-hkjsxx-postgresql-2" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests for bound claims
	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"cluster-hkjsxx","namespace":"default"}, "namespace": "default", "name": "cluster-hkjsxx", "reconcileID": "01bf8c3a-9f0e-43d4-a74e-adeab68feb45", "cluster": "default/cluster-hkjsxx"}
2023-06-26T13:19:11.243Z	ERROR	Reconciler error	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"cluster-hkjsxx","namespace":"default"}, "namespace": "default", "name": "cluster-hkjsxx", "reconcileID": "01bf8c3a-9f0e-43d4-a74e-adeab68feb45", "error": "PersistentVolumeClaim \"data-cluster-hkjsxx-postgresql-2\" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests for bound claims\n  core.PersistentVolumeClaimSpec{\n  \tAccessModes: {\"ReadWriteOnce\"},\n  \tSelector:    nil,\n  \tResources: core.ResourceRequirements{\n  \t\tLimits: nil,\n- \t\tRequests: core.ResourceList{\n- \t\t\ts\"storage\": {i: resource.int64Amount{value: 11811160064}, s: \"11Gi\", Format: \"BinarySI\"},\n- \t\t},\n+ \t\tRequests: core.ResourceList{\n+ \t\t\ts\"storage\": {i: resource.int64Amount{value: 16106127360}, s: \"15Gi\", Format: \"BinarySI\"},\n+ \t\t},\n  \t},\n  \tVolumeName:       \"\",\n  \tStorageClassName: &\"kb-default-sc\",\n  \t... // 3 identical fields\n  }\n"}
2023-06-26T13:19:11.748Z	INFO	update cluster status after applying resources 	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"cluster-hkjsxx","namespace":"default"}, "namespace": "default", "name": "cluster-hkjsxx", "reconcileID": "820dcfc4-728f-4302-a0e7-c541d8c07a6a", "cluster": "default/cluster-hkjsxx"}
2023-06-26T13:19:33.139Z	INFO	replicationSet new role is invalid, please check	{"controller": "event", "controllerGroup": "", "controllerKind": "Event", "Event": {"name":"cluster-hkjsxx-postgresql-2.qpnnvvr2jfzq8l7t","namespace":"default"}, "namespace": "default", "name": "cluster-hkjsxx-postgresql-2.qpnnvvr2jfzq8l7t", "reconcileID": "191354e4-3926-480c-a9bd-1aa51e21f78d", "event": "default/cluster-hkjsxx-postgresql-2.qpnnvvr2jfzq8l7t", "new role": ""}
2023-06-26T13:19:54.138Z	INFO	pod current role label equals to new role, ignore it	{"controller": "event", "controllerGroup": "", "controllerKind": "Event", "Event": {"name":"cluster-hkjsxx-postgresql-2.k44f74c522zlxbdh","namespace":"default"}, "namespace": "default", "name": "cluster-hkjsxx-postgresql-2.k44f74c522zlxbdh", "reconcileID": "a960c66e-d641-41a6-b0c9-45a1ed8046b6", "event": "default/cluster-hkjsxx-postgresql-2.k44f74c522zlxbdh", "new role": "secondary"}
2023-06-26T13:20:00.093Z	INFO	eventhandler.EnqueueRequestForAncestor	cloud not find ancestor object	{"source object gvk": "/, Kind=", "name": "delete-backup-default-cluster-hkjsxx-20230626210552-kxdjd", "up to level": 2}
2023-06-26T13:20:00.094Z	INFO	eventhandler.EnqueueRequestForAncestor	cloud not find ancestor object	{"source object gvk": "/, Kind=", "name": "delete-backup-default-cluster-hkjsxx-20230626210552-kxdjd", "up to level": 2}
2023-06-26T13:20:00.100Z	INFO	eventhandler.EnqueueRequestForAncestor	cloud not find ancestor object	{"source object gvk": "/, Kind=", "name": "delete-backup-default-cluster-hkjsxx-20230626210552-kxdjd", "up to level": 2}
2023-06-26T13:20:04.129Z	INFO	replicationSet new role is invalid, please check	{"controller": "event", "controllerGroup": "", "controllerKind": "Event", "Event": {"name":"cluster-hkjsxx-postgresql-2.bxxmqw4ghz6jwj45","namespace":"default"}, "namespace": "default", "name": "cluster-hkjsxx-postgresql-2.bxxmqw4ghz6jwj45", "reconcileID": "37c886d5-85f0-4535-9a3c-fb9507fbf704", "event": "default/cluster-hkjsxx-postgresql-2.bxxmqw4ghz6jwj45", "new role": ""}
2023-06-26T13:20:04.544Z	INFO	update old primary pod to secondary success	{"controller": "event", "controllerGroup": "", "controllerKind": "Event", "Event": {"name":"cluster-hkjsxx-postgresql-1.hnncjx94pp8gkccp","namespace":"default"}, "namespace": "default", "name": "cluster-hkjsxx-postgresql-1.hnncjx94pp8gkccp", "reconcileID": "3033eefa-c513-4930-8ee7-a1312a94f363", "event": "default/cluster-hkjsxx-postgresql-1.hnncjx94pp8gkccp", "old primary podName": "cluster-hkjsxx-postgresql-0"}
2023-06-26T13:20:04.625Z	INFO	update secondary pod to primary success	{"controller": "event", "controllerGroup": "", "controllerKind": "Event", "Event": {"name":"cluster-hkjsxx-postgresql-1.hnncjx94pp8gkccp","namespace":"default"}, "namespace": "default", "name": "cluster-hkjsxx-postgresql-1.hnncjx94pp8gkccp", "reconcileID": "3033eefa-c513-4930-8ee7-a1312a94f363", "event": "default/cluster-hkjsxx-postgresql-1.hnncjx94pp8gkccp", "new primary podName": "cluster-hkjsxx-postgresql-1"}
2023-06-26T13:20:04.677Z	ERROR	UPDATE *v1.Pod error	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"cluster-hkjsxx","namespace":"default"}, "namespace": "default", "name": "cluster-hkjsxx", "reconcileID": "55eb7ed5-f7cf-404e-99da-e045fd2e3ebe", "cluster": "default/cluster-hkjsxx", "error": "Operation cannot be fulfilled on pods \"cluster-hkjsxx-postgresql-1\": the object has been modified; please apply your changes to the latest version and try again"}
2023-06-26T13:20:04.695Z	INFO	Operation cannot be fulfilled on pods "cluster-hkjsxx-postgresql-1": the object has been modified; please apply your changes to the latest version and try again	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"cluster-hkjsxx","namespace":"default"}, "namespace": "default", "name": "cluster-hkjsxx", "reconcileID": "55eb7ed5-f7cf-404e-99da-e045fd2e3ebe", "cluster": "default/cluster-hkjsxx"}
2023-06-26T13:20:04.695Z	ERROR	Reconciler error	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"cluster-hkjsxx","namespace":"default"}, "namespace": "default", "name": "cluster-hkjsxx", "reconcileID": "55eb7ed5-f7cf-404e-99da-e045fd2e3ebe", "error": "Operation cannot be fulfilled on pods \"cluster-hkjsxx-postgresql-1\": the object has been modified; please apply your changes to the latest version and try again"}
2023-06-26T13:20:05.221Z	ERROR	UPDATE *v1.Pod error	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"cluster-hkjsxx","namespace":"default"}, "namespace": "default", "name": "cluster-hkjsxx", "reconcileID": "736e1813-1f2f-4275-8682-24cbbfd5247f", "cluster": "default/cluster-hkjsxx", "error": "Operation cannot be fulfilled on pods \"cluster-hkjsxx-postgresql-2\": the object has been modified; please apply your changes to the latest version and try again"}
2023-06-26T13:20:05.236Z	INFO	Operation cannot be fulfilled on pods "cluster-hkjsxx-postgresql-2": the object has been modified; please apply your changes to the latest version and try again	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"cluster-hkjsxx","namespace":"default"}, "namespace": "default", "name": "cluster-hkjsxx", "reconcileID": "736e1813-1f2f-4275-8682-24cbbfd5247f", "cluster": "default/cluster-hkjsxx"}
2023-06-26T13:20:05.236Z	ERROR	Reconciler error	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"cluster-hkjsxx","namespace":"default"}, "namespace": "default", "name": "cluster-hkjsxx", "reconcileID": "736e1813-1f2f-4275-8682-24cbbfd5247f", "error": "Operation cannot be fulfilled on pods \"cluster-hkjsxx-postgresql-2\": the object has been modified; please apply your changes to the latest version and try again"}
2023-06-26T13:20:58.088Z	INFO	replicationSet new role is invalid, please check	{"controller": "event", "controllerGroup": "", "controllerKind": "Event", "Event": {"name":"cluster-hkjsxx-postgresql-0.nf28hv9pjfmzc8gd","namespace":"default"}, "namespace": "default", "name": "cluster-hkjsxx-postgresql-0.nf28hv9pjfmzc8gd", "reconcileID": "06d690dc-2acb-4a5c-a05d-9d69c92cad20", "event": "default/cluster-hkjsxx-postgresql-0.nf28hv9pjfmzc8gd", "new role": ""}
2023-06-26T13:21:24.880Z	INFO	replicationSet new role is invalid, please check	{"controller": "event", "controllerGroup": "", "controllerKind": "Event", "Event": {"name":"cluster-hkjsxx-postgresql-1.mjtqbjwpbm86nx2c","namespace":"default"}, "namespace": "default", "name": "cluster-hkjsxx-postgresql-1.mjtqbjwpbm86nx2c", "reconcileID": "f8f48b5f-822a-42c8-882d-8c7a43e9ef65", "event": "default/cluster-hkjsxx-postgresql-1.mjtqbjwpbm86nx2c", "new role": ""}
2023-06-26T13:21:25.724Z	INFO	replicationSet new role is invalid, please check	{"controller": "event", "controllerGroup": "", "controllerKind": "Event", "Event": {"name":"cluster-hkjsxx-postgresql-2.4vjk5rz9lnrg77qq","namespace":"default"}, "namespace": "default", "name": "cluster-hkjsxx-postgresql-2.4vjk5rz9lnrg77qq", "reconcileID": "9dc214c0-ab71-4cc0-abfa-4cd53169d4a2", "event": "default/cluster-hkjsxx-postgresql-2.4vjk5rz9lnrg77qq", "new role": ""}

Expected behavior
postgresql cluster restart success.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

  • OS: [e.g. iOS]
  • Browser [e.g. chrome, safari]
  • Version [e.g. 22]

Additional context
Add any other context about the problem here.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Y-Rookie Y-Rookie

Labels
bug kind/bug Something isn't working
Projects
None yet
Milestone
Release 0.6.0
Development

Successfully merging a pull request may close this issue.

fix: replicationSet parallel update strategy restart failed apecloud/kubeblocks
3 participants
@Y-Rookie @JashBook @apecloud-bot