Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] mysql cluster datafile hscale error: incompatible list #3965

Closed
JashBook opened this issue Jun 26, 2023 · 0 comments · Fixed by #4007
Closed

[BUG] mysql cluster datafile hscale error: incompatible list #3965

JashBook opened this issue Jun 26, 2023 · 0 comments · Fixed by #4007
Assignees
@lynnleelhl
Labels
bug kind/bug Something isn't working
Milestone
Release 0.6.0

Comments

@JashBook
Copy link
Collaborator

JashBook commented Jun 26, 2023 •
edited
Loading

Describe the bug
mysql cluster datafile hscale error: "error": "cue: marshal error: backupTool.spec.physical.restoreCommands: incompatible list lengths (1 and 3) (and 3 more errors)"}

kbcli version
Kubernetes: v1.25.10-eks-c12679a
KubeBlocks: 0.6.0-alpha.23
kbcli: v0.6.0-alpha.23

To Reproduce
Steps to reproduce the behavior:

  1. kbcli kubeblocks install --create-namespace --version 0.6.0-alpha.23 --set snapshot-controller.enabled=false
  2. create cluster
  3. hscale
  4. See error
kubectl get ops,pod,cluster,sts,pvc -l app.kubernetes.io/instance=cluster-ijdzwq
NAME                                                                   TYPE                CLUSTER          STATUS    PROGRESS   AGE
opsrequest.apps.kubeblocks.io/cluster-ijdzwq-horizontalscaling-mwqhp   HorizontalScaling   cluster-ijdzwq   Running   0/1        9m9s

NAME                         READY   STATUS    RESTARTS   AGE
pod/cluster-ijdzwq-mysql-0   4/4     Running   0          9m34s

NAME                                        CLUSTER-DEFINITION   VERSION           TERMINATION-POLICY   STATUS     AGE
cluster.apps.kubeblocks.io/cluster-ijdzwq   apecloud-mysql       ac-mysql-8.0.30   Delete               Updating   14m

NAME                                    READY   AGE
statefulset.apps/cluster-ijdzwq-mysql   1/1     14m

NAME                                                STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS    AGE
persistentvolumeclaim/data-cluster-ijdzwq-mysql-0   Bound    pvc-e78838c7-2399-473f-9362-c415456362ba   6Gi        RWO            kb-default-sc   14m

describe cluster

 kubectl describe cluster cluster-ijdzwq
Name:         cluster-ijdzwq
Namespace:    default
Labels:       app.kubernetes.io/instance=cluster-ijdzwq
              clusterdefinition.kubeblocks.io/name=apecloud-mysql
              clusterversion.kubeblocks.io/name=ac-mysql-8.0.30
Annotations:  kubeblocks.io/ops-request: [{"name":"cluster-ijdzwq-horizontalscaling-mwqhp","type":"HorizontalScaling"}]
API Version:  apps.kubeblocks.io/v1alpha1
Kind:         Cluster
Metadata:
  Creation Timestamp:  2023-06-26T13:08:20Z
  Finalizers:
    cluster.kubeblocks.io/finalizer
  Generation:        8
  Resource Version:  109996
  UID:               98908893-7589-433f-969b-2c5b3418e42f
Spec:
  Affinity:
    Pod Anti Affinity:     Preferred
    Tenancy:               SharedNode
  Cluster Definition Ref:  apecloud-mysql
  Cluster Version Ref:     ac-mysql-8.0.30
  Component Specs:
    Class Def Ref:
      Class:            custom-0.2c0.6g
    Component Def Ref:  mysql
    Monitor:            true
    Name:               mysql
    No Create PDB:      false
    Replicas:           2
    Resources:
      Limits:
        Cpu:     200m
        Memory:  644245094400m
      Requests:
        Cpu:               200m
        Memory:            644245094400m
    Service Account Name:  kb-sa-cluster-ijdzwq
    Volume Claim Templates:
      Name:  data
      Spec:
        Access Modes:
          ReadWriteOnce
        Resources:
          Requests:
            Storage:         6Gi
        Storage Class Name:  kb-default-sc
  Termination Policy:        Delete
Status:
  Cluster Def Generation:  2
  Components:
    Mysql:
      Consensus Set Status:
        Leader:
          Access Mode:  ReadWrite
          Name:         leader
          Pod:          cluster-ijdzwq-mysql-0
      Phase:            Updating
      Pods Ready:       false
  Conditions:
    Last Transition Time:  2023-06-26T13:13:59Z
    Message:               HorizontalScaling opsRequest: cluster-ijdzwq-horizontalscaling-mwqhp is processing
    Reason:                HorizontalScaling
    Status:                False
    Type:                  LatestOpsRequestProcessed
    Last Transition Time:  2023-06-26T13:08:20Z
    Message:               The operator has started the provisioning of Cluster: cluster-ijdzwq
    Observed Generation:   8
    Reason:                PreCheckSucceed
    Status:                True
    Type:                  ProvisioningStarted
    Last Transition Time:  2023-06-26T13:14:38Z
    Message:               cue: marshal error: backupTool.spec.physical.restoreCommands: incompatible list lengths (1 and 3) (and 3 more errors)
    Reason:                ApplyResourcesFailed
    Status:                False
    Type:                  ApplyResources
    Last Transition Time:  2023-06-26T13:14:00Z
    Message:               pods are not ready in Components: [mysql], refer to related component message in Cluster.status.components
    Reason:                ReplicasNotReady
    Status:                False
    Type:                  ReplicasReady
    Last Transition Time:  2023-06-26T13:14:00Z
    Message:               pods are unavailable in Components: [mysql], refer to related component message in Cluster.status.components
    Reason:                ComponentsNotReady
    Status:                False
    Type:                  Ready
  Observed Generation:     8
  Phase:                   Updating
Events:
  Type     Reason                    Age                  From                       Message
  ----     ------                    ----                 ----                       -------
  Normal   ComponentPhaseTransition  15m                  cluster-controller         Create a new component
  Normal   AllReplicasReady          14m                  cluster-controller         all pods of components are ready, waiting for the probe detection successful
  Normal   WaitingForProbeSuccess    14m (x8 over 14m)    cluster-controller         Waiting for probe success
  Normal   ComponentPhaseTransition  13m                  cluster-controller         Running: false, PodsReady: true, PodsTimedout: false
  Warning  Failed                    13m                  cluster-controller         Cluster: cluster-ijdzwq is Failed, check according to the components message
  Normal   Running                   13m                  cluster-controller         Cluster: cluster-ijdzwq is ready, current phase is Running
  Normal   ClusterReady              13m                  cluster-controller         Cluster: cluster-ijdzwq is ready, current phase is Running
  Normal   SysAcctCreate             13m                  system-account-controller  Created accounts for cluster: cluster-ijdzwq, component: mysql, accounts: kbprobe
  Normal   SysAcctCreate             13m                  system-account-controller  Created accounts for cluster: cluster-ijdzwq, component: mysql, accounts: kbadmin
  Normal   SysAcctCreate             13m                  system-account-controller  Created accounts for cluster: cluster-ijdzwq, component: mysql, accounts: kbmonitoring
  Normal   SysAcctCreate             13m                  system-account-controller  Created accounts for cluster: cluster-ijdzwq, component: mysql, accounts: kbdataprotection
  Normal   SysAcctCreate             13m                  system-account-controller  Created accounts for cluster: cluster-ijdzwq, component: mysql, accounts: kbreplicator
  Normal   ComponentPhaseTransition  12m (x2 over 15m)    cluster-controller         Component workload updated
  Normal   VerticalScaling           12m                  ops-request-controller     Start to process the VerticalScaling opsRequest "cluster-ijdzwq-verticalscaling-65gnv" in Cluster: cluster-ijdzwq
  Normal   PreCheckSucceed           12m (x4 over 15m)    cluster-controller         The operator has started the provisioning of Cluster: cluster-ijdzwq
  Normal   ApplyResourcesSucceed     12m (x4 over 15m)    cluster-controller         Successfully applied for resources
  Normal   VolumeExpansion           11m                  ops-request-controller     Start to process the VolumeExpansion opsRequest "cluster-ijdzwq-volumeexpansion-jqchm" in Cluster: cluster-ijdzwq
  Normal   ComponentPhaseTransition  10m (x4 over 13m)    cluster-controller         Running: true, PodsReady: true, PodsTimedout: false
  Normal   Restart                   10m                  ops-request-controller     Start to process the Restart opsRequest "cluster-ijdzwq-restart-tqjz6" in Cluster: cluster-ijdzwq
  Normal   HorizontalScaling         9m39s                ops-request-controller     Start to process the HorizontalScaling opsRequest "cluster-ijdzwq-horizontalscaling-mwqhp" in Cluster: cluster-ijdzwq
  Warning  ApplyResourcesFailed      3m30s (x18 over 9m)  cluster-controller         cue: marshal error: backupTool.spec.physical.restoreCommands: incompatible list lengths (1 and 3) (and 3 more errors)

logs kubeblocks

2023-06-26T13:20:08.974Z	INFO	cue: marshal error: backupTool.spec.physical.restoreCommands: incompatible list lengths (1 and 3) (and 3 more errors)	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"cluster-ijdzwq","namespace":"default"}, "namespace": "default", "name": "cluster-ijdzwq", "reconcileID": "835af12c-e2d2-4790-a02b-187ba7df2857", "cluster": "default/cluster-ijdzwq"}
2023-06-26T13:20:08.974Z	ERROR	Reconciler error	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"cluster-ijdzwq","namespace":"default"}, "namespace": "default", "name": "cluster-ijdzwq", "reconcileID": "835af12c-e2d2-4790-a02b-187ba7df2857", "error": "cue: marshal error: backupTool.spec.physical.restoreCommands: incompatible list lengths (1 and 3) (and 3 more errors)"}
2023-06-26T13:25:37.040Z	INFO	cue: marshal error: backupTool.spec.physical.restoreCommands: incompatible list lengths (1 and 3) (and 3 more errors)	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"cluster-ijdzwq","namespace":"default"}, "namespace": "default", "name": "cluster-ijdzwq", "reconcileID": "8297be90-ed73-4d7f-aa47-3f60c5b8cc5b", "cluster": "default/cluster-ijdzwq"}
2023-06-26T13:25:37.040Z	ERROR	Reconciler error	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"cluster-ijdzwq","namespace":"default"}, "namespace": "default", "name": "cluster-ijdzwq", "reconcileID": "8297be90-ed73-4d7f-aa47-3f60c5b8cc5b", "error": "cue: marshal error: backupTool.spec.physical.restoreCommands: incompatible list lengths (1 and 3) (and 3 more errors)"}

Expected behavior
mysql cluster datafile hscale success.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

  • OS: [e.g. iOS]
  • Browser [e.g. chrome, safari]
  • Version [e.g. 22]

Additional context
Add any other context about the problem here.
@JashBook JashBook added the kind/bug Something isn't working label Jun 26, 2023
@JashBook JashBook added this to the Release 0.6.0 milestone Jun 26, 2023
@lynnleelhl lynnleelhl linked a pull request Jun 28, 2023 that will close this issue
fix: incompatible length if commands more than 1 #4007
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@lynnleelhl lynnleelhl

Labels
bug kind/bug Something isn't working
Projects
None yet
Milestone
Release 0.6.0
Development

Successfully merging a pull request may close this issue.

fix: incompatible length if commands more than 1 apecloud/kubeblocks
3 participants
@lynnleelhl @JashBook @apecloud-bot