Check failed: absl::OkStatus() == status (OK vs. INTERNAL: mysql_real_connect failed: errno: , error: [mysql-error-info='']) MetadataStore cannot be created with the given connection config.

[922tech]

Validation Checklist

• Is this a Kubeflow issue?
• Are you posting in the right repository ?
• Did you follow the Kubeflow installation guideline ?
• Is the issue report properly structured and detailed with version numbers?
• Is this for Kubeflow development ?
• Would you like to work on this issue?
• You can join the CNCF Slack and access our meetings at the Kubeflow Community website. Our channel on the CNCF Slack is here #kubeflow-platform.

Version

1.9

Describe your issue

I followed the instruction for installing kubeflow pipeline on version 1.9.1
After creation of the namespaces and pods:

NAME                                               READY   STATUS             RESTARTS         AGE
cache-deployer-deployment-55458bb9bf-d4vxf         1/1     Running            1 (16h ago)      21h
cache-server-568d98d56f-mxgck                      1/1     Running            34 (58s ago)     21h
metadata-envoy-deployment-b4bc46768-mmx8j          1/1     Running            1 (16h ago)      21h
metadata-grpc-deployment-65fc97b764-6h7k4          0/1     CrashLoopBackOff   99 (3m35s ago)   21h
metadata-writer-86b4945bd5-78tjx                   0/1     CrashLoopBackOff   57 (25s ago)     21h
minio-7c77bc59b8-sks9f                             1/1     Running            1 (16h ago)      21h
ml-pipeline-7f6bcb764-b9d4n                        0/1     Running            120 (6m8s ago)   21h
ml-pipeline-persistenceagent-d9c785cdb-hvxpj       0/1     CrashLoopBackOff  67 (4m56s ago)   21h
ml-pipeline-scheduledworkflow-6d5fc7cb6d-phftf     1/1     Running            1 (16h ago)      21h
ml-pipeline-ui-5785c9d69d-gf2mk                    1/1     Running            1 (107m ago)     21h
ml-pipeline-viewer-crd-5869ccf789-np2wx            1/1     Running            1 (16h ago)      21h
ml-pipeline-visualizationserver-65df54899f-9zzph   1/1     Running            1 (107m ago)     21h
mysql-758cd66576-4q56v                             1/1     Running            1 (16h ago)      21h
workflow-controller-5584bbc656-dlgmr               1/1     Running            2 (106m ago)     21h

As you see some pods do not run.
And I think the main cause of the problem is that metadata-grpc-deployment cannot start working and it crashes with this error log:

WARNING: Logging before InitGoogleLogging() is written to STDERR
F1105 08:28:41.229691     1 metadata_store_server_main.cc:236] Check failed: absl::OkStatus() == status (OK vs. INTERNAL: mysql_real_connect failed: errno: , error:  [mysql-error-info='']) MetadataStore cannot be created with the given connection config.
*** Check failure stack trace: ***

I don't know why on earth it cannot connect to mysql!

Steps to reproduce the issue

Follow the instructions for installation of pipeline.

Put here any screenshots or videos (optional)

No response

[juliusvonkohout]

Please try again with version 1.9.1 or the master branch and report back.

[922tech]

Please try again with version 1.9.1 or the master branch and report back.

I have already tried these several times.
Also deployed cert-manager but did not solve the problem

[juliusvonkohout]

Please follow the instructions and install according to the readme a full kubeflow platform on a kind cluster. If not I have to close or transfer the issue to Kubeflow/pipelines.

[LehmBook]

Is your pv and pvc bond correctly and whats your config of your sql pv?

[juliusvonkohout]

/lifecycle stale

[922tech]

Installing on kind cluster did solve the problem.
But is it possible to have KubeFlow on multiple nodes and multiple clusters in kind?!