Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add support for self-hosted ClickHouse #413

Merged
merged 23 commits into from
Jun 4, 2024
Merged

Add support for self-hosted ClickHouse #413

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
a5aeb8e
Add ClickHouse support
0x2b3bfa0 Jun 1, 2024
b952d61
Helm-Docs update
0x2b3bfa0 Jun 1, 2024
7628ac8
fixup! Add ClickHouse support
0x2b3bfa0 Jun 1, 2024
66f7c49
Helm-Docs update
0x2b3bfa0 Jun 1, 2024
819bbe5
Update values.yaml
0x2b3bfa0 Jun 1, 2024
8e7894b
Helm-Docs update
0x2b3bfa0 Jun 1, 2024
5de6232
Automate DSN generation
0x2b3bfa0 Jun 1, 2024
d3f27de
Helm-Docs update
0x2b3bfa0 Jun 1, 2024
0472419
fixup! Automate DSN generation
0x2b3bfa0 Jun 1, 2024
0dc4467
Update helm-lint-and-install.yaml
0x2b3bfa0 Jun 2, 2024
02a363a
Disable TLS
0x2b3bfa0 Jun 2, 2024
1f78544
Update values.yaml
0x2b3bfa0 Jun 4, 2024
ca259af
Helm-Docs update
0x2b3bfa0 Jun 4, 2024
8ec35bb
Update values.yaml
0x2b3bfa0 Jun 4, 2024
e3217de
Update values.yaml
0x2b3bfa0 Jun 4, 2024
de6a23c
Helm-Docs update
0x2b3bfa0 Jun 4, 2024
13299c7
Update secret-studio.yaml
0x2b3bfa0 Jun 4, 2024
3fd192c
Update helm-lint-and-install.yaml
0x2b3bfa0 Jun 4, 2024
1ca54df
Update secret-studio.yaml
0x2b3bfa0 Jun 4, 2024
0cc10e9
Update secret-studio.yaml
0x2b3bfa0 Jun 4, 2024
a48e44e
Update helm-lint-and-install.yaml
0x2b3bfa0 Jun 4, 2024
3be065f
Update values.yaml
0x2b3bfa0 Jun 4, 2024
aa1356b
Helm-Docs update
0x2b3bfa0 Jun 4, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/helm-lint-and-install.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ jobs:
if: steps.list-changed.outputs.changed == 'true'
run: |
ct lint-and-install --target-branch ${{ github.event.repository.default_branch }} --upgrade --skip-missing-values --debug \
--helm-extra-set-args '--set ci=true --set global.blobvault.persistentVolume.storageClassName="standard" --set imagePullSecrets[0].name=iterativeai --set dockerUsername=${{ vars.ITERATIVE_DOCKER_REGISTRY_USER }} --set dockerPassword=${{ secrets.ITERATIVE_DOCKER_REGISTRY_PASSWORD }} --set dockerServer=docker.iterative.ai'
--helm-extra-set-args '--set ci=true --set global.blobvault.persistentVolume.storageClassName="standard" --set clickhouse.auth.password="clickhouse" --set imagePullSecrets[0].name=iterativeai --set dockerUsername=${{ vars.ITERATIVE_DOCKER_REGISTRY_USER }} --set dockerPassword=${{ secrets.ITERATIVE_DOCKER_REGISTRY_PASSWORD }} --set dockerServer=docker.iterative.ai'
Copy link
Member Author

@0x2b3bfa0 0x2b3bfa0 Jun 2, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://github.com/iterative/helm-charts/actions/runs/9332063406/job/25690736127#step:10:117

Error: UPGRADE FAILED: execution error at (studio/charts/clickhouse/templates/secret.yaml:19:21): 
PASSWORDS ERROR: You must provide your current passwords when upgrading the release.
                 Note that even after reinstallation, old credentials may be needed as they may be kept in persistent volume claims.
                 Further information can be obtained at https://docs.bitnami.com/general/how-to/troubleshoot-helm-chart-issues/#credential-errors-while-upgrading-chart-releases

    'auth.password' must not be empty, please add '--set auth.***' to the command. To get the current value:

        export ADMIN_PASSWORD=$(kubectl get secret --namespace "studio-woa6hxx2un" studio-woa6hxx2un-clickhouse -o jsonpath="{.data.admin-password}" | base64 -d)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@iterative/platform, is there any better solution?


helm-docs:
runs-on: ubuntu-latest
Expand Down
7 changes: 5 additions & 2 deletions charts/studio/Chart.lock
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,5 +5,8 @@ dependencies:
- name: postgresql
repository: https://charts.bitnami.com/bitnami
version: 11.9.13
digest: sha256:4d64be3c68e02f9bf8d4468389b628383c33fc3e1902597f637c49ccf2f777d9
generated: "2024-04-30T15:57:25.19756706Z"
- name: clickhouse
repository: https://charts.bitnami.com/bitnami
version: 6.2.0
digest: sha256:5764f9714b4d0e478f5ea25796091919f7ca334773989ac86235c7377db85405
generated: "2024-06-01T16:58:57.386705962Z"
6 changes: 5 additions & 1 deletion charts/studio/Chart.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ apiVersion: v2
name: studio
description: A Helm chart for Kubernetes
type: application
version: 0.14.2
version: 0.14.3
appVersion: "v2.119.4"
maintainers:
- name: iterative
Expand All @@ -17,3 +17,7 @@ dependencies:
condition: postgresql.enabled
version: "11.9.13"
repository: "https://charts.bitnami.com/bitnami"
- name: clickhouse
condition: clickhouse.enabled
version: "6.2.0"
repository: "https://charts.bitnami.com/bitnami"
8 changes: 7 additions & 1 deletion charts/studio/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# studio

![Version: 0.14.2](https://img.shields.io/badge/Version-0.14.2-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: v2.119.4](https://img.shields.io/badge/AppVersion-v2.119.4-informational?style=flat-square)
![Version: 0.14.3](https://img.shields.io/badge/Version-0.14.3-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: v2.119.4](https://img.shields.io/badge/AppVersion-v2.119.4-informational?style=flat-square)

A Helm chart for Kubernetes

Expand All @@ -14,13 +14,19 @@ A Helm chart for Kubernetes

| Repository | Name | Version |
|------------|------|---------|
| https://charts.bitnami.com/bitnami | clickhouse | 6.2.0 |
| https://charts.bitnami.com/bitnami | postgresql | 11.9.13 |
| https://charts.bitnami.com/bitnami | redis | 17.14.3 |

## Values

| Key | Type | Default | Description |
|-----|------|---------|-------------|
| clickhouse.auth.password | string | `"clickhouse"` | ClickHouse password |
| clickhouse.enabled | bool | `false` | ClickHouse enabled |
| clickhouse.fullnameOverride | string | `"studio-clickhouse"` | ClickHouse name override |
| clickhouse.replicaCount | int | `1` | |
| clickhouse.shards | int | `1` | |
| global.basePath | string | `""` | Studio: Base path (prefix) |
| global.blobvault.accessKeyId | string | `""` | Blobvault S3 access key ID |
| global.blobvault.bucket | string | `""` | Blobvault S3 bucket name |
Expand Down
6 changes: 5 additions & 1 deletion charts/studio/templates/secret-studio.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,4 +72,8 @@ stringData:

{{- $dvcx := .Values.global.dvcx | default dict }}
{{- $dvcxClickhouse := $dvcx.clickHouse | default dict }}
DVCX_CH_DSN: {{ $dvcxClickhouse.dsn | default "" | quote }}
{{- if $dvcxClickhouse.dsn }}
DVCX_CH_DSN: {{ $dvcxClickhouse.dsn | quote }}
{{- else if .Values.clickhouse.enabled }}
DVCX_CH_DSN: clickhouse+native://{{ .Values.clickhouse.auth.username }}:{{ .Values.clickhouse.auth.password }}@{{ .Values.clickhouse.fullnameOverride }}.{{ .Release.Namespace }}.svc.cluster.local/default?secure=False
{{- end }}
15 changes: 15 additions & 0 deletions charts/studio/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -238,6 +238,21 @@ postgresql:
# -- Postgres database
database: iterativeai

clickhouse:
# -- ClickHouse enabled
enabled: false
# -- ClickHouse name override
fullnameOverride: studio-clickhouse

# Shards / replicas configuration
replicaCount: 1
shards: 1
Comment on lines +247 to +249
Copy link
Member Author

@0x2b3bfa0 0x2b3bfa0 Jun 4, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using more shards/replicas will cause eventual consistency issues with table creation; to be fixed whenever we have to scale.


# Change this before deploying
auth:
# -- ClickHouse password
password: "clickhouse"

# -- PgBouncer settings group
pgBouncer:
enabled: false
Expand Down
Loading