Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add clickhouse-replicated dialect #809

Closed
wants to merge 2 commits into from

Conversation

chapsuk
Copy link
Contributor

@chapsuk chapsuk commented Aug 20, 2024

This PR introduces a new clickhouse-replicated dialect, and here’s why it’s needed:

  • The ReplicatedMergeTree engine is required.
  • The ON CLUSTER clause is a must for CREATE TABLE queries.
  • You need to set insert_quorum=2 for inserts and select_sequential_consistency=1 for selects.

These differences between a single-node ClickHouse instance and a replicated cluster are why we’re adding this new dialect.

Added the environment variable GOOSE_CLICKHOUSE_CLUSTER_NAME (default: {cluster}) allows to set Clickhouse cluster name

@chapsuk chapsuk force-pushed the clickhouse-replicated branch 2 times, most recently from ee7131e to 8a0af1a Compare September 21, 2024 08:53
@chapsuk chapsuk force-pushed the clickhouse-replicated branch from 8a0af1a to cf87405 Compare September 21, 2024 08:53
date Date default now(),
tstamp DateTime default now()
)
ENGINE = ReplicatedMergeTree()

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would be awesome to be able to pass keeper path here as well, like

ENGINE = ReplicatedMergeTree('/clickhouse/tables/common/migrations', '{replica}')

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's an endless number of ways to configure clickhouse, which means goose will need to know about all these and expose knobs for them.

I think the most practical solution would be for projects to implement the Store interface, e.g.,

#520 (comment)

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe an alternative would be to expose configuration options via a .yaml file. I think what I'm getting at is it's not sustainable to maintain dozens of custom clickhouse dialects.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Totally agree! The Store interface looks much better

@chapsuk chapsuk closed this Nov 18, 2024
@vdksystem
Copy link

vdksystem commented Nov 20, 2024

@yakovlevmichaelv are you planning on implementing store interface?
it's almost a year since someone tried to add real support for clickhouse and we still don't have it.
I think @mfridman will agree that when we talk about clickhouse and db migrations it's 99.99% about production, where there is almost no sense to introduce analytical highly scalable database with single node version.
So for now goose clickhouse support is only for dev environments.

For now I manually initialize version tables (in my case multiple)

CREATE TABLE goose_db_version on cluster analytics
(
    `version_id` Int64,
    `is_applied` UInt8,
    `date` Date DEFAULT now(),
    `tstamp` DateTime DEFAULT now()
)
ENGINE = KeeperMap('/goose_db_version_edi_service')
PRIMARY KEY version_id;
insert into goose_db_version values(0, 1, toDate(now()), now())

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants