scalability on number of sources

[docteurklein]

Hello! Thanks for this amazing project, I love it!

Wasn't sure if I should have created an issue instead, but here we are (I'd happily create one instead if you prefer).

Imagine I have thousands of source (mysql) databases each containing ~200 tables.
Would Risingwave be able to handle one source per table? (aka 200 000 sources)

How would you then "union all" them automatically, as new sources (for new databases) appear?
I'm trying to handle a multi-tenant app, where data has been manually sharded per database.

PS: One solution I see would be to not use the mysql-cdc connector but instead let debezium do the fusion upstream, but that looks complicated.

[fuyufjh]

Hi, @docteurklein Thanks for the feedback! I suppose you are using some kind of MySQL sharding, right? May I ask which sharding middleware or system you are using now? Technically it's not difficult, but to make it usable, RisingWave has to understand the metadata of various sharding middleware, which might need some work.

For now, yes as you mentioned , you can deploy Debezium and let it stream out changes into one Kafka topic, and then let RisingWave consume from the Kafka topic

[docteurklein]

hi! We're not using any middleware like vitess or something, we're just spawning one mysql server every N tenants, each tenant getting 1 database.

I also think using debezium to merge into one topic might be the better solution, but out of curiosity, how many sources do you think a RW cluster can hold before getting a performance hit?