Skip to content

Latest commit

 

History

History
41 lines (30 loc) · 1.02 KB

sync.md

File metadata and controls

41 lines (30 loc) · 1.02 KB

Sync CDC data

Subscribe to data changes in the source database and sync them to the target.

Prerequisites

  • MySQL: Enables binlog in the source database;
  • PG: Sets wal_level = logical in the source database;
  • Mongo: The source instance must be ReplicaSet;
  • For more information, refer to init test env.

Example: MySQL -> MySQL

Refer to task templates and tutorial

Parallelizer

  • MySQL/PG: parallel_type=rdb_merge
  • Mongo: parallel_type=mongo
  • Redis: parallel_type=redis

Other configurations

  • For [filter] and [router], refer to config details.

  • Refer to task_config.ini in tests:

    • dt-tests/tests/mysql_to_mysql/cdc
    • dt-tests/tests/pg_to_pg/cdc
    • dt-tests/tests/mongo_to_mongo/cdc
    • dt-tests/tests/redis_to_redis/cdc

  • Modify performance parameters if needed:

[pipeline]
buffer_size=16000
checkpoint_interval_secs=10

[sinker]
batch_size=200

[parallelizer]
parallel_size=8