Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Transformer中使用上下文绑定参数 #342

Closed
baisui1981 opened this issue Aug 2, 2024 · 0 comments
Closed

Transformer中使用上下文绑定参数 #342

baisui1981 opened this issue Aug 2, 2024 · 0 comments
Labels
enhancement New feature or request
Milestone
V4.0.1

Comments

@baisui1981
Copy link
Member

baisui1981 commented Aug 2, 2024 •
edited
Loading

问题

在MySQL 同步到目标库 starrock或者doris 库中时，MySQL中采用分库策略存储海量数据，在MySQL库中有一张A表需要同步到下游Doris库中，由于A表的主键采用自增bigint为主键，导致多个分库中的表的主键汇聚到同一个doris表时，存在主键冲突的问题。

解决策略

在执行数据通道的上下文张绑定上，向下文相关参数，例如：$dbName，$tableName 可供用户选择，结合concat transformer算子将 A表Id于上线文参数连接生成新的主键替换原有Id，这样就不会有冲突了

微信：H2M1
@baisui1981 baisui1981 added this to the V4.0.1 milestone Aug 2, 2024
@baisui1981 baisui1981 added the enhancement New feature or request label Aug 3, 2024
baisui1981 added a commit that referenced this issue Aug 3, 2024
@baisui1981
add pipeline context bind params like 'dbName','tableName' for TIS,#342
ad70ebd
baisui1981 added a commit to qlangtech/plugins that referenced this issue Aug 3, 2024
@baisui1981
add pipeline context bind params like 'dbName','tableName' for TIS,da…
b046aba 
…tavane/tis#342
baisui1981 added a commit that referenced this issue Aug 11, 2024
@baisui1981
enable incr sync pipeline context param bind #342
e80962c
baisui1981 added a commit to qlangtech/DataX that referenced this issue Aug 11, 2024
@baisui1981
enable incr sync pipeline context param bind datavane/tis#342
2dd51bb
baisui1981 added a commit that referenced this issue Aug 11, 2024
@baisui1981
enable incr sync pipeline context param bind #342
98d14a4
baisui1981 added a commit to qlangtech/plugins that referenced this issue Aug 12, 2024
@baisui1981
enable incr synchronize for flink when using stream API codestyle ,fo…
fe6ff28 
…r issue datavane/tis#342
baisui1981 added a commit to qlangtech/plugins that referenced this issue Aug 13, 2024
@baisui1981
enable incr sync pipeline context param bind datavane/tis#342
16dc495
baisui1981 added a commit that referenced this issue Aug 13, 2024
@baisui1981
enable incr sync pipeline context param bind #342
e2f53a7
baisui1981 added a commit to qlangtech/plugins that referenced this issue Aug 13, 2024
@baisui1981
enable incr sync pipeline context param bind datavane/tis#342
fb3c15b
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
V4.0.1
Development

No branches or pull requests
1 participant
@baisui1981