Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[feat]需要定义数据同步可自定义表列映射 #225

Closed
baisui1981 opened this issue May 11, 2023 · 5 comments
Closed

[feat]需要定义数据同步可自定义表列映射 #225

baisui1981 opened this issue May 11, 2023 · 5 comments
Labels
enhancement New feature or request
Milestone

Comments

@baisui1981
Copy link
Member

baisui1981 commented May 11, 2023

设置批量同步通道源表->目标表的列是强一致(数量,名称是保持强一致的),但是一般场景下,用户希望自定义列映射关系,例如:

  1. 在目标表上添加额外列
  2. 将源表中的列设置别名到目标表上

功能实现可以利用DataX transform机制来实现 https://github.com/alibaba/DataX/blob/master/transformer/doc/transformer.md

原表选择
编辑

@baisui1981 baisui1981 added this to the v3.8.0 milestone May 11, 2023
@baisui1981 baisui1981 added the enhancement New feature or request label May 11, 2023
@baisui1981 baisui1981 modified the milestones: v3.8.0, v4.0.0 Jul 17, 2023
@piaoyaoi
Copy link

image
还没有发布吗

@Tandoy
Copy link

Tandoy commented Feb 28, 2024

希望自定义列映射关系,例如:

微信截图_20240228100528

@baisui1981
Copy link
Member Author

希望自定义列映射关系,例如:

微信截图_20240228100528

一般表字段名称,source端和sink 端保持一致的吧

@Tandoy
Copy link

Tandoy commented Feb 28, 2024

希望自定义列映射关系,例如:
微信截图_20240228100528

一般表字段名称,source端和sink 端保持一致的吧

如果目标表已经存在,有可能出现目标表字段名不一致的情况。并且还有些抽取部分字段的同步业务场景

@baisui1981 baisui1981 modified the milestones: v4.0.0, V4.1.0 Feb 28, 2024
@baisui1981 baisui1981 modified the milestones: V4.1.0, V4.0.1 Jun 1, 2024
@baisui1981
Copy link
Member Author

baisui1981 commented Aug 14, 2024

已经实现以下transformer算子:

名称 功能说明
Data Masking 用于用户数据脱敏,例如,业务系统私密数据需要导入到数仓,如,用户名、密码之类数据需要进行脱敏处理。处理方法比较简单,字段内容进行遮照处理,例如:原用户名为‘王五’处理后为‘王*’
Concat Fields 将原记录中的多个字段值,通过字符串连接的方式合并成一个记录值
JSON Splitter 将原有记录中具有JSON Object格式的字段,拆分成多个新的字段
Copy Field 将原有记录拷贝到一个新的冗余列上
SubStr 将原有字段取子字符串到新列或者替换原有字段

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants