Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[feat]Flink任务由于机房停电长时间宕机再次启动丢失数据 #224

Closed
baisui1981 opened this issue May 11, 2023 · 1 comment
Labels
3.8.0 enhancement New feature or request
Milestone

Comments

@baisui1981
Copy link
Member

baisui1981 commented May 11, 2023

停电长时间宕机,任务停止之前没有有效的Savepoint 导致消费的MySQL binlog偏移量丢失,Flink任务恢复再次启动从最新的位点开始消费,导致宕机这段时间内数据库的增量更新数据无法同步到下游目标节点中,

以至于需要重新构建全量数据,这样很浪费时间,业务方无法等待,需要快速恢复。

想到的办法是flink在恢复任务时候,利用之前的checkpoint非savepoint来恢复失败的任务,这样就能重新读取位点将补上未同步的历史数据

参考:https://blog.csdn.net/weixin_39977642/article/details/112103160
@喜楽 可协助测试

@baisui1981 baisui1981 added enhancement New feature or request 3.8.0 labels May 11, 2023
@baisui1981 baisui1981 added this to the v3.8.0 milestone May 11, 2023
@baisui1981 baisui1981 changed the title Flink任务由于机房停电长时间宕机再次启动丢失数据 [feat]Flink任务由于机房停电长时间宕机再次启动丢失数据 May 16, 2023
@baisui1981
Copy link
Member Author

baisui1981 commented May 17, 2023

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
3.8.0 enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant