-
Notifications
You must be signed in to change notification settings - Fork 210
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[feat]Flink任务由于机房停电长时间宕机再次启动丢失数据 #224
Labels
Milestone
Comments
baisui1981
changed the title
Flink任务由于机房停电长时间宕机再次启动丢失数据
[feat]Flink任务由于机房停电长时间宕机再次启动丢失数据
May 16, 2023
baisui1981
added a commit
to qlangtech/plugins
that referenced
this issue
Oct 4, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
停电长时间宕机,任务停止之前没有有效的Savepoint 导致消费的MySQL binlog偏移量丢失,Flink任务恢复再次启动从最新的位点开始消费,导致宕机这段时间内数据库的增量更新数据无法同步到下游目标节点中,
以至于需要重新构建全量数据,这样很浪费时间,业务方无法等待,需要快速恢复。
想到的办法是flink在恢复任务时候,利用之前的checkpoint非savepoint来恢复失败的任务,这样就能重新读取位点将补上未同步的历史数据
参考:https://blog.csdn.net/weixin_39977642/article/details/112103160
@喜楽 可协助测试
The text was updated successfully, but these errors were encountered: