Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FLINK-29298] LocalBufferPool request buffer from NetworkBufferPool hanging. #20924

Closed
wants to merge 2 commits into from
Closed

[FLINK-29298] LocalBufferPool request buffer from NetworkBufferPool hanging. #20924

wants to merge 2 commits into from

Conversation

reswqa
Copy link
Member

@reswqa reswqa commented Sep 29, 2022

What is the purpose of the change

when the task thread polled out the last buffer in LocalBufferPool and triggered the onGlobalPoolAvailable callback itself, it will skip this notification (as currently the LocalBufferPool is available), which will cause the BufferPool to eventually become unavailable and will never register a callback to the NetworkBufferPool.

Brief change log

  • Move the callback register after availability check.

Verifying this change

This change added unit test.

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): no
  • The public API, i.e., is any changed class annotated with @Public(Evolving): no
  • The serializers: no
  • The runtime per-record code paths (performance sensitive): yes
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
  • The S3 file system connector: no

Documentation

  • Does this pull request introduce a new feature? no

@flinkbot
Copy link
Collaborator

flinkbot commented Sep 29, 2022 •
edited
Loading

CI report:

Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

@wsry wsry self-assigned this Oct 17, 2022
@reswqa
[FLINK-29298] LocalBufferPool request buffer from NetworkBufferPool h…
f29f9b0 
…anging.

when the task thread polled out the last buffer in LocalBufferPool and triggered the onGlobalPoolAvailable callback itself, it will skip this notification  (as currently the LocalBufferPool is available), which will cause the BufferPool to eventually become unavailable and will never register a callback to the NetworkBufferPool.
xintongsong
xintongsong approved these changes Dec 13, 2022
View reviewed changes
Copy link
Contributor

@xintongsong xintongsong left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

xintongsong pushed a commit that referenced this pull request Dec 13, 2022
@reswqa @xintongsong
[FLINK-29298] LocalBufferPool request buffer from NetworkBufferPool h…
40fbea3 
…anging.

when the task thread polled out the last buffer in LocalBufferPool and triggered the onGlobalPoolAvailable callback itself, it will skip this notification  (as currently the LocalBufferPool is available), which will cause the BufferPool to eventually become unavailable and will never register a callback to the NetworkBufferPool.

This closes #20924
chucheng92 pushed a commit to chucheng92/flink that referenced this pull request Feb 3, 2023
@reswqa @chucheng92
[FLINK-29298] LocalBufferPool request buffer from NetworkBufferPool h…
6375ec5 
…anging.

when the task thread polled out the last buffer in LocalBufferPool and triggered the onGlobalPoolAvailable callback itself, it will skip this notification  (as currently the LocalBufferPool is available), which will cause the BufferPool to eventually become unavailable and will never register a callback to the NetworkBufferPool.

This closes apache#20924
sergeitsar pushed a commit to fentik/flink that referenced this pull request Feb 8, 2023
@reswqa @sergeitsar
[FLINK-29298] LocalBufferPool request buffer from NetworkBufferPool h…
246e020 
…anging.

when the task thread polled out the last buffer in LocalBufferPool and triggered the onGlobalPoolAvailable callback itself, it will skip this notification  (as currently the LocalBufferPool is available), which will cause the BufferPool to eventually become unavailable and will never register a callback to the NetworkBufferPool.

This closes apache#20924
sergeitsar pushed a commit to fentik/flink that referenced this pull request Feb 8, 2023
@reswqa @sergeitsar
[FLINK-29298] LocalBufferPool request buffer from NetworkBufferPool h…
4bcc6ff 
…anging.

when the task thread polled out the last buffer in LocalBufferPool and triggered the onGlobalPoolAvailable callback itself, it will skip this notification  (as currently the LocalBufferPool is available), which will cause the BufferPool to eventually become unavailable and will never register a callback to the NetworkBufferPool.

This closes apache#20924
akkinenivijay pushed a commit to krisnaru/flink that referenced this pull request Feb 11, 2023
@reswqa @akkinenivijay
[FLINK-29298] LocalBufferPool request buffer from NetworkBufferPool h…
b5f5f4a 
…anging.

when the task thread polled out the last buffer in LocalBufferPool and triggered the onGlobalPoolAvailable callback itself, it will skip this notification  (as currently the LocalBufferPool is available), which will cause the BufferPool to eventually become unavailable and will never register a callback to the NetworkBufferPool.

This closes apache#20924
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@xintongsong xintongsong xintongsong approved these changes

Assignees

@wsry wsry

Labels
component=Runtime/Network
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@reswqa @flinkbot @xintongsong @wsry