Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix rare race condition in ZClient causing healthy connections to be discarded #2924

Merged
merged 3 commits into from
Jun 23, 2024

Conversation

kyri-petrou
Copy link
Collaborator

This one has been bugging me for quite a bit of time now but I managed to finally trace it down. In short, there is a race condition which sometimes causes connections to be invalidated from the connection pool without any good reason (request succeeds and server keeps the connection alive)

The issue is that, we need to fulfil the onComplete promise before we invoke the final callback when we collect the async body, but after we removed the handler from the pipeline. The race condition happens because when the request Scope is closed, it also interrupts the onComplete promise here.

So when we make a request like below, if the callback wins the race, then the scope will be closed before we manage to fulfil the promise, and therefore the Channel will be discarded.

ZIO.scoped(ZClient.request(???).flatMap(_.body.asChunk))

The change in this PR ensures that the onComplete promise is fulfilled right before we call the async body handler on the last message, but after we've removed the handler from the pipeline. This is also important since if we remove the handler after we complete the promise, there is another potential race condition where we try to reuse the channel and end up with an error because the handler hasn't been removed yet.

Copy link
Member

@jdegoes jdegoes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These kinds of bugs can be incredibly annoying to track down, and it takes a skilled and dedicated programmer to do that. Thank you!

@jdegoes jdegoes merged commit 7b5d2a1 into zio:main Jun 23, 2024
34 checks passed
@kyri-petrou kyri-petrou deleted the fix-client-race-condition branch June 23, 2024 11:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants