-
Notifications
You must be signed in to change notification settings - Fork 73
Restarting PostgreSQL sometimes causes Pool.getconn() to hang #147
Comments
Oops, forgot to show my complete reproduction script:
If I run this normally, the output is very boring:
But if I start it running, and then in other terminal I run
That is, one query fails while PG is down, and then we're back in business with a new backend. I'd love it if momoko hid that and did the necessary retries for me, but that's not the serious problem. Here is the serious problem, which I see occasionally:
...and that's it. My app is dead, frozen, hung. Nothing happens until I kill it and restart it. Note that I can greatly increase the frequency of hangs by decreasing the sleep after "error from pool.getconn()". Eg. with a 1 sec sleep as shown, roughly 1 in 10 restarts result in a hung process. But if I reduce the sleep to 0.1 sec:
then my process hangs almost every single time. It's dead easy to reproduce. |
And here is the output of running
|
Oh yeah, I'm seeing this on:
|
You are right about both. Sorry for the late response - really busy at work and 10-month old daughter :) The reconnect logic was only conceived to reconnect idle connections that were silently dropped by the server - for example, if you are running a lazy query once in an hour; and 30 minutes in the middle the server was restarted. Regarding the hangs - this is bug indeed. It turns out that there even is a simpler reproduction - instead of doing restarts, just shutdown the server and your script would hang. I've fixed this in master. Can you please have alook? |
And thanks for thorough reproduction (as usual :) |
* commit '5c15607a69a80f1a40c25bb0c88461ce21f15804': Going 2.2.4 Dropping support for Python 3.2 Ping may fail early when all connections are dead. Fixes FSX#147 flake8 cleanup Provide a useful __repr__() method for ConnectionContainer. Improve one small log message. Fix grammar/typo in comment and docstring. Fixed tutorial examples. Closes FSX#143. Fixes 139 - `all_dead` should check for pending Explicitly document that dict parameters are supported on execute. Explicitly document that dict parameters are supported. Fix typo.
Imagine a simple application that runs an infinite loop, periodically querying a database via momoko:
As long as I never restart my PostgreSQL server, this works perfectly. But when I restart PG, bad stuff happens. Specifically:
getconn()
fail withDatabaseNotAvailable: No database connection available
. That's annoying; I would think that a connection pool would take care of reconnecting when my database server restarts. I don't want to wrap a retry loop around momoko, because IMHO momoko should do that for me!pool.getconn()
never yields a result. My query-forever app is dead and has to be manually restarted.The latter bug is what I'm really complaining about. If momoko's position is that PG restarts are an application problem rather than a connection pool problem, and it's my job to retry
getconn()
to handle PG restarts: not ideal, but I can live with it.But if momoko occasionally freezes up when PG restarts, that's a more serious problem. IMHO that needs to be fixed.
The text was updated successfully, but these errors were encountered: