-
-
Notifications
You must be signed in to change notification settings - Fork 5.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix SQLite concurrency problems by using BEGIN IMMEDIATE #10368
Conversation
d14cac2
to
1226096
Compare
Codecov Report
@@ Coverage Diff @@
## master #10368 +/- ##
==========================================
- Coverage 43.7% 43.67% -0.03%
==========================================
Files 586 586
Lines 81411 81411
==========================================
- Hits 35582 35560 -22
- Misses 41422 41442 +20
- Partials 4407 4409 +2
Continue to review full report at Codecov.
|
EDIT: what would be the alternative ... since sqlite is only for smal installations I think querys can be exec sequencial not paralell. But we should leave a big note in the documentation for sqlite and if it is the right desission as backend |
This seems like an appropriate solution. Here is a relevant section from the definitive guide to Sqlite: Deadlocks Both sessions wind up in a deadlock. Session B was the first to try to write to the database and therefore has a pending lock. A attempts to write but fails when INSERT tries to promote its shared lock to a reserved lock. For the sake of argument, let’s say that A decides to just wait around for the database to become writable. So does B. Then at this point, everyone else is effectively locked out too. If you try to open a third session, it won’t even be able to read from the database. The reason is that B has a pending lock, which prevents any sessions from acquiring shared locks. So, not only are A and B deadlocked, they have locked everyone else out of the database as well. Basically, you have a shared lock and one pending lock that don’t want to relinquish control, and until one does, nobody can do anything. How do you avoid a deadlock? It’s not like A and B can sit down in a meeting and work it out with their lawyers. A and B don’t even know each other exists. The answer is to pick the right transaction type for the job. Transaction Types An immediate transaction attempts to obtain a reserved lock as soon as the The crux of the problem in the preceding example is that both sessions ultimately wanted to write to the database, but they made no attempt to relinquish their locks. Ultimately, it was the shared lock that caused the problem. If both sessions had started with The bottom line is this: if you are using a database that no other connections are using, then a simple |
Note that when it says:
in our case it will just block because we're using the sqlite_unlock_notify library. |
) * Test locking immediate for SQLite3 * fix url field separator Co-authored-by: Lunny Xiao <[email protected]> Co-authored-by: zeripath <[email protected]>
This PR proposes a solution for the fortuitous deadlock errors produced in SQLite. The way it works is by preventing any two transactions (
BeginTransaction() => Commit()
) from running at the same time.The reason I think this is needed is because I see no right way of orchestrating our queries in a way that they will never interfere with each other. As we move actions into queues, there are more operations running simultaneously.
I've tested this several times and I couldn't get a deadlock, while it was a 70~80% of getting one without this modification.
As a solution this might be a little extreme, but I'm leaving this here just in case. I labeled it as
status/blocked
until certain consensus is reached.Sources:
https://www.sqlite.org/lockingv3.html
https://www.sqlite.org/unlock_notify.html
https://godoc.org/github.com/mattn/go-sqlite3#SQLiteDriver.Open