Skip to content

Commit

Permalink
Fix occasional "database is locked" errors while loading sample data
Browse files Browse the repository at this point in the history
SQLite doesn't support full-concurrency and often fails with
"SQLite3::BusyException: database is locked" when ActiveJob uses the default
threaded `:async` adapter. ActiveJob is used by ActiveStorage to analyze uploaded
images for metadata.

A side effect of this is that SQLite timeouts are no longed triggered
and the sample data loads sequentially in way less time (e.g. during
sandbox generation).

From https://stackoverflow.com/a/5767295:

SQLite locks the entire database during a write operation (i.e. when a
write is happening on any table, no other write, to any table anywhere
can happen at the same time). Some databases provide concurrent writes
via table-level locks, or sometimes row-level locks. To contrast this
to SQLite's implementation, a table-level lock basically means that
when you're writing data to a given table, no other thread can write
to any record in that table, at the same time (however, writes to
other tables can occur simultaneously, in some circumstances).
Similarly, row-level locks take it even further, and allow only the
necessary rows involved to be locked, allowing concurrent writes to
the same table to happen from multiple threads. The idea here is to
minimize the amount of data you need to lock for a write operation,
which effectively increases the amount of concurrent writes possible
across the database, and depending on your implementation/how you use
your database, this can significantly increase throughput.

Now, back to your question...

The fact that SQLite is threadsafe doesn't mean that multiple threads
can concurrently write to it - it means that it has a way of handling
access from multiple threads - which is to (a) allow timeouts/retries,
and (b) to return a useful error (SQLITE:Busy) when a lock is
currently held on the database. That is, threadsafe means nothing more
than, "Multiple threads can access this data in a way that won't
result in data corruption due to simultaneous access."

Basically, somewhere in the code, one thread is trying to do its
update before another thread has released its lock on the database.
This is a common hurdle with SQLite, because the authors/documentation
will tell you that SQLite can handle concurrency like a champ. The
reality is that what SQLite considers "concurrency support" amounts to
trying to be very fast so that locks on the database are only held for
a very short time, and therefore locks on the database are released
before timeouts are hit. In a lot of cases, this works just fine and
never gets in your way. However, having very short-lived locks is not
the same as actually allowing concurrent writes from multiple threads.

Think of it like the way that iOS does multitasking (at least as of
iOS 5, when I'm writing this) - really what it's doing is putting
other apps on pause, and coming back to them. This has the effect that
(a) battery life is much better due to lower CPU utilization, and (b)
you don't have to start an app from scratch every time you launch it.
This is great, but the actual word "multitasking" as used in iOS
doesn't technically mean the same thing as "multitasking" in other
environments (even Mac OS X).

SQLite is the same way. Do they have "concurrency" support? Well sort
of, but the way they define the word "concurrency" isn't the way the
rest of the DB world defines "concurrency".

No one is really wrong, but in cases like these, it adds to
implementation confusion.

Co-Authored-By: Alberto Vena <[email protected]>
  • Loading branch information
elia and kennyadsl committed Sep 28, 2022
1 parent bf0803d commit e6763c0
Showing 1 changed file with 12 additions and 0 deletions.
12 changes: 12 additions & 0 deletions sample/lib/spree_sample.rb
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,16 @@ class Engine < Rails::Engine

# Needs to be here so we can access it inside the tests
def self.load_samples
original_active_job_adapter = ActiveJob::Base.queue_adapter

# SQLite doesn't support full-concurrency and often fails with
# "SQLite3::BusyException: database is locked" when ActiveJob uses the default
# threaded `:async` adapter. ActiveJob is used by ActiveStorage to analyze uploaded
# images for metadata.
if ActiveRecord::Base.connection.adapter_name == "SQLite"
ActiveJob::Base.queue_adapter = :inline
end

Spree::Sample.load_sample("payment_methods")
Spree::Sample.load_sample("tax_categories")
Spree::Sample.load_sample("tax_rates")
Expand All @@ -27,6 +37,8 @@ def self.load_samples
Spree::Sample.load_sample("orders")
Spree::Sample.load_sample("payments")
Spree::Sample.load_sample("reimbursements")
ensure
ActiveJob::Base.queue_adapter = original_active_job_adapter
end
end
end

0 comments on commit e6763c0

Please sign in to comment.