-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Transaction Support #109
Comments
Adding @piscisaureus, he might want to comment on the way how domains are used here. First of all, I think adding some sort of transactions would be great, as it will allow us to implement transaction-level caching and save database roundtrips when the same object is queried again. Could you add a code sample showing how to do a transaction with commit? How to rollback the transaction on error? Getting transactions right is a very tricky business. Imagine this scenario (Tx1, Tx2 and Tx3 are transactions started by two different HTTP requests)
If we changed Step 3, so that Tx2 gets the updated value ($11), consider alternative scenario, where Tx1 is rolled back in step 5. Swapping the order of steps 2 and 3 makes the situation even more interesting. Things get much complicated when we want to have a single abstraction for SQL and multiple NoSQL stores, because everybody use a very different approaches to achieve consistency. To name a few:
[1] https://docs.basho.com/riak/1.3.0rc1/references/appendices/comparisons/Riak-Compared-to-MongoDB/ (row Data Versioning and Consistency) |
Agreed. When you have concurrent transactions against the same data, isolation can only be maintained by ensuring order. Locking makes this reliable but slow. Snapshot isolation is how I think we would be able to generically handle this. The idea is that a transaction would fail if any updates conflict. The upside is performance and a relatively simple implementation. The downside is users have to handle conflict errors for all transactions.
|
Let's take advanced features (such as 2PC, nested TXs, and NoSQL) out of the discussion and focus the simple 1PC case.
|
@raymondfeng Before taking NoSQL out of the discussion, please consider this: Because we don't support MongoDB's foldable operators and instead we always use $set which works in "last-writer-wins" manner, it's impossible to achieve data consistency in a vanilla LoopBack application at the moment. Unless the plan is to have a set of transaction APIs, a different API for every database-type (SQL, Mongo, Couch, etc.). In which case it's ok to focus on SQL only. (But I don't think such set of APIs could be called "easy to work with".) @ritch Failing transactions with a conflict is a possible solution. We will probably have to associate transaction.id with with Model.id, not a Model instance, because there may be multiple instances (JS objects) representing the same database record Model.id), but that's an implementation detail. |
I did a bit of thinking about this topic in the past days and would like to make few more points.
I am proposing the following incremental steps: 1. Expose native low-level APIThe first step is to allow LB users to re-use single connection for multiple commands/queries and allow them to call native API (send SQL commands, call Mongo's update with foldable operators, etc.). This addresses the following points from Raymond's comment:
/*** Oracle **/
var oracle = loopback.createDataSource({ /* Oracle config */ });
// to avoid an extra callback, the connection is initialised lazily
// as part of sending the first request
var conn = oracle.getConnection();
conn.execute('START TRANSACTION', function(err) {
if (err) { conn.close(); return fail(); }
conn.execute(
'SELECT balance FROM accounts WHERE id = 42',
function(err, res) {
if (err) { /* rollback, close connection, return fail */ }
conn.execute(
'UPDATE accounts SET balance = :bal WHERE id = 42',
{ bal: res[0].balance + 1 },
function(err) {
if (err) { /* rollback, close connection, return fail */ }
conn.execute(
'COMMIT',
function(err) {
conn.close();
if (err) return fail();
done();
}
);
}
);
}
);
});
/*** MongoDB ***/
var mongo = loopback.createDataSource({ /* MongoDB config */ });
var conn = mongo.getConnection();
conn.accounts.update({ _id: 42 }, { $inc: { balance: 1 } }, function(err) {
conn.close();
if (err) return fail();
done();
}); 2. ModelsOnce we have the low-level mechanisms available, we can integrate them with Models. As Raymond wrote:
I see two possible approaches here:
3. RESTProvide a convenient helpers to automatically wrap certain remotable methods in a transaction. Note that this should be an optional detail built on top of existing APIs, because for more complex operations involving multiple external services, users should limit the duration of the db transaction to only the relevant part of the operation (i.e. don't hold db transaction open while you are waiting for a result from SOAP call). This can be implemented via strong-remoting hooks as described by Ritchie, or even by extending LDL and adding a new configuration property. fn.shared = true;
fn.wrapInTransaction = true;
fn.accepts = // etc. |
+1 |
This is a great idea. |
After contemplating this a bit more, this is where I landed as well. We don't have to relegate features for compatibility / abstraction. We should expose features as they exist and clearly explain the discrepancies. |
Have you implemented this? (transactions). because, I really need to implement this in my project |
Not yet, it hasn't been a priority for us. |
Uhmmm, the data consistency should be a priority. Especially if the framework works with databases. |
If the DB driver supports transactions you should be able to call natively from the LB connector. We're not implementing transactions at the Juggler level at this time. 2PC is difficult enough when the multiple instances of a data source support it, let alone across multiple disparate data sources. Our priorities for @bajtos and the rest of the very small team are being driven by our customer obligations and community feedback such as yours. We appreciate the +1 in this area. If there's an urgent commercial need or support please feel free to contact our sales team. |
ok, thanks for the answers. I'll wait later versions. |
Absence of transactions is a pain. |
👍 |
2 similar comments
+1 |
+1 |
Hello, Have you implemented "Transactions" yet? |
Not yet. I'm in the process of refactoring the RDBMS connector implementations and it will become easier to add the transaction support. |
+1 |
At Strongloop, is possible only attach a transaction to a Model? Is possible to use multiple models operations inside a Transaction block without chaining transactions, as in SQL? |
There is no reliable way in Node to implicitly propagate the transaction context over the async invocation chain. As a result, we choose to use |
is there any to pass this options to native sql query also ? |
@visusharma please open a new issue to discuss that use case. |
/to @raymondfeng @altsang
/cc @bajtos
In the interest of LoopBack reliability, I'd like to start a discussion on making LoopBack ACID compliant. After brushing off earlier attempts to look into the issue I had an idea.
I think Transactions + Multiversioning will get us 80% or more to ACID.
Transactions
The above requires the
dataSource
/connector
to implement methods that delegate to driver transactions, or multiversioning. To support multiversioning, an outer wrapper of the connector could be devised that essentially tracks each method call and model instance. As well as changing the behavior slightly.Multiversioning
By default all update operations during a transaction would create copies of the given model instance and tag them with the
transaction.id
. Model's created during the transaction would also be tagged with thetransaction.id
.By default, all queries would exlclude instances tagged with
transaction.id
s.Failure / Cleanup
If the transaction fails. All models tagged with a
transaction.id
are removed. This could also be ensured on startup / by a job of some sort to essentially garbage collect tagged instances.If the transaction completes without error all tagged model instances would replace matching untagged models.
Concerns
During the final phase of a succesful transaction, since transactions aren't isolated by locking, there is a potential for cleanup to happen out of order. If the
transaction.id
is sortable we might be able to avoid this.The text was updated successfully, but these errors were encountered: