Data Importing

[ritch]

There are several use cases for importing data into a loopback data source. I've listed the ones that I can think of, feel free to add any.

1. As a developer working on a new project, I need test data to work with. This could be for integration tests, prototyping, or even modeling data.
2. As a developer working on an existing project with exsiting data, I need to constantly update my local development data with that which mirrors, closely mirrors, or simulates prodution data.
3. LoopBack internals require data to be imported during app startup. Although it is important to note that the core could be modified in a way that does not depend on bootstrapped data.

Here is a basic example of an API and JSON format that could make implementing these use cases fairly straightforward and simple.

var app = loopback();
var db = app.dataSource('db', {connector: loopback.Memory});
var Car = app.model('Car', {dataSource: 'db'});
var Dealership = app.model('Dealership', {dataSource: 'db'});
var Option = app.model('Option', {dataSource: 'db'});
var Make = app.model('Make', {dataSource: 'db'});

Dealership.hasMany(Car);
Car.hasMany(Option);
Make.hasMany(Car);

// data.json
{
  data: {
    dealerships: [
      {
        name: 'acme cars',
        $cars: [
          {model: '500', $make: '#fiat', $options: ['#leather', '#ac']},
          {model: 's4', $make: '#audi', $options: '.audi-options'},
          {model: 'm3', $make: '#bmw'},
        ]
      }
    ],
    makes: [
      {name: 'audi', $$id: 'audi'}
      {name: 'fiat', $$id: 'fiat'}
      {name: 'bmw', $$id: 'bmw'}
    ],
    options: [
      {name: 'leather', $$id: 'leather'},
      {name: 'ac', $$id: 'ac'},
      {name: 'audi option 1', $$class: 'audi-options'},
      {name: 'audi option 2', $$class: 'audi-options'}
    ]
  }
}

var testData = app.createDataSet({
  file: 'data.json', //or data: someDataObj
});

testData.import({drop: true}, function (err) {
  console.log(err || 'Data imported');
});

A bit of reference for the JSON format:

• $$ - indicates a reference only for use by the importer, allowing references / relations to be created before the actual object is inserted and id generated
• $$id - for referencing a single instance
• $$class - for referencing several instances
• $ - indicates a special property. For now, this is used only for relations
• $myRleationProperty - indicates the value for this key is a string or array of strings referencing classes or ids (see above).
• .my-class-name - similar to how classes work in CSS: allows you to reference a set of objects tagged with a $$class: 'my-class-name.
• #my-id - similar to how ids work in CSS: allows you to reference a single object by id

[bajtos]

@ritch Could you please elaborate a little bit more on the meaning of dolare and double-dolar keys?

It seems that $make is depending on name values, does your design allow to configure the lookup (e.g. $owner should look up a user by login instead of owner by name)?

Perhaps we can auto-detect string (path to JSON file) vs. object (in-memory data), so that app.createDataSet takes a single argument?

There are two functions needed to perform the import (app.createDataSet and testData.import), why not a single function app.importDataSet(options, callback)?

[ritch]

I'd like to have room to create more methods on the data set itself. It would also be nice if the DataSet was an event emitter (perhaps a writeable stream?). This is just a best practice / idiom I see in node core. Eg. server = http.createServer().

[ritch]

@bajtos - updated the original post with a brief reference

[raymondfeng]

Importing data set for individual models seems to be straightforward. The connected/related data set are tricky, for example, a customer and a list of orders placed by the customer.

[bajtos]

The connected/related data set are tricky, for example, a customer and a list of orders placed by the customer.

I believe Ritchie's proposal solves this problem in rather elegant way. "orders placed by a customer" is the same relationship as "cars made by a manufacturer", you can find the latter in the sample code.

[Schoonology]

@ritch I agree that a separate data type for importable data makes sense, but I would invert the relationship somewhat (if practical - thinking out loud here). I would look more to:

var set = loopback.DataSet.loadFromFile('data.json'); // Static method to load, could use load(stringOrObj), etc.
app.importDataSet(set, { drop: true }, function callback(err) { ... });

The DataSet, then, is just a decorated representation of importable data, able to be (de)serialized and passed around. The application knows how to import that decorated version into a set of models.

This grants all kinds of cool powers, too. Replication, anyone? What about sync? Fun times.

[bajtos]

As part of our issue triage process, I am closing items that have been inactive for an extended period of time. If anybody thinks the feature proposed here is still relevant and should be prioritised, then please leave a comment to let us know.