Soft design doc updates (without recomputing the views)

[j3k0]

Summary

The idea is to be able to update a design document without invalidating the existing index.

Desired Behaviour

When updating a design document, make it possible that the new index uses the existing one as a starting point.

Possible Solution

The user can specify keep_index=true as a query parameter when changing the design document with PUT /{db}/_design/{ddoc}

Internally, copy or rename the existing index file (I'm not sure if that would work out of the box)

Additional context

On large database (Terabytes) updating an index with a custom JavaScript map function takes a very long time (days).

Many times it would be fine to update the view without recomputing the full index. For example we are often updating the view to process a new type of data that isn't already present in the database: this new type of data will enter the database when we roll out a feature to production.

In that case, we could update the view without recomputing the index on existing data: we know for a fact that the update has no impact on existing data. It would save days of delays to release the new feature.

In another instance, the view had a little bug. Being able to deploy a "hot fix" so new documents are not impacted by the bug would be incredibly helpful. Having to tell users to wait a few days for the fix isn't great.

[janl]

can you share a concrete example of JS functions that you think should not reset the index?

[j3k0]

Sure, a simplified pseudo example (our actual views are hundreds of lines)

Context: an "events" database that contains the interactions (REST and Webhook calls) with various payment providers.

We want to emit the history of states for all transactions.

Before updating the design document

function (event) {
  switch (event.type) {

    case "appstore.webhook":
      event.appleTransactions.forEach(function(appleTransaction) {
        emit([appleTransaction.id, event.date], appleTransaction.transactionState);
      }); break;

    case "appstore.verifyReceipt":
      // handle specific data stored in "apple.verifyReceipt" events.
  }
}

After updating the design document

Now we want to add support for Google Play to our system.

function (event) {
  switch (event.type) {

    // UNCHANGED
    case "appstore.webhook":
      event.appleTransactions.forEach(function(appleTransaction) {
        emit([appleTransaction.id, event.date], appleTransaction.transactionState);
      }); break;

    case "appstore.verifyReceipt":
      // handle specific data stored in "apple.verifyReceipt" events.

    // +++ ADDITION HERE +++ we add support for google-play-style server-to-server notifications.
    case "googleplay.notification":
      emit([event.googlePurchase.purchaseToken, event.date], event.googleNotification.type);
      break;

    // and more event types...
  }
}

We could in principle redeploy this view to production without recomputing the index, because there's no event with type: "googleplay.notification" in the database yet. Note: our DB is 5TB and it contains 500M documents.