Evaluating AB testing variants, given an experiment definition and some context.

We have a microservice that is in charge of knowing whether a user should be part of an experiment or not, and if they are, whether they should be in the control group, or the experiment group.

The reasons why a user might participate or not are very varied, and change drastically from experiment to experiment.

We want to have the experiment's microservice be somewhat agnostic of the rest of the microservice infrastructure; so whoever consumes the information must provide enough information about the user for us to choose what experiments they should be part of.

We found that knowing if a user participates in an experiment, and it figures out what variant they fall into are really similar concepts, so we came up with the idea of having "variants all the way". So an experiment is really just a variant on the world. The same syntax can apply to narrow down the focus of the experiment. A good example of this can be found in the Variant all the way example.

We borrowed heavily for Mongo's query language.

These are the fields that start with a $. There are two families of operators, describe below.

These operations change the context, either narrowing down the provided context, or by plucking some information from the World. Think of them as :: Context -> Context.

These operations yield weather the context is or not par of the variant. Think of them as :: Context -> Bool.

Where a context changer can be nested, evaluators need to be terminal.

Even though there are both $and and $eq operators; they are rarely used, as they can be expressed more concisely. Refer to the And example for more information.

You can play around with different experiments and contexts in our koncierge playground.

Everyone is participating in the experimentEXP001, and only half of the users (chosen at random) will see the experiment.

{
    "EXP001": {
        "$children": {
            "participating": { "$rand": { "$gt": 0.5 } },
            "control": { "$always":  true }
        }
    }
}

Playground

The output for half the user base will be: EXP001.participating and EXP001.control for the other half.

{
    "EXP001": {
        "$date": { "$gt": "2020-01-01 15:00:00" },
        "$children": {
            "participating": { "$rand": { "$gt": 0.5 } },
            "control": { "$always":  true }
        }
    }
}

Playground

The output will be the same as the simple example, but only if it is later than the January first 2020.

{
    "EXP001": {
        "$children": {
          "never": { "$always":  false }
        }
    }
}

Playground The output will be EXP001 even if it has no active child.

{
    "EXP001": {
        "$date": { "$gt": "2020-01-01 15:00:00" },
        "beta": "yes"
    }
}

Playground The output will be EXP001 if the context has a key beta as yes and is queried after January first 2020.

{
    "EXP001": {
        "beta": {
            "$always": true,
            "$not": { "$eq": null }
        }
    }
}

Playground The output will be EXP001 if the context has a key beta, and it is not null.

{
   "EXP001": {
        "$children": {
            "control": { "$always":  true },
            "participating": { "$rand": { "$gt": 0.5 } }
        }
    }
}

Playground

This will always output EXP001.control, as we parse children rules sequentially.

You can check out the /test folder for more examples and edge cases, as well as playing in the Playground

The library is divided into two major namespaces.

1. Parse (Left Kan).
2. Interpret (Right Kan).

A sub-section of the interpretation is also to validate that the given context can be matched to the rules.

A function that takes the full JSON definition of the experiments and transforms it into objects that can be then interpreted

A function that takes the parsed definition, the context provided, and a World and outputs the list of experiments (and variants) the context is part of

We validate the context in an interpreter-agnostic way. This means that even though semantically, some rules we might never call (see the Uncalled for example), the context will still need to match every possible child rule.

Both $gt and $lt only work with numbers, so dates will be compared by their epoch second. The parser can interpret some date formats (read more about the date formats here). Formats with no timezone, we choose the default system timezone. This is not recommended.

Even though $rand and $chaos might look similar; they differ on the seed for its randomness. With $rand, any random value generated with the same context and variant name, will be the same output.

For this reason, most of the times you'll want to narrow down the context before applying $rand. For example, given this context:

{
    "user_id": 3,
    "last_login": "2020-10-15T10:00:00.000Z"
}

the last_login will probably change throughout the time, but we need the experiment to be fixed by the user_id. In such a scenario, we can write the experiment as such:

{
    "EXP001": {
        "user_id": {
            "$rand": { "$gt": 0.5 }
        }
    }
}

Every time the user_id: 3 queries the experiment, the $rand value will be the same.

In contrast, $chaos will generate a new value each time, so there is no guarantee in what variant user_id: 3 will fall.

As we generate the same number for each context, you can safely update the threshold and rest assured that the minimum number of clients will change variants.

1. Escape the $ to be able to match to $ keys, and the . in keys to match not-nested keys with .