Trained models can be tied to specific backdoors

[VRehnberg]

An example of this issue as @ejnnr observed, is for the WaNet Backdoor. A trained model with this type of backdoor responds to a specific randomly generated warping field. Later you would typically like to evaluate a detector detector based on absence or presence of this warping field in question. As code currently looks this isn't possible.

I'd like to take this issue to consider some possible design choices.

Usage

This benchmark is about evaluating a Detector on how well anomalous behaviour can be detected in a Model for some input.

To make this possible there needs to be a known Task for which a dataset that is known to produce anomalous and normal behaviour in a model has to be known.

A Detector is then evaluated on a Task. Some Tasks need to initialized which in the case of the Backdoor task consists of:

• Training a model on a Dataset passed through a BackdoorTransform.

When a Task has been initialized it should be possible to load this Task, at least in cases when initialization is costly (e.g. training a model). However, it is also of interest to load a modified Task. For example a different transform than the one used in training the model to see that these are not detected as anomalous.

Design choices

Main script is

eval_detector --task TASK --detector DETECTOR

so far this is the same set-up as before. Before this step there will be a

train_detector ...
init_task ...  # e.g. train_classifier

I don't see any need to change anything with the detector set-up.

The changes will have to be with how the task is handled both in initialization and the command line arguments to eval detector.

Choice 1

Add a --task from_run similar as for the detector. This should preferably take as much as possible from the initialization of the task and then any additional settings.

Most of the information already exist in the config.yml and model/ under dir. All that is missing is either a:

• backdoor/ that contains a saved state of the backdoor used or
• seed in config.yml that is used to initialize the same warping field

For saving the backdoor I would probably suggest to pass the path to where to save the backdoor in a call to train_loader.transforms.save_state(cfg.dir.path) in train_classifier.py and add stuff in Transform and its children where needed to save what states are necessary for reproduction.

Choice 2

Add an option --task.my_task.run_path, task.my_task.[...].run_path or similar only when needed. In that case I would strongly prefer to have a reasonable default that uses an equivalent instance of a Task as was used when the Task was initialized.

I can't think of a good way to set the default in e.g. WanetBackdoor such that this is the same path as was specified in train_classifier.py if this was called in that function. And then this something equivalent should also be the done when loading in eval_detector.py.

---

I don't think any of these alternatives is obviously good. The easiest one and the one I would probably pick in the short term is probably to just add something like

class WanetBackdoor:
    ...
    init_seed: int = np.random.randint(2**64 - 1)

    def __post_init__(self):
        ...
        tmp_rng = np.random.default_rng(seed=self.seed)
        self.control_grid = 2 * tmp_rng.random((control_grid_shape)) - 1

I'm not sure how this is/should be picked in eval_detector.py but the information should exist in config.yml so shouldn't be too hard to retrieve.

[ejnnr]

Thanks for writing up these options!

I think saving the actual control grid is a better solution long-term than just using the same seed, for a few reasons:

• Seems less error-prone to me
• Makes it easier to run things across different machines (where RNGs might differ); e.g. you might want to train a model on a machine with better GPUs but then just quickly evaluate something locally.
• As you say, we might have cases in the future where storing results like this is important because they're expensive to compute.

Here's a potential way of implementing this, taking some inspiration from your choices:

• Have store and load methods in Transform, which by default just don't do anything
• WanetBackdoor would store/load the control grid in those methods
• The BackdoorDetection task config would use its run_path to call the load method in _get_anomalous_test_data
• train_classifier.py calls store() on all the transforms used for training
• The path argument to store and load could just be the base run path, and each transform could use a subdirectory/file with a name (e.g. "wanet") to support storing/loading several transforms

Instead of load, we could also have a path field in Transform and just let subclasses use that if they want, but maybe being explicit is better here.

One thing I don't like about this is that it assumes you always want to reuse transforms. Probably true most of the time, but this would make it completely impossible to e.g. test on a new control grid as an ablation. I suppose Transform could have a CLI flag that suppresses loading and turns load into a no-op? Something like

from simple_parsing.dataclasses import field
...
no_load: bool = field(action="store_true")

...
def load(path):
    if self.no_load:
        return

Then by default you'd reuse the grid, but you could always add --task.backdoor.no_load to change that.

If you have ideas for making this more elegant, happy to discuss more, but otherwise I think this would be good enough from a practical perspective and doesn't require changing things too much.

[ejnnr]

We could try to fix path specifications more thoroughly right away (there are a few cases where you need to specify paths multiple times). If that's something you want to tackle anyway, it probably makes sense to do both at once. Otherwise I'll probably look into a more general fix at some later point.

[ejnnr]

Fixed in #5