Sweep MVP

[norabelrose]

You can now do stuff like
elk sweep --models gpt2 gpt2-medium gpt2-large gpt2-xl --datasets imdb amazon_polarity
and it'll run elicit on the Cartesian product of the models and datasets you list, storing it in a special folder ELK_DIR/sweeps/<memorable_name>.

[lauritowal]

Looks good to me.
Although, don't we also want transfer evaluations e.g. gpt2 trained on imdb, but evaluated on ag_news and vice versa? Could be another task / pull-request for later...

[norabelrose]

Looks good to me.

Although, don't we also want transfer evaluations e.g. gpt2 trained on imdb, but evaluated on ag_news and vice versa? Could be another task / pull-request for later...

Yeah we can add eval in the next PR. Also can you use bash brace expansion in the README example so that people know you can use it? For example
--models gpt2-{tiny,medium,large,xl}. This is just the shell expanding things out for you, I didn't have to write any code for it, but it's really convenient.

[norabelrose]

Another thing for the next PR is probably to add like --config-path foo.yaml or some way of specifying nondefault hparams that are shared across the sweep