Multiple choice datasets encounter a tensor error

[ethanedwards]

Multiple choice datasets such as "glue:mnli" returns a tensor error.

The command

elk elicit gpt2 "glue:mnli" --num_gpus 2

returns this output

Using 2 of 8 GPUs: [0, 4]
glue:mnli using 'train' for training and 'validation_matched' for validation
Found cached dataset generator (/home/ethan/.cache/huggingface/datasets/generator/default-a729374a01df289c/0.0.0)
Found cached dataset generator (/home/ethan/.cache/huggingface/datasets/generator/default-5453548fe42aba36/0.0.0)
Output directory at /home/ethan/elk-reporters/gpt2/glue:mnli/kind-golick
0%| | 0/13 [00:08<?, ?it/s]
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/mnt/ssd-1/spar/ethan/miniconda3/lib/python3.10/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "/mnt/ssd-1/spar/ethan/elkethan2/elk/elk/training/train.py", line 133, in apply_to_layer
**evaluate_preds(val_gt, val_credences, mode).to_dict(),
File "/mnt/ssd-1/spar/ethan/elkethan2/elk/elk/metrics/eval.py", line 75, in evaluate_preds
auroc = roc_auc_ci(to_one_hot(y_true, c).long(), y_logits)
File "/mnt/ssd-1/spar/ethan/elkethan2/elk/elk/metrics/roc_auc.py", line 103, in roc_auc_ci
raise ValueError("y_true and y_pred should be 1D or 2D tensors")
ValueError: y_true and y_pred should be 1D or 2D tensors
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/mnt/ssd-1/spar/ethan/miniconda3/bin/elk", line 8, in
sys.exit(run())
File "/mnt/ssd-1/spar/ethan/elkethan2/elk/elk/main.py", line 27, in run
run.execute()
File "/mnt/ssd-1/spar/ethan/elkethan2/elk/elk/main.py", line 19, in execute
return self.command.execute()
File "/mnt/ssd-1/spar/ethan/elkethan2/elk/elk/run.py", line 98, in execute
self.apply_to_layers(func=func, num_devices=num_devices)
File "/mnt/ssd-1/spar/ethan/elkethan2/elk/elk/run.py", line 175, in apply_to_layers
for df_dict in tqdm(mapper(func, layers), total=len(layers)):
File "/mnt/ssd-1/spar/ethan/miniconda3/lib/python3.10/site-packages/tqdm/std.py", line 1195, in iter
for obj in iterable:
File "/mnt/ssd-1/spar/ethan/miniconda3/lib/python3.10/multiprocessing/pool.py", line 873, in next
raise value
ValueError: y_true and y_pred should be 1D or 2D tensors