--trust_remote_code does it actually do anything?

[devzzzero]

I'm running lm_eval --model hf --model_args pretrained=myllamamodel --device cuda:0 --batch_size 4 --tasks lambada_openai,hellaswag,piqa,arc_easy,arc_challenge,winogrande,openbookqa --log_samples --write_out --trust_remote_code -v debug

but it keeps on reporting this:

/home/ai/MiniConda3/envs/full-peft/lib/python3.10/site-packages/datasets/load.py:1486: FutureWarning: The repository for hellaswag contains custom code which must be executed to correctly load the dataset. You can inspect the repository content at https://hf.co/datasets/hellaswag
You can avoid this message in future by passing the argument `trust_remote_code=True`.
Passing `trust_remote_code=True` will be mandatory to load this dataset from the next major release of `datasets`.
  warnings.warn(

What am I doing wrong?

[StellaAthena]

You aren't doing anything wrong. To suppress that warning trust_remote_code=True needs to be passed to the dataset constructor, which you don't have direct access to when querying the library. This is something that needs to be changed under the hood.

Never mind, I'm apparently wrong.

[haileyschoelkopf]

--trust_remote_code is meant to do this and suppress the warning, so something must be up. Reopening.

[devzzzero]

Thank you for taking a look at this!

[baberabb]

Looks like it's because of this line:

if config.HF_DATASETS_TRUST_REMOTE_CODE and self.trust_remote_code is None:
    warnings.warn("....")

Not quite sure what they are going for here. Maybe the env variable currently defaults to True?

[devzzzero]

Looks like it's because of this line:

if config.HF_DATASETS_TRUST_REMOTE_CODE and self.trust_remote_code is None:
    warnings.warn("....")

Not quite sure what they are going for here. Maybe the env variable currently defaults to True?

Yea it's confusing!
Because I see that warning message, self.trust_remote_code must be None
This implies that the gated actions depending upon self.trust_remote_code is not carried out. So @haileyschoelkopf is right. It does seem like --trust_remote_code is likely a no-op :-(

[abzb1]

I performed mmlu eval using datasets version 2.20.0. Even if I use the --trust_remote_code argument, a trust_remote_code related error occurs. In datasets version 2.19.2, only warnings related to trust_remote_code are raised and evaluation is performed. I recommend keeping the datasets version below 2.19.2 until internal code changes.

[abzb1]

@baberabb
At lm_eval/main.py, trust_remote_code arg was added to model_args.
and also setting HF_DATASETS_TRUST_REMOTE_CODE = True.
However, it seems that model_args is only used to load the model and not the dataset.
At lm_eval/tasks/init.py, dataset configs are taken only from the yaml file of the task

[haileyschoelkopf]

Hi @abzb1 @devzzzero , I managed to track this down. Fix and description of what the issue is in #1998 !