Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Don't try loading models from the cwd ever #223

Merged
merged 4 commits into from
Apr 29, 2023
Merged

Conversation

norabelrose
Copy link
Member

This fixes a very annoying quirk of HuggingFace from_pretrained methods: if you try to load, say, huggyllama/llama-7b when there's a folder named huggyllama/llama-7b in the current working directory, it'll try to load from that folder instead of the HF Hub (or the relevant cache dir). We basically never want this behavior, and it becomes a problem when you happen to be cded into the elk-reporters directory for example.

This PR fixes this with a little hack: we just move the offending local directory to a temp dir for a short time while we call Auto<whatever>.from_pretrained, then move it back. AFAICT there's not a better way to do this.

Also, as a bonus, this PR makes the change that Christy and I talked about like a week ago, where datasets like "super_glue boolq" are now named like super_glue:boolq so that you don't have to wrap it in double quotes on the command line

@AlexTMallen
Copy link
Collaborator

I did mkdir gpt2 from the elk repo root, and then got the following result:

❯ elk elicit gpt2 amazon_polarity --max_examples 10 --debug
Traceback (most recent call last):
  File "/mnt/ssd-2/spar/alexm/miniconda3/envs/elk/bin/elk", line 8, in <module>
    sys.exit(run())
  File "/mnt/ssd-2/spar/alexm/elk/elk/__main__.py", line 27, in run
    run.execute()
  File "/mnt/ssd-2/spar/alexm/elk/elk/__main__.py", line 19, in execute
    return self.command.execute()
  File "/mnt/ssd-2/spar/alexm/elk/elk/run.py", line 50, in execute
    self.datasets = [
  File "/mnt/ssd-2/spar/alexm/elk/elk/run.py", line 51, in <listcomp>
    extract(
  File "/mnt/ssd-2/spar/alexm/elk/elk/extraction/extraction.py", line 310, in extract
    with temporary_dir_move(cfg.model):
  File "/mnt/ssd-2/spar/alexm/miniconda3/envs/elk/lib/python3.10/contextlib.py", line 135, in __enter__
    return next(self.gen)
  File "/mnt/ssd-2/spar/alexm/elk/elk/utils/data_utils.py", line 64, in temporary_dir_move
    path.rename(dest)
  File "/mnt/ssd-2/spar/alexm/miniconda3/envs/elk/lib/python3.10/pathlib.py", line 1234, in rename
    self._accessor.rename(self, target)
OSError: [Errno 18] Invalid cross-device link: 'gpt2' -> '/tmp/tmpwn7ysohc/gpt2'```

Copy link
Collaborator

@AlexTMallen AlexTMallen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

elk elicit gpt2 "glue:qnli" --max_examples 10 --debug works on my end despite a conflicting directory name and the code LGTM

@norabelrose norabelrose merged commit 84571d2 into main Apr 29, 2023
@norabelrose norabelrose deleted the no-cwd-loading branch April 29, 2023 22:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants