Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Failed to open: ../registry/data/social_iqa/few_shot.jsonl with custom registry #1394

Closed
LoryPack opened this issue Nov 3, 2023 · 0 comments · Fixed by #1395
Closed

Failed to open: ../registry/data/social_iqa/few_shot.jsonl with custom registry #1394

LoryPack opened this issue Nov 3, 2023 · 0 comments · Fixed by #1395
Labels
bug Something isn't working

Comments

@LoryPack
Copy link
Contributor

LoryPack commented Nov 3, 2023

Describe the bug

I have created a new eval within a custom registry and installed evals from pypi (using a recent checkout from github). Calling oaievals works fine if I do not use few_show samples, but it does not if you include few shot examples.

As can be seen in the error thread below, the issue is caused by calling self._prefix_registry_path(self.few_shot_jsonl) (see _prefix_registry_path) which transforms the path string into a Path object. Then, open_by_file_pattern (here) still treats it as a string, unaware of the fact that it is actually a Path

To Reproduce

  1. Create a new eval with few-shot samples.
  2. Call oaieval, for instance:
oaieval text-ada-001 social_iqa_few_shot --registry_path=../registry
  1. You'll get an error thread as follows:
[2023-11-03 14:40:52,931] [registry.py:254] Loading registry from /home/lorenzo/venv/recog-LLM_capabilities/lib/python3.9/site-packages/evals/registry/evals
[2023-11-03 14:40:54,382] [registry.py:254] Loading registry from /home/lorenzo/.evals/evals
[2023-11-03 14:40:54,382] [registry.py:254] Loading registry from ../registry/evals
[2023-11-03 14:40:55,007] [oaieval.py:189] �[1;35mRun started: 2311031440552KUYNQGH�[0m
[2023-11-03 14:40:55,010] [data.py:90] Fetching ../registry/data/social_iqa/few_shot.jsonl
Traceback (most recent call last):
  File "/home/lorenzo/venv/recog-LLM_capabilities/lib/python3.9/site-packages/evals/data.py", line 54, in open_by_file_pattern
    if filename.endswith(".gz"):
AttributeError: 'PosixPath' object has no attribute 'endswith'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/lorenzo/venv/recog-LLM_capabilities/bin/oaieval", line 8, in <module>
    sys.exit(main())
  File "/home/lorenzo/venv/recog-LLM_capabilities/lib/python3.9/site-packages/evals/cli/oaieval.py", line 274, in main
    run(args)
  File "/home/lorenzo/venv/recog-LLM_capabilities/lib/python3.9/site-packages/evals/cli/oaieval.py", line 215, in run
    eval: Eval = eval_class(
  File "/home/lorenzo/venv/recog-LLM_capabilities/lib/python3.9/site-packages/evals/elsuite/basic/match.py", line 28, in __init__
    self.few_shot = evals.get_jsonl(self._prefix_registry_path(self.few_shot_jsonl))
  File "/home/lorenzo/venv/recog-LLM_capabilities/lib/python3.9/site-packages/evals/data.py", line 129, in get_jsonl
    return _get_jsonl_file(path)
  File "/home/lorenzo/venv/recog-LLM_capabilities/lib/python3.9/site-packages/evals/data.py", line 91, in _get_jsonl_file
    with open_by_file_pattern(path, mode="r") as f:
  File "/home/lorenzo/venv/recog-LLM_capabilities/lib/python3.9/site-packages/evals/data.py", line 75, in open_by_file_pattern
    raise RuntimeError(f"Failed to open: {filename}") from e
RuntimeError: Failed to open: ../registry/data/social_iqa/few_shot.jsonl

Code snippets

No response

OS

Ubuntu 20.04

Python version

python 3.9

Library version

git+https://github.com/openai/evals.git@dd96814dd96bd64f3098afca8dc873aa8d8ce4c8

@LoryPack LoryPack added the bug Something isn't working label Nov 3, 2023
LoryPack pushed a commit to LoryPack/evals that referenced this issue Nov 3, 2023
@LoryPack LoryPack mentioned this issue Nov 3, 2023
logankilpatrick pushed a commit that referenced this issue Jan 3, 2024
Simple change to fix #1394 .
jacobbieker pushed a commit to withmartian/-ARCHIVED--router-evals that referenced this issue Jan 9, 2024
Simple change to fix openai#1394 .
Linmj-Judy pushed a commit to TablewareBox/evals that referenced this issue Feb 27, 2024
Simple change to fix openai#1394 .
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant