-
Notifications
You must be signed in to change notification settings - Fork 2.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Failed to open: ../registry/data/social_iqa/few_shot.jsonl
with custom registry
#1394
Labels
bug
Something isn't working
Comments
LoryPack
pushed a commit
to LoryPack/evals
that referenced
this issue
Nov 3, 2023
Merged
logankilpatrick
pushed a commit
that referenced
this issue
Jan 3, 2024
jacobbieker
pushed a commit
to withmartian/-ARCHIVED--router-evals
that referenced
this issue
Jan 9, 2024
Simple change to fix openai#1394 .
Linmj-Judy
pushed a commit
to TablewareBox/evals
that referenced
this issue
Feb 27, 2024
Simple change to fix openai#1394 .
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
I have created a new eval within a custom registry and installed evals from pypi (using a recent checkout from github). Calling
oaievals
works fine if I do not use few_show samples, but it does not if you include few shot examples.As can be seen in the error thread below, the issue is caused by calling
self._prefix_registry_path(self.few_shot_jsonl)
(see _prefix_registry_path) which transforms the path string into aPath
object. Then,open_by_file_pattern
(here) still treats it as a string, unaware of the fact that it is actually aPath
To Reproduce
oaieval
, for instance:Code snippets
No response
OS
Ubuntu 20.04
Python version
python 3.9
Library version
git+https://github.com/openai/evals.git@dd96814dd96bd64f3098afca8dc873aa8d8ce4c8
The text was updated successfully, but these errors were encountered: