-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Update Zeno Integration #1175
Comments
Regarding tests, we could run a test on our end. E.g., we have integration tests set up where we have a project created on push. Would not alert you if anything breaks, though. I can also help set up a test that uploads a project to Zeno in your codebase if you want to. |
We've seen some patterns in useful metadata recently. For example, the model output length in freeform answers is often interesting. We could think about additional metadata that would make sense per task to further enhance the created Zeno project. |
For tests, see #1221 |
For additional metadata, see #1222 |
There are a few things we'll need to do to fix edge cases in our Zeno integration.
cc @lintangsutawika for your awareness because changing how metrics / aggregations are computed to unify them might harm Zeno's reliance on per-example metrics.
cc @Sparkier !
The text was updated successfully, but these errors were encountered: