New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AnthropicSolver #1498

Merged

JunShern merged 1 commit into openai:main from thesofakillers:anthropic-solver

Mar 21, 2024

Contributor

thesofakillers commented Mar 19, 2024

This PR contributes an AnthropicSolver class, a solver for using models served by the Anthropic Claude API, such as claude 3.

Besides basic functionality, the solver provides the following features

Handles backoff
Handles CoT and other solvers with non-alternating roles
token usage estimate

Notes:

logit biasing not supported by the anthropic API
checking for context length limits not supported; anthropic have not released a tokenizer yet (like tiktoken from openai)
supports chat models only. if anthropic releases base models at some point, we will address that when it arises


open source anthropic solver

b4e7c99

thesofakillers requested review from andrew-openai, etr2460 and katyhshi as code owners

March 19, 2024 10:26

JunShern self-requested a review

March 21, 2024 04:13

JunShern approved these changes

View reviewed changes

Collaborator

JunShern left a comment

Thanks for adding this! Great that we can start comparing eval results across different model providers.

JunShern merged commit e30e141 into openai:main

2 checks passed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment