feat: add local evaluation support using LlamaCppChatGenerator #7745
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Related Issues
Proposed Changes:
This implements a WIP version of local evaluation support with the feedback received in the respective issue.
During my work on this, there were some changes to the files, so I fixed the conflicts and merged. I will need to go over those changes more in depth and see if it makes sense to implement some of the changes for the llama.cpp version.
How did you test it?
Added tests (copies of OpenAI tests slightly modified for llama cpp implementation.
Notes for the reviewer
The implementation will require anyone using llm_evaluator to install llama-cpp-python which might not be ideal, but I don't see a way around it (lazy import best solution?)
There is a TODO in the code that I will need to figure out. Ideally, I would like to be able to extract whether the specific model supports the system prompt from Llama, but that doesn't seem easy. My current thought is to allow an optional flag and default to supporting system prompt when no flag is provided.
The link to LlamaCppChatGenerator does not work as it is not live yet.
Installing a model is necessary to run the tests. I added logic to handle that the same way we handle it in the llama cpp generator.
Checklist
fix:
,feat:
,build:
,chore:
,ci:
,docs:
,style:
,refactor:
,perf:
,test:
.