improve eval performance by caching per-repo/version conda environments #104
Comments
waterson
added a commit
to waterson/SWE-bench
that referenced
this issue
Apr 26, 2024
This change provides a `path_conda` to use for the eval in the testbed directory that will be reused across evaluations, and modifies the context manager's behavior so that a non-existent `path_conda` will be initialized and populated in the same way that a temporary context would be. Fixes princeton-nlp#104.
|
I believe Auto Code Rover has an implementation of this in which they group non-redundant conda environments and cache them. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the feature
Right now running an eval (e.g., using the SWE-agent evaluation/evaluation.py script) runs in such a way that a temporary conda environment is created each time you run an eval. It seems like the conda environments could be created once per repo/version, and then reused again and again across different evaluations.
Potential Solutions
One way to do this (for which I'll attach a PR) is to simply configure a reaonable
path_condain the eval args; e.g.,The text was updated successfully, but these errors were encountered: