Share conda environment across evals #105
Conversation
This change provides a `path_conda` to use for the eval in the testbed directory that will be reused across evaluations, and modifies the context manager's behavior so that a non-existent `path_conda` will be initialized and populated in the same way that a temporary context would be. Fixes princeton-nlp#104.
This simplifies the parsing a bit by simply using the `--json` option to provide an easy-to-parse list.
|
Hi @waterson thanks for the suggestion + contribution and your patience! We have just published a major release I left a more extensive reply to @aorwall at #109. In a nutshell, with Put simply, if SWE-bench evaluation is run 2+ times, if the Also, the Thanks again for the issue - it provided the confirmation we needed to move forward with incorporating this feature, I really appreciate it a lot! |
This change provides a
path_condato use for the eval in the testbed directory that will be reused across evaluations, and modifies the context manager's behavior so that a non-existentpath_condawill be initialized and populated in the same way that a temporary context would be.I realize that it might make some sense to have a bit of discussion about optionalizing this behavior, but I wanted to just get something out there to talk about. :)
Fixes #104.