Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Apibench] Resume interrupted LLM generations from last generation #484

Open
ShishirPatil opened this issue Jun 24, 2024 · 0 comments
Open
Labels
apibench-data APIBench data good first issue Good for newcomers

Comments

@ShishirPatil
Copy link
Owner

Describe the issue
When we try to launch LLM generations on APIBench, if the generation is interrupted midway, currently the partial file is deleted and restart generation from start. What we should instead do is check where the generation stopped and resume from the last generated index. This is specially critical for large models where generation can take >1 hours.

ID datapoint
2. Provider: This affects all 3: TorchHub/HuggingFace/PyTorch Hub

File that needs to be edited: https://github.com/ShishirPatil/gorilla/blob/main/eval/get_llm_responses.py

@ShishirPatil ShishirPatil added good first issue Good for newcomers apibench-data APIBench data labels Jun 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
apibench-data APIBench data good first issue Good for newcomers
Projects
None yet
Development

No branches or pull requests

1 participant