Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Batch Inference off of previously trained (and saved) model #350

Open
johnml1135 opened this issue Mar 26, 2024 · 0 comments
Open

Batch Inference off of previously trained (and saved) model #350

johnml1135 opened this issue Mar 26, 2024 · 0 comments
Assignees
Labels
enhancement New feature or request

Comments

@johnml1135
Copy link
Collaborator

We now have the ability to save models from #268. There are users that may want to do a quick inference off of a previously saved model, (rather than retraining). This would reduce the time to get new drafts from 8 hours to under 15 minutes (5 minutes?). This would require:

  • API design in Serval to indicate "just inference, don't retrain"
  • Scripture Forge work to choose "just inference, don't retrain"
  • Machine.py updates to enable referencing the previously saved model

API design

  • The Nmt engine is marked for persistence.
  • For build options:
    • overloard parent_model_name so that if it is set to saved
    • set train_params.do_train to false.
  • Choose new queue that includes the 3090's for fast inference - needs: New NMT option - choose ClearML queue machine#151

SF design:

  • Some form of toggle to choose previous build or do training afresh (and communicate 8 hours vs. 15 minutes)

Should it be auto-magical?

  • What about a pop-up to the user in SF that no (or minimal) training text has changed - do they want to just inference?
@johnml1135 johnml1135 added the enhancement New feature or request label Mar 26, 2024
@johnml1135 johnml1135 changed the title Inference off of previously trained (and saved) model Batch Inference off of previously trained (and saved) model Mar 26, 2024
@johnml1135 johnml1135 self-assigned this Mar 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
Status: 🆕 New
Development

No branches or pull requests

2 participants