-
Notifications
You must be signed in to change notification settings - Fork 157
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Effect of evaluate!(mach, ...)
is unpredictable. Retrain on all data and re-evaluate?
#1027
Comments
I think that this is a fantastic change. As you said, the current |
If we set aside learning networks, it seems like we don't really need Is there a way to write a non-mutating |
I agree with @CameronBieganek that it would have been best if |
This is great feedback, thanks. I'm torn between dumping mutation altogether (advantage: simplicity) and proceeding as I propose above (convenience). Some context: In a summer project we are working on auto logging of workflows using MLFlow, and in this context it seemed natural to log a training score, and a serialised set of learned parameters, each time a model is "evaluated". This saves having to specify metrics a second time if wanting the training score.
@CameronBieganek Not sure what you mean here. Perhaps you mean calling |
Okay, I now remember the reason for the existing behaviour. The use case is evaluating models that support "warm restart". If using On another matter, perhaps a better way to get training scores, is to add a resampling strategy |
Currently,
evaluate!(mach, ...)
will mutatemach
becausemach
is repeatedly retrained on different views of the data;mach
is never trained on all supplied data (or all specifiedrows=...
).But there is really no reason for the user to guess the final state of the
mach
, as she is unlikely to know what the last train/test fold is without looking at the code, especially in the case ofacceleration=CPUParallel
(where I think it is impossible) oracceleration=CPUThreads
.So,
mach
, after theevaluate!(mach, ...)
is in an ambiguous state, and so kind of useless unless the user explicitly retrains.I wonder what others feel of the following suggestion:
evaluate!(mach, ...)
performs it's current function,mach
is trained one last time on all specifiedrows
(all rows by default) and the metrics are evaluated one last time to get training scores, which are added to the returnedPerformanceEvaluation
object.retrain=true
which can be set tofalse
to suppress the final retrain.Probably we add the keyword option to
evaluate(model, data...)
as well, but maybe withretrain=false
default??@ExpandingMan @OkonSamuel
The text was updated successfully, but these errors were encountered: