-
Notifications
You must be signed in to change notification settings - Fork 18.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Solver improvements: gather all training parameters variations in solver.prototxt #6110
Comments
1 and 2 are definitely interesting and I will be happy to merge such PRs. For 3 however, this requires some code to overwrite a portion of specified net, layer by layer, name-wise - correct? This won't break any existing setups? |
As for 3, I expect to implement it in Solver::InitTrainNet() while preparing net_param before calling net_.reset(new Net(net_param)) as well as in Solver::InitTestNets() and I don't foresee now a nessesarity to touch code of layers. Possibly I'm too optimistic :). Let's see a result when it will be ready. |
…points to a directory. See issue BVLC#6110 proposed improvement No.2
…points to a directory. See issue BVLC#6110 proposed improvement No.2
…points to a directory. See issue BVLC#6110 proposed improvement No.2
Improvements (1) and (2) are implemented and merged to master with PR #6123. |
…points to a directory. See issue BVLC#6110 proposed improvement No.2
…points to a directory. See issue BVLC#6110 proposed improvement No.2
Preface
Currently parameters, defining training processes, are spread over 3 locations: (a) initial weights are in command file ("caffe train --weights ..." parameter), (b) parameters concerning training data, their transformation and augmentation are in net.prototxt file and (c) path to net.prototxt file and other learning parameters are in solver.prototxt file. When you are experimenting with varying training parameters for the same network architecture this fact results in multiplication of copies of different files of 3 types and mess in their correspondence to each other and to training results.
This confusion can be partially resolved by using python scripts but it actually means developing some self-made training framework over Caffe.
I propose (and I am going to implement) some minor improvements to make it possible to locate all varying parameters in a single place (solver file) and simplifying keeping consistency between them and training results:
Proposed improvements:
1. Add parameter "weights" to solver parameters. This parameter has the same meaning as --weights command line parameters for "caffe train" command and allows to avoid changing command file. If --weights command line parameters is still specified, it has higher priority over parameter in solver file (mainly for compatibility).
2. If "snapshot_prefix" parameter is not set in solver.prototxt, solver file name without extention is used for it. It simplifies keeping information about parameters used to get some training results and keeping correspondence between them.
3. If "net" and "net_param" parameters are both present in solver.prototxt file, they are merged. "net_param" is applied over "net", unlike current behaviour when "net" parameters are ignored in such a case. As a result for example for changing image set for the same network architecture instead of creating new copy of net.prototxt, solver.prototxt file can be changed as follows:
MySolver.prototxt
The same for train_net train_net_param and test_net/test_net_param parameters.
An only important aspect is that layers should be merged according to their names, not positions.
What about these ideas?
The text was updated successfully, but these errors were encountered: