Solver improvements: gather all training parameters variations in solver.prototxt

[IlyaOvodov]

Preface

Currently parameters, defining training processes, are spread over 3 locations: (a) initial weights are in command file ("caffe train --weights ..." parameter), (b) parameters concerning training data, their transformation and augmentation are in net.prototxt file and (c) path to net.prototxt file and other learning parameters are in solver.prototxt file. When you are experimenting with varying training parameters for the same network architecture this fact results in multiplication of copies of different files of 3 types and mess in their correspondence to each other and to training results.

This confusion can be partially resolved by using python scripts but it actually means developing some self-made training framework over Caffe.

I propose (and I am going to implement) some minor improvements to make it possible to locate all varying parameters in a single place (solver file) and simplifying keeping consistency between them and training results:

Proposed improvements:

• 1. Add parameter "weights" to solver parameters. This parameter has the same meaning as --weights command line parameters for "caffe train" command and allows to avoid changing command file. If --weights command line parameters is still specified, it has higher priority over parameter in solver file (mainly for compatibility).

• 2. If "snapshot_prefix" parameter is not set in solver.prototxt, solver file name without extention is used for it. It simplifies keeping information about parameters used to get some training results and keeping correspondence between them.

• 3. If "net" and "net_param" parameters are both present in solver.prototxt file, they are merged. "net_param" is applied over "net", unlike current behaviour when "net" parameters are ignored in such a case. As a result for example for changing image set for the same network architecture instead of creating new copy of net.prototxt, solver.prototxt file can be changed as follows:

MySolver.prototxt

...
net: "D:/CaffeNets/lenet_ext.prototxt"
net_param {
  layer {
    name: "data" 
    image_data_param {
      source: "C:/MyImages/ListVersion2.txt" # only  this parameter is changed in original net lenet_ext.prototxt
    }
  }
}
...

The same for train_net train_net_param and test_net/test_net_param parameters.
An only important aspect is that layers should be merged according to their names, not positions.
What about these ideas?

[Noiredd]

1 and 2 are definitely interesting and I will be happy to merge such PRs. For 3 however, this requires some code to overwrite a portion of specified net, layer by layer, name-wise - correct? This won't break any existing setups?

[IlyaOvodov]

As for 3, I expect to implement it in Solver::InitTrainNet() while preparing net_param before calling net_.reset(new Net(net_param)) as well as in Solver::InitTestNets() and I don't foresee now a nessesarity to touch code of layers. Possibly I'm too optimistic :). Let's see a result when it will be ready.

[IlyaOvodov]

Improvements (1) and (2) are implemented and merged to master with PR #6123.
Improvement (3) is not yet.