[train] Remove base config deepcopy when initializing the trainer actor

[justinvyu]

Why are these changes needed?

merge_dicts first creates a deepcopy of the base config before doing a deep update. The deepcopy is unnecessary for this usage in Ray Train, so we can skip it and just perform a deep update.

This causes issues with large objects passed into the trainer, increasing the peak memory usage of the Ray Train coordinator actor (which is labeled _Inner in the Ray dashboard). For example, this problem surfaced for Ray Data datasets that held a lot of metadata being passed into the trainer. (The size of the datasets is a separate issue that will be fixed.)

The large objects are deep copied, which increases the memory usage of the Trainer actor by 2x. The original copy comes from the object store via tune.with_parameters. This copy should get garbage collected immediately after the Trainer Trainable setup is called, but for some reason the copy's memory usage sticks around for the rest of training.

Related issue number

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[matthewdeng]

lgtm