Use In-place operator to prevent memory spikes in optimizer updates

[anirudhacharya]

Description

The update rules in Nadam, Adadelta, Adamax and SGLD optimizers have been changed to using in-place operators to prevent memory spikes during execution.

Checklist

Essentials

Please feel free to remove inapplicable items for your PR.

• The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant JIRA issue created (except PRs with tiny changes)
• Changes are complete (i.e. I finished coding on this PR)
• All changes have test coverage:
• Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
• Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
• Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
• Code is well-documented:
• For user-facing API changes, API doc string has been updated.
• For new C++ functions in header files, their functionalities and arguments are documented.
• For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
• Check the API doc at http:https://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
• To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

[anirudhacharya]

Some statistics from profiler.dump() on running with an mnist example.

Adadelta Old

Profile Statistics.
	Note that counter items are counter values and not time units.
Device Storage
=================
Name                          Total Count        Time (ms)    Min Time (ms)    Max Time (ms)    Avg Time (ms)
----                          -----------        ---------    -------------    -------------    -------------
Memory: cpu/0                        1908         627.2000           0.0000        2007.0400        1003.5200

Adadelta New

Profile Statistics.
	Note that counter items are counter values and not time units.
Device Storage
=================
Name                          Total Count        Time (ms)    Min Time (ms)    Max Time (ms)    Avg Time (ms)
----                          -----------        ---------    -------------    -------------    -------------
Memory: cpu/0                        1764         627.2000           0.0000        1606.1440         803.0720

Adamax Old

Profile Statistics.
	Note that counter items are counter values and not time units.
Device Storage
=================
Name                          Total Count        Time (ms)    Min Time (ms)    Max Time (ms)    Avg Time (ms)
----                          -----------        ---------    -------------    -------------    -------------
Memory: cpu/0                        1728         627.2000           0.0000        2009.6000        1004.8000

Adamax New

Profile Statistics.
	Note that counter items are counter values and not time units.
Device Storage
=================
Name                          Total Count        Time (ms)    Min Time (ms)    Max Time (ms)    Avg Time (ms)
----                          -----------        ---------    -------------    -------------    -------------
Memory: cpu/0                        1656         627.2000           0.0000        1606.6560         803.3280

[anirudhacharya]

@mxnet-label-bot add [pr-awaiting-review]

@szha @eric-haibin-lin

[anirudhacharya]

For 5 batches of training with a deep embeddings example the decrease in memory consumption is ~0.27 factor for Nesterov Adam optimizer.

Old Nadam -

New Nadam( with in-place operators) -

[vandanavk]

LGTM

[vandanavk]

@mxnet-label-bot update [pr-awaiting-merge]

[ankkhedia]

@sandeep-krishnamurthy Could you please review/merge this PR?

[anirudhacharya]

Thanks for merging and thanks to @szha for the tip here - #13683 (comment)