[OpPerf] Add norm, cast ops, remaining optimizer ops

[ChaiBapchya]

Description

Adds following ops to OpPerf [once all the OpPerf PRs get merged, all MXNet ops in NDArray namespace will be completed in OpPerf coverage (minus deprecated, _contrib ops and few other ops with known issues)]

• norm op to reduction category
• Following Optimizer ops
  • mp_nag_mom
  • nag_mom
  • lambd_update_phase_1 and 2
  • preloaded_multi_*
  • multi_*
• Cast ops
  • cast
  • amp_cast
  • amp_multicast

Checklist

Essentials

Please feel free to remove inapplicable items for your PR.

• Changes are complete (i.e. I finished coding on this PR)
• All changes have test coverage:
• Code is well-documented:
• To the best of my knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Comments

Ops falling in the category of multi_* and preloaded_multi_* were notorious
They don't have standard keyworded arguments (data=data)
They expect variable positional args (*data)
It was previously handled such that only 1 value of variable positional arg could be passed

Now, you can pass as many values for positional args as you want.
However for running run_performance_test, inputs have to be key:value pair

That is handled as args0, args1, arg2, etc
So as long as the name of the key starts with args it will be considered as input for the operator.

This is a very rare case that the operator doesn't take keyworded args. Most other ops in MXNet NDArray namespace take keyworded args.

[ChaiBapchya]

Category specific operators gist : https://gist.github.com/ChaiBapchya/b4b49632d845abd9a451ab37809c575b

Basically result of running

from benchmark.opperf.nd_operations.nn_optimizer_operators import run_optimizer_operators_benchmarks
run_optimizer_operators_benchmarks()

from benchmark.opperf.nd_operations.reduction_operators import run_mx_reduction_operators_benchmarks
run_mx_reduction_operators_benchmarks()

from benchmark.opperf.nd_operations.unary_operators import run_mx_unary_operators_benchmarks
run_mx_unary_operators_benchmarks()

[ChaiBapchya]

Entire OpPerf Suite
CPU results : https://gist.github.com/ChaiBapchya/1c26e5a904d9ce9342d61b29f195c5cf [old]

[connorgoggins]

Great implementation! Just one small edit.

[connorgoggins]

Also can we see updated perf results?

[ChaiBapchya]

Previously entire NDArray that was input used to get printed
This would not be rendered correctly on Markdown (come up as new lines).
Now after this fix, only the shape of NDArray is stored as a String (much more readable and rendered correctly on Markdown)

CPU : https://gist.github.com/ChaiBapchya/12d7fdd4ac15703e537aefbc8055c981
GPU : https://gist.github.com/ChaiBapchya/c62c7fff4deb115b64ec69b5c4aa6a7d

[ChaiBapchya]

@mxnet-label-bot add [pr-awaiting-review]

[connorgoggins]

Great job!

[ChaiBapchya]

Updated OpPerf results after dtype fix
CPU : https://gist.github.com/ChaiBapchya/dbe941cb65d493c6e435c68a6bbc5b19
GPU : https://gist.github.com/ChaiBapchya/479b3b8efb68398d290a5e5863c7809f

[apeforest]

Good work!