Implementation for equivalence of tf.moments

[haojin2]

Description

https://www.tensorflow.org/api_docs/python/tf/nn/moments

Checklist

Essentials

Please feel free to remove inapplicable items for your PR.

• The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant JIRA issue created (except PRs with tiny changes)
• Changes are complete (i.e. I finished coding on this PR)
• All changes have test coverage:
• Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
• Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
• Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
• Code is well-documented:
• For user-facing API changes, API doc string has been updated.
• For new C++ functions in header files, their functionalities and arguments are documented.
• For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
• Check the API doc at http:https://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
• To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Changes

• Forward & Backward
• Unit Test

Comments

Related to #11283, pre-requisite of group norm implementation.
Flakiness Check:

MXNET_TEST_COUNT=10000 nosetests tests/python/unittest/test_operator.py:test_moments
[INFO] Setting module np/mx/python random seeds, use MXNET_MODULE_SEED=862626152 to reproduce.
.
----------------------------------------------------------------------
Ran 1 test in 1380.770s

OK

MXNET_TEST_COUNT=10000 nosetests tests/python/gpu/test_operator_gpu.py:test_moments
[INFO] Setting module np/mx/python random seeds, use MXNET_MODULE_SEED=688103580 to reproduce.
.
----------------------------------------------------------------------
Ran 1 test in 2116.433s

OK

[haojin2]

@eric-haibin-lin @reminisce @szha ping for review.

[wkcn]

Great! Is it convenient to support the tensor with arbitrary dimension? Thank you!

[haojin2]

One thing is that usually tensors do not have >5 dims. The kernel can actually handle any # of dims, but we need to modify the mxnet::TShape/Tuple class to enable GPUs to read from mxnet::TShape/Tuple.

[eric-haibin-lin]

Would you mind sharing some common use cases for this operator? Thanks!

[haojin2]

@eric-haibin-lin The op gives variance of the input on arbitrary axis/axes, so it's technically the np.var operator's equivalence with an extra mean output. The impl of this op could also become the common helper function for various versions of normalizations.
Plus: My current work on group norm is depending on the impl of this op or I'll have to write the same impl twice within the operator.

[haojin2]

@eric-haibin-lin @hetong007 Is this good for merge?

[hetong007]

LGTM

[haojin2]

@eric-haibin-lin @szha Can any of you take a final look and help with merging this op? Thanks!

[haojin2]

@eric-haibin-lin @szha Ping for one more review.