Update CustomOp doc with changes for GPU support

[rondogency]

Description

Improve the custom operator example doc by adding how to write GPU custom operator.

This is a continuation of the CustomOp tutorial doc #17241

This PR is not

• Updating docs on the MXNet website on extending MXNet
• Consolidating operator related docs on the website

Those will be addressed in the coming PRs

Checklist

Essentials

Please feel free to remove inapplicable items for your PR.

• The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant JIRA issue created (except PRs with tiny changes)
• Changes are complete (i.e. I finished coding on this PR)
• All changes have test coverage:
• Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
• Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
• Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
• Code is well-documented:
• For user-facing API changes, API doc string has been updated.
• For new C++ functions in header files, their functionalities and arguments are documented.
• For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
• Check the API doc at https://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
• To the best of my knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Changes

• Feature1, tests, (and when applicable, API doc)
• Feature2, tests, (and when applicable, API doc)

Comments

• If this change is a backward incompatible change, why must this change be made.
• Interesting edge cases to note here

[samskalicky]

@rondogency please add the fix for the gpu example test program here from #17270 (comment)

[rondogency]

@samskalicky the fix is in #17516 and you can review it now

[aaronmarkham]

Any idea why the formatting changes to italics here?
https://github.com/apache/incubator-mxnet/pull/17486/files/1ef0af458f8496f7a7f8d0e6d6300904afeb2b56#diff-1727b2876a104444973921644eac97bdR255

[samskalicky]

@aaronmarkham it looks fine to me here:
https://github.com/apache/incubator-mxnet/blob/9b93000c4a8f8da05e51547312bd605ec3798b3f/example/extensions/lib_custom_op/README.md

Looks like its just weird GitHub source-diff showing odd italics.

[a550461053]

Hi, @rondogency I am trying to use the dynamic operator and have some questions.
How can we use mxnet::NDArray or mshadow_op used in xxx-inl.h of mxnet-backend operator? And I want to know how to use the mutateInputs function like in BatchNorm-like operators.
Thank you very much.

[samskalicky]

Hi, @rondogency I am trying to use the dynamic operator and have some questions.
How can we use mxnet::NDArray or mshadow_op used in xxx-inl.h of mxnet-backend operator? And I want to know how to use the mutateInputs function like in BatchNorm-like operators.
Thank you very much.

Hi @a550461053, MXNet custom operators have been designed not to integrate with any MXNet backend classes (mxnet::NDArray) to simplify building. We've designed MXTensor to be equivalent. Is there some other functionality you were wanting from NDArray thats not in MXTensor?

The mutateInputs API allows you to do the same as the backend APIs, so you can specify the index of the input that you want to be mutable to make a BatchNorm-like operator.

[a550461053]

Hi, @rondogency I am trying to use the dynamic operator and have some questions.
How can we use mxnet::NDArray or mshadow_op used in xxx-inl.h of mxnet-backend operator? And I want to know how to use the mutateInputs function like in BatchNorm-like operators.
Thank you very much.

Hi @a550461053, MXNet custom operators have been designed not to integrate with any MXNet backend classes (mxnet::NDArray) to simplify building. We've designed MXTensor to be equivalent. Is there some other functionality you were wanting from NDArray thats not in MXTensor?

The mutateInputs API allows you to do the same as the backend APIs, so you can specify the index of the input that you want to be mutable to make a BatchNorm-like operator.

Ok, thank you. I want to create a custom operator calling another operator which input NDArray. Both operator is async pushed to engine, I think this way is possible. Also, I am not clear on using mutateInputs API, how and when to use it? If you can provide a example of the mutateInputs API, I will be grateful to you~

[samskalicky]

Ok, thank you. I want to create a custom operator calling another operator which input NDArray. Both operator is async pushed to engine, I think this way is possible. Also, I am not clear on using mutateInputs API, how and when to use it? If you can provide a example of the mutateInputs API, I will be grateful to you~

Hi @a550461053 currently the custom operator design focused on separating the custom operator from the MXNet backend source code complexity. This means that your custom operator can (must) be entirely separated from MXNet. So you cannot call a regular built-in MXNet operator from your custom operator. We have an item here #17006 for adding support in the future to be able to do this, but it is not implemented yet.

As for how to use mutateInputs, it works exactly as the doc describes:

This function allows you to mark some inputs to be mutable inputs. It is useful when using aux parameters for BatchNorm-like operators.

So lets say you have an operator with 5 inputs, you can mark the indices of the inputs that you want to be mutable like this (for example mark the last two inputs as mutable):

MXReturnValue batchNorm_mutateInputs(std::map<std::string, std::string> attrs,
                                     std::vector<int> &input_indices) {
  // mark mutable inputs                                                                        
  input_indices.push_back(3);
  input_indices.push_back(4);
  return MX_SUCCESS;
}

[rondogency]

@a550461053 Thanks for the questions. Currently you cannot call MXNet built-in operators from a custom operator. We are making a plan for adding this support, but it cannot be easily implemented and may be a separate project. Also I added a snippet of how to use mutable input, hope it would help.

[samskalicky]

LGTM, thanks @rondogency!

[a550461053]

Ok, thank you. I want to create a custom operator calling another operator which input NDArray. Both operator is async pushed to engine, I think this way is possible. Also, I am not clear on using mutateInputs API, how and when to use it? If you can provide a example of the mutateInputs API, I will be grateful to you~

Hi @a550461053 currently the custom operator design focused on separating the custom operator from the MXNet backend source code complexity. This means that your custom operator can (must) be entirely separated from MXNet. So you cannot call a regular built-in MXNet operator from your custom operator. We have an item here #17006 for adding support in the future to be able to do this, but it is not implemented yet.

As for how to use mutateInputs, it works exactly as the doc describes:

This function allows you to mark some inputs to be mutable inputs. It is useful when using aux parameters for BatchNorm-like operators.

So lets say you have an operator with 5 inputs, you can mark the indices of the inputs that you want to be mutable like this (for example mark the last two inputs as mutable):

MXReturnValue batchNorm_mutateInputs(std::map<std::string, std::string> attrs,
                                     std::vector<int> &input_indices) {
  // mark mutable inputs                                                                        
  input_indices.push_back(3);
  input_indices.push_back(4);
  return MX_SUCCESS;
}

Thank you and @rondogency . I am clear on use of mutateInputs API.
But now I can only build mxnet to create c++ customOp and I have a problem on compiling into python wheel at #17577

[samskalicky]

But now I can only build mxnet to create c++ customOp and I have a problem on compiling into python wheel at #17577

@a550461053 just to clarify, you do not need to recompile MXNet from source to use a custom operator, you can dynamically load your library into MXNet at runtime using one of the nightly builds from here: https://repo.mxnet.io/dist/index.html

[samskalicky]

@aaronmarkham or @wkcn can we merge this please?

[wkcn]

The PR has been merged. Thanks for your contribution! : )