Implement all miscellaneous ops

[connorgoggins]

Description

This PR serves to implement all miscellaneous (not fitting into preexisting categories) operators in OpPerf. To achieve this, I created a file (benchmark/opperf/nd_operations/misc_operators.py) with a function (run_mx_misc_operators_benchmarks) to run all miscellaneous ops. Within this function, I used calls to run_performance_test for the misc ops with positional arguments and a call to a new function I wrote (get_remaining_miscellaneous_operators) in conjunction with run_op_benchmarks for the misc ops with keyword args. I also added a call to run_mx_misc_operators_benchmarks in opperf.py.

I also added functionality to disable backwards runs on ops that do not support them (SequenceLast) and registered a custom operator inline to test the Custom op.

Checklist

Essentials

Please feel free to remove inapplicable items for your PR.

• Changes are complete (i.e. I finished coding on this PR)
• All changes have test coverage
• Code is well-documented
• To the best of my knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Changes

• A benchmark/opperf/nd_operations/misc_operators.py
• M benchmark/opperf/opperf.py
• M benchmark/opperf/rules/default_params.py
• M benchmark/opperf/utils/benchmark_utils.py
• M benchmark/opperf/utils/op_registry_utils.py

Comments

Tested on p2.16xl w/ubuntu 16.04 and Mac OS with:

1. Checkout branch and call function run_mx_misc_operators_benchmarks - runs all misc ops on relevant data
2. Checkout branch and run opperf.py (full run of all ops)

Performance Results

Group of operator test - all misc ops (CPU)
Full OpPerf test (CPU)

Group of operator test - all misc ops (GPU)
Full OpPerf test (GPU)

@apeforest @access2rohit @ChaiBapchya

[connorgoggins]

@mxnet-label-bot add [pr-awaiting-review]

[ChaiBapchya]

Looks like the alias mapping needs to get updated to search for the correct name in the MXNet profile.
For eg in case of Custom op, the profile will have CustomAddOne

[connorgoggins]

Custom op profiler output: https://gist.github.com/connorgoggins/84601f1fe836dc81eafeebb6b09678d2

[ChaiBapchya]

"Custom": "SetValueOp" still looks incorrect. Will have dive deep to find what is the correct way of benchmarking CustomOp (since it is user defined so maybe there's no right/wrong answer.

But SetValueOp is an operator call that will happen for every op.

For eg even when we run profiler for add this is how the profile for operator part looks

Name                          Total Count        Time (ms)    Min Time (ms)    Max Time (ms)    Avg Time (ms)
    ----                          -----------        ---------    -------------    -------------    -------------
    DeleteVariable                        196           1.4490           0.0040           0.0250           0.0074
    _backward_broadcast_add               100         521.2320           4.8070           8.5970           5.2123
    SetValueOp                            100         645.8060           5.8820          10.0380           6.4581
    broadcast_add                         100         394.8910           3.5230           5.8790           3.9489

So here as well you can see DeleteVar and SetValueOp. So they can't be counted per se for Custom or any op.

Makes sense?

[connorgoggins]

@ChaiBapchya makes sense. The reason I chose SetValueOp was because the only other option in the profiler output was DeleteVariable. Will dive deep on this to find the correct way of benchmarking Custom.

[ChaiBapchya]

LGTM

[ChaiBapchya]

Again since we have updated this over and over, would be great to have the results updated (especially group of misc operators) Thanks

[connorgoggins]

@ChaiBapchya all perf results are updated!

[access2rohit]

LGTM!

[apeforest]

Thanks for your contribution!