[Quantization]support exclude operators while quantization #15910

xinyu-intel · 2019-08-15T12:11:31Z

Description

Two functionally enhancement for quantization tool:

support exclude operators while quantization.
address Model Quantization with CUDNN #15796 , automatically exclude operator which is not registered with a compute function on the target device.

Checklist

Essentials

Please feel free to remove inapplicable items for your PR.

The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant JIRA issue created (except PRs with tiny changes)
Changes are complete (i.e. I finished coding on this PR)
All changes have test coverage:
Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
Code is well-documented:
For user-facing API changes, API doc string has been updated.
For new C++ functions in header files, their functionalities and arguments are documented.
For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
Check the API doc at https://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Changes

Feature1, tests, (and when applicable, API doc)
Feature2, tests, (and when applicable, API doc)

Comments

If this change is a backward incompatible change, why must this change be made.
Interesting edge cases to note here

pengzhao-intel · 2019-08-19T03:59:47Z

@ZhennanQin @ciyongch please take a review :)

pengzhao-intel · 2019-08-19T04:03:40Z

include/mxnet/c_api.h

+ * \param num_excluded_sym_names number of layers excluded from being quantized in the input symbol
+ * \param excluded_sym_names node names to be excluded from being quantized
+ * \param num_excluded_op_names number of operators excluded from being quantized in the input symbol
+ * \param excluded_op_names operator names to be excluded from being quantized


Is it possible to use one group of the parameter to implement two functionality in here?

some models may define layer names with specific style. so, it's not easy to group these two functions:(

ZhennanQin · 2019-08-19T04:46:58Z

src/operator/quantization/quantize_graph_pass.cc

+ auto qnode = q_ptr(node->attrs);
+ if (!isRegistered(qnode, dev_type)) {
+ LOG(INFO) << "Neither FCompute nor FComputeEx registered, " << node->op()->name
+ << " excluded automatically.";


excluded => is excluded

ZhennanQin · 2019-08-19T04:49:03Z

src/operator/quantization/quantize_graph_pass.cc

+ DFSVisit(subgraph_sym->outputs, [&](const nnvm::NodePtr& n) {
+ if (n->is_variable()) return;
+ if (excluded_nodes.count(n->attrs.name) ||
+ excluded_ops.count(node->op()->name)) {


Fused op is a new op, so we shouldn't check its inner node.

I found we cannot exclude fused conv layers when settingexcluded_op_names=['Convolution']. Is it necessary to check the inner node here?

ciyongch

Any test case to cover subgraph ops?

ciyongch · 2019-08-19T08:26:38Z

python/mxnet/contrib/quantization.py

- https://www.tensorflow.org/performance/quantization.
- The calibration implementation borrows the idea of Nvidia's 8-bit Inference with TensorRT:
- https://on-demand.gputechconf.com/gtc/2017/presentation/s7310-8-bit-inference-with-tensorrt.pdf
- and adapts the method to MXNet.


Keep this notes here for future reference.

file too long (>1000L):(

ciyongch · 2019-08-19T08:27:13Z

python/mxnet/contrib/quantization.py

- https://www.tensorflow.org/performance/quantization.
- The calibration implementation borrows the idea of Nvidia's 8-bit Inference with TensorRT:
- https://on-demand.gputechconf.com/gtc/2017/presentation/s7310-8-bit-inference-with-tensorrt.pdf
- and adapts the method to MXNet.


Keep this notes here for future reference.

file too long (>1000L):(

ciyongch · 2019-08-19T08:36:29Z

tests/python/quantization/test_quantization.py

@@ -678,6 +678,10 @@ def check_quantized_bn(data_shape, qdtype):

 @with_seed()
 def test_quantize_params():
+ if is_test_for_native_cpu():
+ print('skipped testing quantized_pooling for native cpu since it is not supported yet')


test name? testing quantize_params ?

pengzhao-intel · 2019-08-21T05:33:52Z

@ZhennanQin @ciyongch please take a final review :)

ciyongch

LGTM.

ZhennanQin

LGTM

pengzhao-intel

merging now and thanks for the contribution.

xinyu-intel added 4 commits August 14, 2019 19:21

skip unregistered op and add exclude_op_names

b14339c

wrap a function

a9b6b72

add testcase for excluded op

604696a

Merge remote-tracking branch 'upstream/master' into add_exclude_op

003a570

xinyu-intel requested review from anirudh2290, eric-haibin-lin and szha as code owners August 15, 2019 12:11

skip test for native cpu since all op will be excluded

8a68d98

pengzhao-intel added the MKLDNN label Aug 16, 2019

xinyu-intel added 3 commits August 16, 2019 14:32

trigger

38a3c2b

trigger again

3b3dd03

Merge remote-tracking branch 'upstream/master' into add_exclude_op

0f8a046

pengzhao-intel reviewed Aug 19, 2019

View reviewed changes

ZhennanQin reviewed Aug 19, 2019

View reviewed changes

revert check inner node op name

cbdd028

ciyongch reviewed Aug 19, 2019

View reviewed changes

xinyu-intel added 4 commits August 19, 2019 16:55

test subgraph

350e159

trigger

41b03b4

Merge remote-tracking branch 'upstream/master' into add_exclude_op

fec976b

trigger

8b3dfed

ciyongch approved these changes Aug 21, 2019

View reviewed changes

ZhennanQin approved these changes Aug 21, 2019

View reviewed changes

pengzhao-intel approved these changes Aug 21, 2019

View reviewed changes

pengzhao-intel merged commit 0b5526b into apache:master Aug 21, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Quantization]support exclude operators while quantization #15910

[Quantization]support exclude operators while quantization #15910

xinyu-intel commented Aug 15, 2019

pengzhao-intel commented Aug 19, 2019

pengzhao-intel Aug 19, 2019

xinyu-intel Aug 19, 2019

ZhennanQin Aug 19, 2019

ZhennanQin Aug 19, 2019

xinyu-intel Sep 6, 2019

ciyongch left a comment

ciyongch Aug 19, 2019

xinyu-intel Aug 19, 2019

ciyongch Aug 19, 2019

xinyu-intel Aug 19, 2019

ciyongch Aug 19, 2019

xinyu-intel Aug 19, 2019

pengzhao-intel commented Aug 21, 2019

ciyongch left a comment

ZhennanQin left a comment

pengzhao-intel left a comment

[Quantization]support exclude operators while quantization #15910

[Quantization]support exclude operators while quantization #15910

Conversation

xinyu-intel commented Aug 15, 2019

Description

Checklist

Essentials

Changes

Comments

pengzhao-intel commented Aug 19, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ciyongch left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pengzhao-intel commented Aug 21, 2019

ciyongch left a comment

Choose a reason for hiding this comment

ZhennanQin left a comment

Choose a reason for hiding this comment

pengzhao-intel left a comment

Choose a reason for hiding this comment