[v1.6] Fix the monitor_callback invalid issue during calibration with variable input shapes

[ciyongch]

Description

When doing calibration with variable input shapes, a new executor will be created here in the case of the current input has different shape compared to the previous one. While the callback function is only bound to the very first executor instead of passed down to the succeeding executors which shares the same symbol.
This PR enables passing down the callback function, to address the calibration skipping issue.

Checklist

Essentials

Please feel free to remove inapplicable items for your PR.

• The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant JIRA issue created (except PRs with tiny changes)
• Changes are complete (i.e. I finished coding on this PR)
• All changes have test coverage:
• Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
• Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
• Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
• Code is well-documented:
• For user-facing API changes, API doc string has been updated.
• For new C++ functions in header files, their functionalities and arguments are documented.
• For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
• Check the API doc at https://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
• To the best of my knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Changes

• Feature1, tests, (and when applicable, API doc)
• Feature2, tests, (and when applicable, API doc)

Comments

• If this change is a backward incompatible change, why must this change be made.
• Interesting edge cases to note here

@pengzhao-intel @TaoLv @ChaiBapchya @szha

[mxnet-bot]

Hey @ciyongch , Thanks for submitting the PR
All tests are already queued to run once. If tests fail, you can trigger one or more tests again with the following commands:

• To trigger all jobs: @mxnet-bot run ci [all]
• To trigger specific jobs: @mxnet-bot run ci [job1, job2]

---

CI supported jobs: [windows-gpu, website, centos-cpu, windows-cpu, unix-gpu, sanity, edge, miscellaneous, clang, centos-gpu, unix-cpu]

---

Note:
Only following 3 categories can trigger CI :PR Author, MXNet Committer, Jenkins Admin.
All CI tests must pass before the PR can be merged.

[pengzhao-intel]

Great job to root cause this bug :)

[ciyongch]

@ChaiBapchya @leezu it looks like there's CI issues in current v1.6.x, which was existed in previous commit #18586. Do you know if there's anyone working on this? Thanks!

edge

[2020-06-28T03:57:59.801Z] + python setup.py bdist_wheel --universal
[2020-06-28T03:57:59.801Z] Traceback (most recent call last):
[2020-06-28T03:57:59.801Z]   File "setup.py", line 23, in <module>
[2020-06-28T03:57:59.801Z]     from setuptools import find_packages # This must precede distutils
[2020-06-28T03:57:59.801Z] ImportError: No module named setuptools

unix-gpu

[2020-06-28T04:11:22.856Z] /work/runtime_functions.sh: line 2083: build_ubuntu_gpu_cuda101_cudnn7_mkldnn_cpp_test: command not found

[pengzhao-intel]

@sandeep-krishnamurthy @ChaiBapchya for helps :)

[ChaiBapchya]

@ciyongch @PatricZhao
Thelatest commit in 1.6 branch was erroneously merged before it passed CI. That commit tried to fix edge/centos-cpu/gpu pipelines.

Going forward, I created another PR on 1.6.x branch: #18597
That

• revert the erroneously merged commit
• fix centos link issue
• fix edge pipeline

However, it fails on setuptools as you pointed out. I'll try to get that fixed so that we can get the CI fixed for 1.6.x
Once merged we can rebase this PR.

[ChaiBapchya]

It passed all 11 why did we have to retrigger? Is codecov blocking merge?
Also we should try to use mxnet-bot for re-triggering specific pipelines if any.

[ciyongch]

Hi @ChaiBapchya , I saw the codecov test cases failed and the mxnet-bot doesn't support re-trigger. Not sure if they're a merge blocker or not, I just re-trigger the cases.

[ChaiBapchya]

I don't think that's the case.
@szha @leezu can confirm if code-cov is a blocker.
If its not a blocker, lets get this PR merged.
Also I'm guessing since this fix is made to executor,

1. this is part of other branches as well?
2. also this change doesn't have any test, is that already tested somewhere? how can we confirm?
   @ciyongch

[ciyongch]

@mxnet-bot run ci [unix-cpu]

[mxnet-bot]

Jenkins CI successfully triggered : [unix-cpu]

[ciyongch]

• this is part of other branches as well?

Yes, this is a common issue for all the current branches, I will do the backport to other branches as well.

• also this change doesn't have any test, is that already tested somewhere? how can we confirm?

Currently, we've only verified this via a customized case which is kind of complicated, I will try to add some tests later to cover it.

[ciyongch]

Codecov failures are still there...which shouldn't be the blocker I think.

[sandeep-krishnamurthy]

Codecov is not a blocker.

[sandeep-krishnamurthy]

@pengzhao-intel @TaoLv this will be good to go after your review and approval

[ChaiBapchya]

I will try to add some tests later to cover it.

Can we add a basic test to verify this? I guess reviewers would feel confident to approve this once they know there is a proper test to verify it and that it passes. @sandeep-krishnamurthy wdyt?

[leezu]

To "fix" the codecov showing up on the 1.x branches, you can include the 3 lines from https://github.com/apache/incubator-mxnet/pull/18497/files
cc @sandeep-krishnamurthy @ciyongch @ChaiBapchya

[ciyongch]

@ChaiBapchya We've verified the fix via an offline customized cases, anyway, it's quite reasonable to add a UT to cover this case. I will try to add this today.
@leezu thanks to point it out.

[pengzhao-intel]

LGTM

[ciyongch]

Hi @ChaiBapchya @leezu @pengzhao-intel @TaoLv , now all the CI passed and the UT is added as well, please help to merge, thanks.

[ChaiBapchya]

Thanks for adding the UT. LGTM!