Revert "Improve FC perf when no_bias=False"

[roywei]

Reverts #15033
@stu1130 and I are running a binary search on the commits that may cause #15084

This seems to be the one. The reason causing CI build failure is still unknown, as this only happens with CI enviroments, builidng locally with GPU + MKLDNN is fine on master.

I m creating this PR to test if CI check can pass

[szha]

cc @anirudh2290

[anirudh2290]

Really weird! CI check passed even on my PR. So if CI check doesn't fail, does that tell much ?

[szha]

We certainly need to find the root cause on why CI wasn't reliable in this case. (e.g. could it have been just a cache issue?)

[larroy]

The root cause seems to be not enough shared memory in the docker container used in CI.

Using time ci/build.py --docker-registry mxnetci --platform ubuntu_build_cuda --docker-build-retries 3 --shm-size 8G /work/runtime_functions.sh build_ubuntu_gpu_mkldnn

fixes the build.

We have some files which are very slow to compile and produce big binaries, I think is excess of inline.

[larroy]

#15100

[szha]

Hmm, if this was required, how did the original PR pass CI check?

[anirudh2290]

I was able to reproduce this with make with on my local machine with command:

make DEV=1 ENABLE_TESTCOVERAGE=1 USE_CPP_PACKAGE=1 USE_BLAS=openblas USE_CUDA=1 USE_CUDA_PATH=/usr/local/cuda USE_CUDNN=1 'CUDA_ARCH=-gencode=arch=compute_52,code=sm_52 -gencode=arch=compute_70,code=sm_70' USE_SIGNAL_HANDLER=1 -j$(nproc)

Note: I was able to reproduce only with a clean build after make clean. for example, after building on top of previous commit i wasn't able to reproduce this.

[roywei]

@stu1130 has tried with ~2G and failed. It also seems 8G in #15100 failed.
I just tried ~10G passed

ci/build.py --docker-registry mxnetci --platform ubuntu_build_cuda --docker-build-retries 3 --shm-size 10000m /work/runtime_functions.sh build_ubuntu_gpu_mkldnn

We have tried with 500m before #15033, everything works fine. Everything after this commit failed with 500m.
For local tests, also able to reproduce now with @anirudh2290 's command.
So this can't be only shared memory related.

1. why Improve FC perf when no_bias=False #15033 code change caused CI requiring more space? (it changed requiring additional temp space to not limit to MKLDNN only, but should not affect build?)
2. why CI check passed for initial PR?
3. how to fix local build?

I would prefer to merge this and keep #15084 open until we find root cause.

[anirudh2290]

@roywei sounds good to me.

[aaronmarkham]

Is there are next step for those of use with PRs that failed CI due to this issue?

[anirudh2290]

@aaronmarkham Rebase and retrigger CI and it should be fine.

[larroy]

We need to investigate what's going on, I thought it passed for me at 8G but then my PR validation failed. Could be a linker bug or an offset linking bigger than 2^32, it's still not 100% clear to me.

[larroy]

I'd suggest strace dump of linker

[larroy]

Just failed with 16G for me.