CCache does not work for NVCC or make-based builds

[marcoabreu]

At the moment, we don't apply ccache to NVCC compilation or to make-based builds in general. This is due to third parties not supporting our make-wrappers.

[marcoabreu]

Problems with NVCC - CCache log:

[2018-07-01T23:46:51.232513 6414 ] === CCACHE 3.4.2 STARTED =========================================
[2018-07-01T23:46:51.232513 6414 ] Config: (default) base_dir =
[2018-07-01T23:46:51.232513 6414 ] Config: (environment) cache_dir = /work/ccache
[2018-07-01T23:46:51.232513 6414 ] Config: (default) cache_dir_levels = 2
[2018-07-01T23:46:51.232513 6414 ] Config: (default) compiler =
[2018-07-01T23:46:51.232513 6414 ] Config: (default) compiler_check = mtime
[2018-07-01T23:46:51.232513 6414 ] Config: (default) compression = false
[2018-07-01T23:46:51.232513 6414 ] Config: (default) compression_level = 6
[2018-07-01T23:46:51.232513 6414 ] Config: (default) cpp_extension =
[2018-07-01T23:46:51.232513 6414 ] Config: (default) direct_mode = true
[2018-07-01T23:46:51.232513 6414 ] Config: (default) disable = false
[2018-07-01T23:46:51.232513 6414 ] Config: (default) extra_files_to_hash =
[2018-07-01T23:46:51.232513 6414 ] Config: (default) hard_link = false
[2018-07-01T23:46:51.232513 6414 ] Config: (default) hash_dir = true
[2018-07-01T23:46:51.232513 6414 ] Config: (default) ignore_headers_in_manifest =
[2018-07-01T23:46:51.232513 6414 ] Config: (default) keep_comments_cpp = false
[2018-07-01T23:46:51.232513 6414 ] Config: (default) limit_multiple = 0.8
[2018-07-01T23:46:51.232513 6414 ] Config: (environment) log_file = /work/mxnet/ccache-log
[2018-07-01T23:46:51.232513 6414 ] Config: (default) max_files = 0
[2018-07-01T23:46:51.232513 6414 ] Config: (environment) max_size = 500.0G
[2018-07-01T23:46:51.232513 6414 ] Config: (default) path =
[2018-07-01T23:46:51.232513 6414 ] Config: (default) prefix_command =
[2018-07-01T23:46:51.232513 6414 ] Config: (default) prefix_command_cpp =
[2018-07-01T23:46:51.232513 6414 ] Config: (default) read_only = false
[2018-07-01T23:46:51.232513 6414 ] Config: (default) read_only_direct = false
[2018-07-01T23:46:51.232513 6414 ] Config: (default) recache = false
[2018-07-01T23:46:51.232513 6414 ] Config: (default) run_second_cpp = true
[2018-07-01T23:46:51.232513 6414 ] Config: (default) sloppiness =
[2018-07-01T23:46:51.232513 6414 ] Config: (default) stats = true
[2018-07-01T23:46:51.232513 6414 ] Config: (environment) temporary_dir = /tmp/ccache
[2018-07-01T23:46:51.232513 6414 ] Config: (default) umask =
[2018-07-01T23:46:51.232513 6414 ] Config: (default) unify = false
[2018-07-01T23:46:51.232712 6414 ] Command line: nvcc -Werror cross-execution-space-call -std=c++11 -Xcompiler -D_FORCE_INLINES -O3 -ccbin g++ -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_60,code=sm_60 -gencode arch=compute_61,code=sm_61 -gencode arch=compute_70,code=[sm_70,compute_70] --fatbin-options -compress-all -Xcompiler -DMSHADOW_FORCE_STREAM -Wall -Wsign-compare -g -Werror -O3 -DNDEBUG=1 -I/work/mxnet/3rdparty/mshadow/ -I/work/mxnet/3rdparty/dmlc-core/include -fPIC -I/work/mxnet/3rdparty/tvm/nnvm/include -I/work/mxnet/3rdparty/dlpack/include -I/work/mxnet/3rdparty/tvm/include -Iinclude -funroll-loops -Wno-unused-parameter -Wno-unknown-pragmas -Wno-unused-local-typedefs -msse3 -mf16c -I/usr/local/cuda/include -DMSHADOW_USE_CBLAS=1 -DMSHADOW_USE_MKL=0 -DMSHADOW_RABIT_PS=0 -DMSHADOW_DIST_PS=0 -DMSHADOW_USE_PASCAL=0 -DMXNET_USE_OPENCV=1 -I/usr/include/opencv -fopenmp -DMXNET_USE_OPERATOR_TUNING=1 -DMXNET_USE_LAPACK -DMSHADOW_USE_CUDNN=1 -fno-builtin-malloc -fno-builtin-calloc -fno-builtin-realloc -fno-builtin-free -DUSE_JEMALLOC  -DMXNET_USE_DIST_KVSTORE -I/work/mxnet/3rdparty/ps-lite/include -I/work/mxnet/deps/include -I/work/mxnet/3rdparty/cub -DMXNET_ENABLE_CUDA_RTC=1 -DMXNET_USE_NCCL=0 -DMXNET_USE_LIBJPEG_TURBO=0 -M -MT build/src/operator/quantization/quantized_conv_gpu.o src/operator/quantization/quantized_conv.cu
[2018-07-01T23:46:51.232757 6414 ] Hostname: a31ae91b2cfb
[2018-07-01T23:46:51.232792 6414 ] Working directory: /work/mxnet
[2018-07-01T23:46:51.232831 6414 ] cross-execution-space-call is not a regular file, not considering as input file
[2018-07-01T23:46:51.232862 6414 ] arch=compute_30,code=sm_30 is not a regular file, not considering as input file
[2018-07-01T23:46:51.232881 6414 ] arch=compute_35,code=sm_35 is not a regular file, not considering as input file
[2018-07-01T23:46:51.232897 6414 ] arch=compute_50,code=sm_50 is not a regular file, not considering as input file
[2018-07-01T23:46:51.232913 6414 ] arch=compute_52,code=sm_52 is not a regular file, not considering as input file
[2018-07-01T23:46:51.232930 6414 ] arch=compute_60,code=sm_60 is not a regular file, not considering as input file
[2018-07-01T23:46:51.232947 6414 ] arch=compute_61,code=sm_61 is not a regular file, not considering as input file
[2018-07-01T23:46:51.232964 6414 ] arch=compute_70,code=[sm_70,compute_70] is not a regular file, not considering as input file
[2018-07-01T23:46:51.232984 6414 ] Compiler option -M is unsupported
[2018-07-01T23:46:51.233003 6414 ] Failed; falling back to running the real compiler
[2018-07-01T23:46:51.233016 6414 ] Executing /usr/local/cuda/bin/nvcc -Werror cross-execution-space-call -std=c++11 -Xcompiler -D_FORCE_INLINES -O3 -ccbin g++ -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_60,code=sm_60 -gencode arch=compute_61,code=sm_61 -gencode arch=compute_70,code=[sm_70,compute_70] --fatbin-options -compress-all -Xcompiler -DMSHADOW_FORCE_STREAM -Wall -Wsign-compare -g -Werror -O3 -DNDEBUG=1 -I/work/mxnet/3rdparty/mshadow/ -I/work/mxnet/3rdparty/dmlc-core/include -fPIC -I/work/mxnet/3rdparty/tvm/nnvm/include -I/work/mxnet/3rdparty/dlpack/include -I/work/mxnet/3rdparty/tvm/include -Iinclude -funroll-loops -Wno-unused-parameter -Wno-unknown-pragmas -Wno-unused-local-typedefs -msse3 -mf16c -I/usr/local/cuda/include -DMSHADOW_USE_CBLAS=1 -DMSHADOW_USE_MKL=0 -DMSHADOW_RABIT_PS=0 -DMSHADOW_DIST_PS=0 -DMSHADOW_USE_PASCAL=0 -DMXNET_USE_OPENCV=1 -I/usr/include/opencv -fopenmp -DMXNET_USE_OPERATOR_TUNING=1 -DMXNET_USE_LAPACK -DMSHADOW_USE_CUDNN=1 -fno-builtin-malloc -fno-builtin-calloc -fno-builtin-realloc -fno-builtin-free -DUSE_JEMALLOC  -DMXNET_USE_DIST_KVSTORE -I/work/mxnet/3rdparty/ps-lite/include -I/work/mxnet/deps/include -I/work/mxnet/3rdparty/cub -DMXNET_ENABLE_CUDA_RTC=1 -DMXNET_USE_NCCL=0 -DMXNET_USE_LIBJPEG_TURBO=0 -M -MT build/src/operator/quantization/quantized_conv_gpu.o src/operator/quantization/quantized_conv.cu
[2018-07-01T23:46:51.233082 6414 ] Acquired lock /work/ccache/6/stats.lock
[2018-07-01T23:46:51.233326 6414 ] Releasing lock /work/ccache/6/stats.lock
[2018-07-01T23:46:51.233347 6414 ] Unlink /work/ccache/6/stats.lock
[2018-07-01T23:46:51.233375 6414 ] Result: unsupported compiler option

Statement from CCache maintainers: https://stackoverflow.com/a/31587717

According to https://docs.nvidia.com/cuda/cuda-compiler-driver-nvcc/index.html#options-for-specifying-compilation-phase the -M flag here seems to mean something different:
``

--generate-dependencies	-M	Generate a dependency file that can be included in a make file for the.c/.cc/.cpp/.cxx/.cu input file (more than one are not allowed in this mode).
``

[frankfliu]

Thank you for submitting the issue! We are labeling it so MXNet community members can help resolve it.

[larroy]

Why are we writing a dependency file when compiling with CMake?

[KellenSunderland]

Can we use -MD instead of -M. Sounds like it might work around the issue https://stackoverflow.com/questions/29703938/ccache-doesnt-work-with-gcc-m-flag

[marcoabreu]

I pretty much got it to work in #11520

[KellenSunderland]

NICE

[larroy]

This is slowing down considerably looking for regressions on GPU as compilation takes a long time.

[larroy]

Still doesn't work with nvcc. I think we should leave open.

[leezu]

This is working after updating ccache to 3.7.9 ccache/ccache#563