[MXNET-554] Add ccache reporting to CI

[KellenSunderland]

Description

Adding some diagnostic ccache output to builds using ccache. This output will allow us to get a sense of both the global level of caching for the system, and will provide diagnostic information for the specific builds via the log files.

Checklist

Essentials

Please feel free to remove inapplicable items for your PR.

• The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant JIRA issue created (except PRs with tiny changes)
• Changes are complete (i.e. I finished coding on this PR)
• All changes have test coverage:
• Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
• Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
• Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
• Code is well-documented:
• For user-facing API changes, API doc string has been updated.
• For new C++ functions in header files, their functionalities and arguments are documented.
• For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
• Check the API doc at http:https://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
• To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Comments

• I've tried to provide a balance of information so that we can see what is actionable, but we don't fill our logs with verbose output.

[KellenSunderland]

@larroy Any tips for this change?

@marcoabreu How's this look for ccache diagnostic info?

[marcoabreu]

I like it general, I'm just afraid it's a bit too verbose since it increases the log by about 300%.

It would be a bit tedious, but we could try to reproduce ccache -s locally by parsing the log. We could aggregate the 'Result' lines:

Result: cache hit (direct)
Result: called for link
Result: no input file

wdyt?

Edit1: It seems like ccache -s is doing exactly that: https://ccache.samba.org/manual.html#_cache_statistics

[KellenSunderland]

Feel the same. I'd be in favour of leaving it as is while we're debugging ccache. Then once we're comfortable with it summarize the results via script.

…

On Sun, Jun 17, 2018, 7:14 PM Marco de Abreu ***@***.***> wrote: I like it general, I'm just afraid it's a bit too verbose since it increases the log by about 300%. It would be a bit tedious, but we could try to reproduce ccache -s locally by parsing the log. We could aggregate the 'Result' lines: Result: cache hit (direct) Result: called for link Result: no input file wdyt? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#11322 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHGTE9oHG2sZaVa_zIyj8hHIVtvTY63Aks5t9o5YgaJpZM4UqwGE> .

[marcoabreu]

Okay, sounds good to me.