[MXNET-312] Added Matthew's Correlation Coefficient to metrics #10524

dabraude · 2018-04-12T11:20:13Z

Description

Added Matthew's Correlation Coefficient to metrics

Checklist

Essentials

Please feel free to remove inapplicable items for your PR.

The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant JIRA issue created (except PRs with tiny changes)
Changes are complete (i.e. I finished coding on this PR)
All changes have test coverage:
Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
Code is well-documented:
For user-facing API changes, API doc string has been updated.
For new C++ functions in header files, their functionalities and arguments are documented.
For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
Check the API doc at http:https://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Changes

Added MCC binary classification metric (and when applicable, API doc)

Comments

In documentation gave example where it shows the difference with F1

* send as char * fix bug on pull response, and rowsparse on worker side * three modes * default to mode 0 and add support for row sparse * refactor sparse * rowsparse numbytes fixes * WIP tests * update test sync * remove prints * refactoring * Revert "refactoring" This reverts commit 05ffa1b. * undo refactoring to keep PR simple * add wait to stored in pull default * lint fixes * undo static cast for recvblob * lint fixes * mode 1 changes * sparse bug fix dtype * mshadow default * remove unused var * remove debug statements * clearer variables, reduced multiplication, const vars * add const for more vars, comments * comment syntax, code watcher, test default val * remove unnecessary print in test * trigger ci * multi precision mode (debugging race condition) * working rsp pushes * finish multiprecision for row sparse * rename num-bytes * fix bug due to rename of numbytes, and remove debug logs * address comments * add integration test * trigger ci * integration test * integration test * fix path of script * update mshadow * disable f16c for amalgamation * fix amalgamation build * trigger ci * disable f16c for jetson

* changed url references from dmlc to apache/incubator-mxnet * prepping scala landing pages * infer api info added

* Fix infer_storage_type * Add test * Fix lint * Trigger CI

* add slice_like and doc * pass unittest and lint

* initial update on setting up scala ide with mxnet * moving images to web-data project * updated links to images; added readme for root folder * scala hello world feature added * workaround for make transitive error * fixed systempath * minor updates * table fix * added some spacing * more spacing

…apache#10510)

…to matthews

update

…into matthews

dabraude · 2018-04-12T11:37:36Z

@marcoabreu can you check the CI?

marcoabreu · 2018-04-12T11:42:20Z

What's the matter?

dabraude · 2018-04-12T11:44:45Z

I meant can you check the test I added is ok

marcoabreu

Lgtm

piiswrong · 2018-04-12T18:29:26Z

python/mxnet/metric.py

+ def __init__(self, name='mcc',
+ output_names=None, label_names=None, average="macro"):
+ self.average = average
+ self.metrics = _BinaryClassificationMetrics()


_average and _metrics

ThomasDelteil · 2018-04-12T20:33:57Z

python/mxnet/metric.py

+ While slower to compute the MCC can give insight that F1 or Accuracy cannot.
+ For instance, if the network always predicts the same result
+ then the MCC will immeadiately show this. The MCC is also symetric with respect
+ to positive and negative catagorisation, however, there needs to be both


categorisation*

Thanks fixed (and changed to US spelling)

dabraude · 2018-04-16T09:45:00Z

@piiswrong Would you mind updating your review?

dabraude · 2018-06-06T16:09:08Z

Hey is there anything I can fix on this?

szha · 2018-06-28T22:12:05Z

tests/python/unittest/test_metric.py

@@ -122,6 +123,54 @@ def test_f1():
 np.testing.assert_almost_equal(microF1.get()[1], fscore_total)
 np.testing.assert_almost_equal(macroF1.get()[1], (fscore1 + fscore2) / 2.)

+def test_mcc():


~~could you add a consistency test with Matthew's Correlation Coefficient in numpy?~~

never mind. I didn't realize that numpy doesn't have it.

I tested that this implementation of micro mcc is consistent with http:https://scikit-learn.org/stable/modules/generated/sklearn.metrics.matthews_corrcoef.html

…e#10524) * Remove Fermi from cmake (apache#10486) * updated R docs (apache#10473) * [MXNET-120] Float16 support for distributed training (apache#10183) * send as char * fix bug on pull response, and rowsparse on worker side * three modes * default to mode 0 and add support for row sparse * refactor sparse * rowsparse numbytes fixes * WIP tests * update test sync * remove prints * refactoring * Revert "refactoring" This reverts commit 05ffa1b. * undo refactoring to keep PR simple * add wait to stored in pull default * lint fixes * undo static cast for recvblob * lint fixes * mode 1 changes * sparse bug fix dtype * mshadow default * remove unused var * remove debug statements * clearer variables, reduced multiplication, const vars * add const for more vars, comments * comment syntax, code watcher, test default val * remove unnecessary print in test * trigger ci * multi precision mode (debugging race condition) * working rsp pushes * finish multiprecision for row sparse * rename num-bytes * fix bug due to rename of numbytes, and remove debug logs * address comments * add integration test * trigger ci * integration test * integration test * fix path of script * update mshadow * disable f16c for amalgamation * fix amalgamation build * trigger ci * disable f16c for jetson * Fix rat excludes (apache#10499) * MXNET-308 added missing license (apache#10497) * refactored example (apache#10484) * [MXNET-298] Scala Infer API docs landing page (apache#10474) * changed url references from dmlc to apache/incubator-mxnet * prepping scala landing pages * infer api info added * Fix infer storage type (apache#10507) * Fix infer_storage_type * Add test * Fix lint * Trigger CI * [MXNET-306] Add slice_like operator (apache#10491) * add slice_like and doc * pass unittest and lint * Minor simplifications in ci/build.py (apache#10496) * [MXNET-305] Scala tutorial table fix (apache#10488) * initial update on setting up scala ide with mxnet * moving images to web-data project * updated links to images; added readme for root folder * scala hello world feature added * workaround for make transitive error * fixed systempath * minor updates * table fix * added some spacing * more spacing * added ability to set search path for Accelerate library * [MXNET-311] change test needs a docker with sudo, hence image changed (apache#10510) * added new metric * changed back from branch to upstream * lint changed * Fixed typo * Clarified interpretation * Changes for variable names * fixed variable names * replay the unit tests * changed comment

cjolivier01 and others added 18 commits April 12, 2018 11:58

Remove Fermi from cmake (apache#10486)

095f4bc

updated R docs (apache#10473)

c3b45f3

Fix rat excludes (apache#10499)

4c9ef27

MXNET-308 added missing license (apache#10497)

4102a27

refactored example (apache#10484)

e78a416

[MXNET-298] Scala Infer API docs landing page (apache#10474)

ac74371

* changed url references from dmlc to apache/incubator-mxnet * prepping scala landing pages * infer api info added

Fix infer storage type (apache#10507)

b43954d

* Fix infer_storage_type * Add test * Fix lint * Trigger CI

[MXNET-306] Add slice_like operator (apache#10491)

7409468

* add slice_like and doc * pass unittest and lint

Minor simplifications in ci/build.py (apache#10496)

0283cb1

[MXNET-311] change test needs a docker with sudo, hence image changed (…

e8d6b8d

…apache#10510)

added ability to set search path for Accelerate library

2670736

added new metric

8b73cab

Merge branch 'master' of https://github.com/apache/incubator-mxnet in…

7c9f77a

…to matthews

Merge pull request #4 from apache/master

5718e7d

update

changed back from branch to upstream

cbb32f3

Merge branch 'matthews' of https://github.com/dabraude/incubator-mxnet …

7ba9239

…into matthews

dabraude requested a review from szha as a code owner April 12, 2018 11:20

lint changed

4bf7813

marcoabreu approved these changes Apr 12, 2018

View reviewed changes

dabraude added 2 commits April 12, 2018 14:15

Fixed typo

4ab13b1

Clarified interpretation

fb1048a

piiswrong suggested changes Apr 12, 2018

View reviewed changes

ThomasDelteil reviewed Apr 12, 2018

View reviewed changes

dabraude added 2 commits April 12, 2018 22:20

Changes for variable names

a8157fe

fixed variable names

cdf3bcb

dabraude added 3 commits June 6, 2018 09:25

Merge branch 'master' into matthews

98c52ca

replay the unit tests

264de7b

changed comment

5ab3849

szha reviewed Jun 28, 2018

View reviewed changes

piiswrong merged commit 044afa3 into apache:master Jun 28, 2018

dabraude deleted the matthews branch June 29, 2018 05:10

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[MXNET-312] Added Matthew's Correlation Coefficient to metrics #10524

[MXNET-312] Added Matthew's Correlation Coefficient to metrics #10524

dabraude commented Apr 12, 2018 •

edited

Loading

dabraude commented Apr 12, 2018

marcoabreu commented Apr 12, 2018

dabraude commented Apr 12, 2018

marcoabreu left a comment

piiswrong Apr 12, 2018

dabraude Apr 12, 2018

ThomasDelteil Apr 12, 2018

dabraude Apr 12, 2018

dabraude commented Apr 16, 2018

dabraude commented Jun 6, 2018

szha Jun 28, 2018 •

edited

Loading

szha Jun 28, 2018

szha Jun 28, 2018

[MXNET-312] Added Matthew's Correlation Coefficient to metrics #10524

[MXNET-312] Added Matthew's Correlation Coefficient to metrics #10524

Conversation

dabraude commented Apr 12, 2018 • edited Loading

Description

Checklist

Essentials

Changes

Comments

dabraude commented Apr 12, 2018

marcoabreu commented Apr 12, 2018

dabraude commented Apr 12, 2018

marcoabreu left a comment

Choose a reason for hiding this comment

piiswrong Apr 12, 2018

Choose a reason for hiding this comment

dabraude Apr 12, 2018

Choose a reason for hiding this comment

ThomasDelteil Apr 12, 2018

Choose a reason for hiding this comment

dabraude Apr 12, 2018

Choose a reason for hiding this comment

dabraude commented Apr 16, 2018

dabraude commented Jun 6, 2018

szha Jun 28, 2018 • edited Loading

Choose a reason for hiding this comment

szha Jun 28, 2018

Choose a reason for hiding this comment

szha Jun 28, 2018

Choose a reason for hiding this comment

dabraude commented Apr 12, 2018 •

edited

Loading

szha Jun 28, 2018 •

edited

Loading