-
-
Notifications
You must be signed in to change notification settings - Fork 610
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ddp precision recall #1646
ddp precision recall #1646
Conversation
pytorch#1612) * For v0.4.3 - Add more versionadded, versionchanged tags - Change v0.5.0 to v0.4.3 * Update ignite/contrib/metrics/regression/canberra_metric.py Co-authored-by: vfdev <[email protected]> * Update ignite/contrib/metrics/regression/manhattan_distance.py Co-authored-by: vfdev <[email protected]> * Update ignite/contrib/metrics/regression/r2_score.py Co-authored-by: vfdev <[email protected]> * Update ignite/handlers/checkpoint.py Co-authored-by: vfdev <[email protected]> * address PR comments Co-authored-by: vfdev <[email protected]>
* added TimeLimit handler with its test and doc * fixed documentation * fixed docstring and formatting * flake8 fix trailing whitespace :) * modified class logger , default value and tests * changed rounding to nearest integer * tests refactored , docs modified * fixed default value , removed global logger * fixing formatting * Added versionadded * added test for engine termination Co-authored-by: vfdev <[email protected]>
* Fixes pytorch#1614 - Updated handlers EarlyStopping and TerminateOnNan - Replaced `logging.getLogger` with `setup_logger` in the mentioned handlers * Updated `TimeLimit` handler. Replaced use of `logger.getLogger` with `setup_logger` from `ignite.utils` Co-authored-by: Pradyumna Rahul K <[email protected]> Co-authored-by: Sylvain Desroziers <[email protected]>
* Starter code for managing deprecation * Make functions deprecated using the `@deprecated` decorator * Add arguments to the @deprecated decorator to customize it for each function * Improve `@deprecated` decorator and add tests * Replaced the `raise` keyword with added `warnings` * Added tests several possibilities of the decorator usage * Removing the test deprecation to check tests * Add static typing, fix mypy errors * Make `@deprecated` to raise Exceptions or Warning * The `@deprecated` decorator will now always emit warning unless explicitly asked to raise an Exception * Fix mypy errors * Fix mypy errors (hopefully) * Fix the test `test_deprecated_setup_any_logging` * Change the test to work with the `@deprecated` decorator * Change to snake_case, handle mypy ignores * Improve Type Annotations * Update common.py * For v0.4.3 - Add more versionadded, versionchanged tags - Change v0.5… (pytorch#1612) * For v0.4.3 - Add more versionadded, versionchanged tags - Change v0.5.0 to v0.4.3 * Update ignite/contrib/metrics/regression/canberra_metric.py Co-authored-by: vfdev <[email protected]> * Update ignite/contrib/metrics/regression/manhattan_distance.py Co-authored-by: vfdev <[email protected]> * Update ignite/contrib/metrics/regression/r2_score.py Co-authored-by: vfdev <[email protected]> * Update ignite/handlers/checkpoint.py Co-authored-by: vfdev <[email protected]> * address PR comments Co-authored-by: vfdev <[email protected]> * `version` -> version Co-authored-by: vfdev <[email protected]> Co-authored-by: François COKELAER <[email protected]> Co-authored-by: Sylvain Desroziers <[email protected]>
…1620) * modified CONTRIBUTING.md * bash instead of sh
* Added Checkpoint.get_default_score_fn to simplify best_model_handler creation * Added score_sign argument * Updated docs
* Change pre-commit config and CONTRIBUTING.md - Update hook versions - Remove seed-isort-config - Add black profile to isort * Fix files based on new pre-commit config * Add meaningful exclusions to prettier - Also update actions workflow files to match local pre-commit
* added requirements.txt and updated readme.md * Update examples/contrib/cifar10/README.md Co-authored-by: vfdev <[email protected]> * Update examples/contrib/cifar10/requirements.txt Co-authored-by: vfdev <[email protected]> Co-authored-by: vfdev <[email protected]>
* Updates for cifar10 example * Updates for cifar10 example * More updates * Updated code * Fixed code-formatting
* Updates for cifar10 example * Updates for cifar10 example * More updates * Updated code * Fixed code-formatting * Fixed typo and failing CI * Fixed hvd spawn fail and better synced qat code
- updated default pth image for gpu tests - updated TORCH_CUDA_ARCH_LIST - fixed /merge -> /head in trigger ci pipeline
* [docker] Pillow -> Pillow-SIMD (pytorch#1509) * [docker] Pillow -> Pillow-SIMD * replace pillow with pillow-simd in base docker files * chore(docker): apt-get autoremove after pillow-simd installation * apt-get install at once, autoremove g++ * install g++ in pillow installation layer Co-authored-by: Sylvain Desroziers <[email protected]> * Fix g++ install issue Co-authored-by: Jeff Yang <[email protected]> Co-authored-by: Sylvain Desroziers <[email protected]>
* fix run_multinode_tests_in_docker.sh : run tests with docker python version * add missing modules * build an image with test env and add 'nnodes' 'nproc_per_node' 'gpu' as parameters * pytorch#1615 : change nproc_per_node default to 4 * pytorch#1615 : fix for gpu enabled tests / container rm step at the end of the script * add xfail decorator for tests/ignite/engine/test_deterministic.py::test_multinode_distrib_cpu * fix script gpu_options * add default tol=1e-6 for _test_distrib_compute_on_criterion * fix for "RuntimeError: trying to initialize the default process group twice!" * tolerance for test_multinode_distrib_cpu case only * fix assert None error * autopep8 fix Co-authored-by: vfdev <[email protected]> Co-authored-by: Sylvain Desroziers <[email protected]> Co-authored-by: fco-dv <[email protected]>
@fco-dv well done ! Please could you update docstring ? ignite/ignite/metrics/recall.py Line 53 in 63a6f1b
ignite/ignite/metrics/precision.py Line 112 in 63a6f1b
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the PR @fco-dv ! Please, take a look at the comment on how to improve the test.
): | ||
pr = Precision(average=False, is_multilabel=True) | ||
|
||
pr = Precision(average=False, is_multilabel=True) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
First of all, could you please fix the issue we spotted during the call with unused metric_device
that should go to Precision
definition (line 795).
Ultimately, we would like to enable the tests in the same way as test_multilabel_input_NCHW
where we run _test(average=True, ...)
and _test(average=False, ...)
. In case of average False, there is no equivalent option in sklearn (as probably it does not make much sense neither). So, please, take a look how we test things in test_multilabel_input_NCHW
and do same thing here. Then we can simply remove this dummy test below.
PS.: average=False and is_multilabel=True gives as a result for precision = multilabel precision per sample
import torch
from ignite.metrics import Precision
pr = Precision(average=False, is_multilabel=True)
pr.update((torch.tensor(y_pred[:5, :]), torch.tensor(y_true[:5, :])))
pr.update((torch.tensor(y_pred[5:, :]), torch.tensor(y_true[5:, :])))
print(pr._true_positives, pr._positives, pr.compute(), pr.compute().mean().item())
> (tensor([2., 2., 2., 2., 2., 2., 1., 0.], dtype=torch.float64),
tensor([2., 3., 2., 2., 2., 2., 1., 1.], dtype=torch.float64),
tensor([1.0000, 0.6667, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 0.0000],
dtype=torch.float64),
0.8333333333333333)
@fco-dv could you please udpate the tests according the suggestion: #1646 (comment) |
…ilabel_input_NCHW
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM! Thanks @fco-dv !
Fixes #1283
Description:
Enable the case for Recall/Precision in multi_label, not averaged configuration for DDP
Remove associated warnings, tests updated by scaling output size with
idist.get_world_size()
usage of
metric_device
should be checked inignite/tests/ignite/metrics/test_precision.py
Line 771 in 002b595
Check list:
New tests are added (if a new feature is added)
New doc strings: description and/or example code are in RST format
Documentation is updated (if required)