Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Removed wrong docstrings for Metric update method's input type #2849

Conversation

sallycaoyu
Copy link
Contributor

@sallycaoyu sallycaoyu commented Feb 5, 2023

Fixes #2054

Description:

  1. Removed input form {'y_pred': y_pred, 'y': y, ...} from docstrings and from docs/source/metrics.rst for the ignite.Metric.update methods as Wrong docstrings for Metric update method's input type #2054 suggested, since the code does not support this input form.

  2. Added descriptions of update method's accepted input form and output_transform's usage in docstrings for ignite.Metric.PSNR and ignite.Metric.SSIM.

  3. Fixed a bug in ignite.metric.metric.py's CustomMetric example (line 172).

  4. Question: There exist two types of function signatures for update methods that take (y_pred, y) as their inputs:
    def update(self, output: Sequence[torch.Tensor]) -> None:, and
    def update(self, output: Tuple[torch.Tensor, torch.Tensor]) -> None: (in EpochMetric)
    Should this be made consistent somehow? I think the first option would allow users to pass either a tuple or a list as the input, whereas the second option will ask the input to stick to the tuple type.

Check list:

  • New tests are added (if a new feature is added)
  • New doc strings: description and/or example code are in RST format
  • Documentation is updated (if required)

@github-actions github-actions bot added docs module: metrics Metrics module labels Feb 5, 2023
Copy link
Collaborator

@vfdev-5 vfdev-5 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR @sallycaoyu
I left few comments. There is a difference in supported types between Metric.update and Metric.iteration_completed. As Engine's output goes first into Metric.iteration_completed which can handle dicts into tuple and pass this tuple into Metric.update. Seems like this is the source of the confusion... I understand that this is not ideal but I'm not what would be the best to fix that.

.gitignore Outdated Show resolved Hide resolved
docs/source/metrics.rst Outdated Show resolved Hide resolved
ignite/metrics/mean_absolute_error.py Outdated Show resolved Hide resolved
ignite/metrics/metric.py Outdated Show resolved Hide resolved
@sallycaoyu
Copy link
Contributor Author

sallycaoyu commented Feb 6, 2023

Thank you @vfdev-5 for the explanation! For now, I have made another commit to revert the changes except for the ones only for ignite.Metric.update.

For future improvements to this issue, I think one way would be to keep the code as-is and add more details for the docstring

- ``update`` must receive output of the form ``(y_pred, y)``.

...to explain how Metric.iteration_completed can help handle the dicts for ignite.Metric.update, and that ignite.Metric.update needs to take (y_pred, y) eventually, like:

- ``update`` must receive output of the form ``(y_pred, y)``. If ``update`` is automatically called by attaching the metric instance to the engine, then ``Metric.iteration_completed`` can convert the output it received to the form ``(y_pred, y)`` by default. If ``update`` is directly called, the output it received must be converted to the form``(y_pred, y)``.

Another way would be to change the code implementation, which I am still considering how to do. One way I could think of is to handle the dicts in the base Metric class instead of in Metric.iteration_completed, so that this statement will still be true:

- ``update`` must receive output of the form ``(y_pred, y)`` or ``{'y_pred': y_pred, 'y': y}``.

and other child classes can either run super(Metric, self).update() to handle the dicts as the base method defines, or ignore and override it if users want to build a custom method. But in this way, ignite.Metric.update will no longer be an abstract method, also I don't think this implementation will look as elegant as the current one does...

@vfdev-5
Copy link
Collaborator

vfdev-5 commented Feb 6, 2023

For future improvements to this issue, I think one way would be to keep the code as-is and add more details for the docstring

@sallycaoyu I agree on updating docs. Thanks for the updates !

Copy link
Collaborator

@vfdev-5 vfdev-5 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks @sallycaoyu !

@vfdev-5 vfdev-5 merged commit d2020e4 into pytorch:master Feb 8, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Wrong docstrings for Metric update method's input type
2 participants