This repository has been archived by the owner on Nov 17, 2023. It is now read-only.
softmax for fp16 with fp32 accumulator #14098
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@ptrendx @DickJC123 I'd appreciate your review on this. Thanks. |
ptrendx
suggested changes
Feb 8, 2019
|
I don't think we can change the return type of log_softmax in minor release. |
|
I will add an optional dtype option for this. |
|
@mxnet-label-bot add [Operator, pr-awaiting-review] |
larroy
reviewed
Feb 11, 2019
larroy
reviewed
Feb 11, 2019
larroy
reviewed
Feb 11, 2019
|
@larroy thanks. Accuracy would be tied to a model and a dataset and is thus harder to reason. The more immediate effect is it's harder to overflow when accumulating e^x. |
|
I intend to change the PR so that GradUseInOut only happens when dtype is specified and different from input type. |
szha
force-pushed
the
softmax_fp16
branch
4 times, most recently
from
bdc0d6d
to
704b0e7
Compare
eric-haibin-lin
suggested changes
Feb 13, 2019
eric-haibin-lin
approved these changes
Feb 15, 2019
|
LGTM. @ptrendx any other concerns? Would AMP skip the cast if softmax accumulation for fp16 inputs is done in fp32? |
jessr92
pushed a commit
to jessr92/incubator-mxnet
that referenced
this pull request
Feb 28, 2019
* softmax for fp16 with fp32 accumulator * return AType in kernel * add dtype * kernel * grad use in-out only when dtype override * simplify infer type * address comments
jessr92
pushed a commit
to jessr92/incubator-mxnet
that referenced
this pull request
Feb 28, 2019
* softmax for fp16 with fp32 accumulator * return AType in kernel * add dtype * kernel * grad use in-out only when dtype override * simplify infer type * address comments
jessr92
pushed a commit
to jessr92/incubator-mxnet
that referenced
this pull request
Feb 28, 2019
* softmax for fp16 with fp32 accumulator * return AType in kernel * add dtype * kernel * grad use in-out only when dtype override * simplify infer type * address comments
drivanov
pushed a commit
to drivanov/incubator-mxnet
that referenced
this pull request
Mar 4, 2019
* softmax for fp16 with fp32 accumulator * return AType in kernel * add dtype * kernel * grad use in-out only when dtype override * simplify infer type * address comments
vdantu
pushed a commit
to vdantu/incubator-mxnet
that referenced
this pull request
Mar 31, 2019
* softmax for fp16 with fp32 accumulator * return AType in kernel * add dtype * kernel * grad use in-out only when dtype override * simplify infer type * address comments
This was referenced Apr 17, 2019
8 tasks
haohuanw
pushed a commit
to haohuanw/incubator-mxnet
that referenced
this pull request
Jun 23, 2019
* softmax for fp16 with fp32 accumulator * return AType in kernel * add dtype * kernel * grad use in-out only when dtype override * simplify infer type * address comments
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
softmax for fp16 with fp32 accumulator
Checklist
Essentials
Please feel free to remove inapplicable items for your PR.
Changes