This repository has been archived by the owner on Nov 17, 2023. It is now read-only.
Enable serializing/deserializing ndarrays in np_shape semantics #15090
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
zheng-da
approved these changes
May 29, 2019
szha
reviewed
May 29, 2019
|
@mxnet-label-bot add [NDArray] |
reminisce
force-pushed
the
loosen_ndarray_serialization_constraint
branch
from
ab68e7e
to
610606b
Compare
szha
approved these changes
May 31, 2019
|
@zheng-da Please test this PR in DGL to see if any test of saving/loading zero-size tensors is broken before merge. |
reminisce
changed the title
marcoabreu
reviewed
May 31, 2019
| @@ -1580,13 +1581,20 @@ static const uint32_t NDARRAY_V1_MAGIC = 0xF993fac8; | |||
| /* magic number for ndarray version 2, with storage type */ | |||
| static const uint32_t NDARRAY_V2_MAGIC = 0xF993fac9; | |||
|
|
|||
| // magic number for ndarray version 3, with np shape semantics. | |||
| // The ndarray must be saved and loaded within np shape semantics. | |||
| static const uint32_t NDARRAY_V3_MAGIC = 0xF993faca; | |||
There was a problem hiding this comment.
Do we have any tests for handling legacy storage types?
There was a problem hiding this comment.
|
@reminisce I checked with DGL. it works fine. |
haohuanw
pushed a commit
to haohuanw/incubator-mxnet
that referenced
this pull request
Jun 23, 2019
…he#15090) * Loosen the contraint on serializing/deserializing ndarrays within the scope of np_shape * Support save/load dense ndarrays in np_shape semantics
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
np_shapesemantics was introduced to support future NumPy operators where scalar tensors and zero-size tensors are common to see. Due to the concern on the potential issues of backward compatibility when this semantics is enabled, such as different handling on scalar tensors w/ or w/o this semantics, serializing/deserializing was simply marked as unsupported when this semantics is enabled.At the moment, DGL developers want to enable this semantics in their work to support zero-size tensors. Simply disabling serializing/deserializing ndarrays of all types: dense, sparse, zero-size, and scalars would make their unit tests fail in
np_shapesemantics.After careful consideration, we decided to loosen the constraint to support serialization/deserialization in the semantics of
np_shapefor ndarrays satisfying ALL the following three conditions as it would be the same as handling future NumPy ndarrays.(2, 0, 3).().Checklist
Essentials
Please feel free to remove inapplicable items for your PR.