This repository has been archived by the owner on Nov 17, 2023. It is now read-only.
[MXNET-539] Allow Scala users to specify data/label names for NDArrayIter #11256
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
nswamy
reviewed
Jun 14, 2018
| if (!label.isEmpty) { | ||
| val labelList = label.map(ndArray => {ndArray.slice(0, keepSize)}) | ||
| val labelList = label.map { case (name, ndArray) => (name, ndArray.slice(0, keepSize)) } |
There was a problem hiding this comment.
how will this slice handler be disposed?
| require(cursor < numData, "DataIter needs reset.") | ||
| if (data == null) { | ||
| null | ||
| } else { | ||
| if (cursor + dataBatchSize <= numData) { | ||
| data.map(ndArray => {ndArray.slice(cursor, cursor + dataBatchSize)}).toIndexedSeq | ||
| data.map { case (_, ndArray) => ndArray.slice(cursor, cursor + dataBatchSize) } |
There was a problem hiding this comment.
Is it the user's(of this API) responsiblity to dispose the slice handler?, how would he know that.
There was a problem hiding this comment.
In next() it returns DataBatch and DataBatch can be disposed thus dispose the sliced data/label. For now it is a standard process for DataIters. Users can decide to use the returned DataBatch later thus we cannot dispose them here.
nswamy
approved these changes
Jun 15, 2018
zheng-da
pushed a commit
to zheng-da/incubator-mxnet
that referenced
this pull request
Jun 28, 2018
…Iter (apache#11256) * improve NDArrayIter to have Builder and ability to specifying names
XinYao1994
pushed a commit
to XinYao1994/incubator-mxnet
that referenced
this pull request
Aug 29, 2018
…Iter (apache#11256) * improve NDArrayIter to have Builder and ability to specifying names
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
So far Scala users cannot specify the names for NDArrayIter - they are inferred automatically (data_0, data_1, ..., etc)
We should allow users to do that since the input names are from model.
A builder class added so as to simplify object construction.
Checklist
Essentials
Please feel free to remove inapplicable items for your PR.