This repository has been archived by the owner on Dec 16, 2022. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 2.2k
Data V2 #3700
Merged
Merged
Data V2 #3700
Changes from 1 commit
Commits
Show all changes
59 commits
Select commit
Hold shift + click to select a range
0c42cb9
example for feedback
DeNeutoy 5ffedfc
Merge branch 'master' into data-v2
DeNeutoy 80049f8
remove all existing multiprocessing
DeNeutoy 6f58c2a
sneak torch datasets inside DatasetReader
DeNeutoy 1b3ad9a
lint
DeNeutoy effc445
trainer_v2, We Love To See It
DeNeutoy 9d44ad6
datasets have index_with now, not iterators
DeNeutoy 7e89ea6
use iter, custom collate function in allennlp wrapper
DeNeutoy 883b6d7
we don't even need the data in the trainer anymore
DeNeutoy 56d022a
all trainer tests passing
DeNeutoy 01e12f5
black
DeNeutoy 5aea291
make find learning rate work
DeNeutoy f026946
update test fixtures to new config
DeNeutoy 5973b50
get train command tests mostly working
DeNeutoy a23f47a
lazily construct samplers, index lazy datasets
DeNeutoy a76ea0a
Merge branch 'master' into data-v2
DeNeutoy ebf3854
update some fixtures
DeNeutoy 57a67e5
evaluate tests passing
DeNeutoy 7d21ed8
all command tests passing
DeNeutoy 24a500c
lint
DeNeutoy fb13769
update model test case, common and module tests passing
DeNeutoy ef5187f
fix test interdependence introduced by #3762
DeNeutoy b1ea845
more test interdependence
DeNeutoy 0231616
tests tests tests
DeNeutoy 01d76bb
remove unnecessary brackets
DeNeutoy 12b6efb
Merge branch 'master' into data-v2
DeNeutoy 859d3ca
update a chunk of the configs
DeNeutoy c22dee3
fix archival test, couple more configs
DeNeutoy fe5b470
rm pointless gan test
DeNeutoy 7533c91
more tests passing
DeNeutoy ad45659
add current state of from params changes
DeNeutoy f944840
Revert "add current state of from params changes"
DeNeutoy 3b12a2f
Merge branch 'master' into data-v2
DeNeutoy be1f58c
updated understanding of Lazy
DeNeutoy ebdabe0
add discussion of None comparison to Lazy
DeNeutoy 8693739
lint
DeNeutoy b9b0650
it's a hard doc life
DeNeutoy 88314c7
pull samplers into separate file
DeNeutoy 14296a1
more docs updates
DeNeutoy 8a08899
fold in #3812
DeNeutoy 3520280
remove torch dataset
DeNeutoy 0f1d8a4
add example to lazy
DeNeutoy 93e1e89
rename to collate
DeNeutoy 40dd695
no kwargs
DeNeutoy da3b1b4
Revert "fold in #3812"
DeNeutoy 801a8f5
don't break up dataset
DeNeutoy 007fd0c
add comment to iterable dataset len
DeNeutoy d00e1a9
Merge branch 'master' into data-v2
DeNeutoy c066804
improve docstrings, build dataloader using partial_objects
DeNeutoy 61c7b14
flake
DeNeutoy 2b56b14
give dataloader a default implementation
DeNeutoy 354010a
safer default for DataLoader init
DeNeutoy 568291d
more coherent dir structure
DeNeutoy a016103
update imports
DeNeutoy 47db16a
Merge branch 'master' into data-v2
DeNeutoy 04fdb70
add a test for the BucketBatchSampler
DeNeutoy d1d5c4a
split bucket sampler into own file, tests
DeNeutoy 5f0c8db
PR comments
DeNeutoy 6f63a53
Merge branch 'master' into data-v2
DeNeutoy File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
flake
- Loading branch information
commit 61c7b14a8081c01427ccc34d8d38a75c4d119751
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -36,16 +36,19 @@ def __iter__(self) -> Iterable[List[int]]: | |
@Sampler.register("sequential") | ||
class SequentialSampler(Sampler, data.SequentialSampler): | ||
""" | ||
A registerable version of pytorch's [SequentialSampler](https://pytorch.org/docs/stable/data.html#torch.utils.data.SequentialSampler). | ||
A registerable version of pytorch's | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
|
||
[SequentialSampler](https://pytorch.org/docs/stable/data.html#torch.utils.data.SequentialSampler). | ||
""" | ||
|
||
def __init__(self, data_source: data.Dataset): | ||
super().__init__(data_source) | ||
|
||
|
||
@Sampler.register("random") | ||
class RandomSampler(Sampler, data.RandomSampler): | ||
""" | ||
A registerable version of pytorch's [RandomSampler](https://pytorch.org/docs/stable/data.html#torch.utils.data.RandomSampler). | ||
A registerable version of pytorch's | ||
[RandomSampler](https://pytorch.org/docs/stable/data.html#torch.utils.data.RandomSampler). | ||
Samples elements randomly. If without replacement, then sample from a shuffled dataset. | ||
If with replacement, then user can specify `num_samples` to draw. | ||
|
||
|
@@ -68,7 +71,8 @@ def __init__( | |
@Sampler.register("subset_random") | ||
class SubsetRandomSampler(Sampler, data.SubsetRandomSampler): | ||
""" | ||
A registerable version of pytorch's [SubsetRandomSampler](https://pytorch.org/docs/stable/data.html#torch.utils.data.SubsetRandomSampler). | ||
A registerable version of pytorch's | ||
[SubsetRandomSampler](https://pytorch.org/docs/stable/data.html#torch.utils.data.SubsetRandomSampler). | ||
Samples elements randomly from a given list of indices, without replacement. | ||
|
||
# Parameters | ||
|
@@ -83,7 +87,8 @@ def __init__(self, indices: List[int]): | |
@Sampler.register("weighted_random") | ||
class WeightedRandomSampler(Sampler, data.WeightedRandomSampler): | ||
""" | ||
A registerable version of pytorch's [WeightedRandomSampler](https://pytorch.org/docs/stable/data.html#torch.utils.data.WeightedRandomSampler). | ||
A registerable version of pytorch's | ||
[WeightedRandomSampler](https://pytorch.org/docs/stable/data.html#torch.utils.data.WeightedRandomSampler). | ||
Samples elements from `[0,...,len(weights)-1]` with given probabilities (weights). | ||
|
||
# Parameters: | ||
|
@@ -112,7 +117,8 @@ def __init__(self, weights: List[float], num_samples: int, replacement: bool = T | |
@BatchSampler.register("basic") | ||
class BasicBatchSampler(BatchSampler, data.BatchSampler): | ||
""" | ||
A registerable version of pytorch's [BatchSampler](https://pytorch.org/docs/stable/data.html#torch.utils.data.BatchSampler). | ||
A registerable version of pytorch's | ||
[BatchSampler](https://pytorch.org/docs/stable/data.html#torch.utils.data.BatchSampler). | ||
Wraps another sampler to yield a mini-batch of indices. | ||
|
||
# Parameters | ||
|
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Might be useful to have a docstring pointing to the pytorch classes for all of these.