Skip to content
This repository has been archived by the owner on Dec 16, 2022. It is now read-only.

Data V2 #3700

Merged
merged 59 commits into from
Feb 26, 2020
Merged

Data V2 #3700

Changes from 1 commit
Commits
Show all changes
59 commits
Select commit Hold shift + click to select a range
0c42cb9
example for feedback
DeNeutoy Jan 30, 2020
5ffedfc
Merge branch 'master' into data-v2
DeNeutoy Feb 19, 2020
80049f8
remove all existing multiprocessing
DeNeutoy Feb 19, 2020
6f58c2a
sneak torch datasets inside DatasetReader
DeNeutoy Feb 19, 2020
1b3ad9a
lint
DeNeutoy Feb 19, 2020
effc445
trainer_v2, We Love To See It
DeNeutoy Feb 19, 2020
9d44ad6
datasets have index_with now, not iterators
DeNeutoy Feb 19, 2020
7e89ea6
use iter, custom collate function in allennlp wrapper
DeNeutoy Feb 19, 2020
883b6d7
we don't even need the data in the trainer anymore
DeNeutoy Feb 19, 2020
56d022a
all trainer tests passing
DeNeutoy Feb 20, 2020
01e12f5
black
DeNeutoy Feb 20, 2020
5aea291
make find learning rate work
DeNeutoy Feb 20, 2020
f026946
update test fixtures to new config
DeNeutoy Feb 20, 2020
5973b50
get train command tests mostly working
DeNeutoy Feb 20, 2020
a23f47a
lazily construct samplers, index lazy datasets
DeNeutoy Feb 20, 2020
a76ea0a
Merge branch 'master' into data-v2
DeNeutoy Feb 20, 2020
ebf3854
update some fixtures
DeNeutoy Feb 20, 2020
57a67e5
evaluate tests passing
DeNeutoy Feb 20, 2020
7d21ed8
all command tests passing
DeNeutoy Feb 20, 2020
24a500c
lint
DeNeutoy Feb 20, 2020
fb13769
update model test case, common and module tests passing
DeNeutoy Feb 20, 2020
ef5187f
fix test interdependence introduced by #3762
DeNeutoy Feb 21, 2020
b1ea845
more test interdependence
DeNeutoy Feb 21, 2020
0231616
tests tests tests
DeNeutoy Feb 21, 2020
01d76bb
remove unnecessary brackets
DeNeutoy Feb 21, 2020
12b6efb
Merge branch 'master' into data-v2
DeNeutoy Feb 21, 2020
859d3ca
update a chunk of the configs
DeNeutoy Feb 21, 2020
c22dee3
fix archival test, couple more configs
DeNeutoy Feb 21, 2020
fe5b470
rm pointless gan test
DeNeutoy Feb 21, 2020
7533c91
more tests passing
DeNeutoy Feb 21, 2020
ad45659
add current state of from params changes
DeNeutoy Feb 21, 2020
f944840
Revert "add current state of from params changes"
DeNeutoy Feb 21, 2020
3b12a2f
Merge branch 'master' into data-v2
DeNeutoy Feb 21, 2020
be1f58c
updated understanding of Lazy
DeNeutoy Feb 21, 2020
ebdabe0
add discussion of None comparison to Lazy
DeNeutoy Feb 21, 2020
8693739
lint
DeNeutoy Feb 21, 2020
b9b0650
it's a hard doc life
DeNeutoy Feb 21, 2020
88314c7
pull samplers into separate file
DeNeutoy Feb 21, 2020
14296a1
more docs updates
DeNeutoy Feb 22, 2020
8a08899
fold in #3812
DeNeutoy Feb 22, 2020
3520280
remove torch dataset
DeNeutoy Feb 22, 2020
0f1d8a4
add example to lazy
DeNeutoy Feb 22, 2020
93e1e89
rename to collate
DeNeutoy Feb 22, 2020
40dd695
no kwargs
DeNeutoy Feb 23, 2020
da3b1b4
Revert "fold in #3812"
DeNeutoy Feb 23, 2020
801a8f5
don't break up dataset
DeNeutoy Feb 23, 2020
007fd0c
add comment to iterable dataset len
DeNeutoy Feb 23, 2020
d00e1a9
Merge branch 'master' into data-v2
DeNeutoy Feb 23, 2020
c066804
improve docstrings, build dataloader using partial_objects
DeNeutoy Feb 23, 2020
61c7b14
flake
DeNeutoy Feb 23, 2020
2b56b14
give dataloader a default implementation
DeNeutoy Feb 24, 2020
354010a
safer default for DataLoader init
DeNeutoy Feb 24, 2020
568291d
more coherent dir structure
DeNeutoy Feb 24, 2020
a016103
update imports
DeNeutoy Feb 24, 2020
47db16a
Merge branch 'master' into data-v2
DeNeutoy Feb 24, 2020
04fdb70
add a test for the BucketBatchSampler
DeNeutoy Feb 24, 2020
d1d5c4a
split bucket sampler into own file, tests
DeNeutoy Feb 24, 2020
5f0c8db
PR comments
DeNeutoy Feb 26, 2020
6f63a53
Merge branch 'master' into data-v2
DeNeutoy Feb 26, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
lint
  • Loading branch information
DeNeutoy committed Feb 19, 2020
commit 1b3ad9a3cf1eca45db648974e81ed7ee86649097
13 changes: 10 additions & 3 deletions torch_datasets.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@

from torch.utils.data import Dataset as TorchDataset
from torch.utils.data import DataLoader
from torch.utils.data import Sampler, BatchSampler, SequentialSampler, SubsetRandomSampler
from torch.utils.data import BatchSampler, SequentialSampler

from allennlp.common.registrable import Registrable
from allennlp.common.util import add_noise_to_dict_values, lazy_groups_of
Expand All @@ -19,6 +19,7 @@

logger = logging.getLogger(__name__)


class Dataset(TorchDataset, Registrable):
def __init__(self):
self.vocab: Vocabulary = None
Expand Down Expand Up @@ -98,8 +99,13 @@ def text_to_instance(self, premise: str, hypothesis: str, label: str = None) ->


class BatchInstanceSampler(BatchSampler):

def __init__(self, data, batch_size: int, sorting_keys: List[Tuple[str, str]] = None, padding_noise: float = 0.1):
def __init__(
self,
data,
batch_size: int,
sorting_keys: List[Tuple[str, str]] = None,
padding_noise: float = 0.1,
):

self.vocab = data.vocab
self._sorting_keys = sorting_keys
Expand Down Expand Up @@ -184,6 +190,7 @@ def allennlp_collocate(batch):
batch = AllennlpBatch(batch)
return batch.as_tensor_dict(batch.get_padding_lengths())


batch_generator = DataLoader(data, batch_sampler=batch_sampler, collate_fn=allennlp_collocate)

iterator = iter(batch_generator)
Expand Down