homura.vision.data package¶
Submodules¶
homura.vision.data.classification module¶
- class homura.vision.data.classification.ExtraSVHN(root, train=True, transform=None, download=False)[source]¶
Bases:
object
homura.vision.data.detection module¶
homura.vision.data.segmentation module¶
homura.vision.data.statistics module¶
- class homura.vision.data.statistics.PerChannelStatistics(num_samples)[source]¶
Bases:
object
estimates per channel image dataset statistics (mean, stdev)
>>> estimator = PerChannelStatistics(100) >>> estimator.from_directory("data/images") >>> estimator.from_batch(img_tensor) >>> estimator.estimated
- Parameters
num_samples (int) –
- property estimated¶
- property mean¶
- property stdev¶
homura.vision.data.visionset module¶
- class homura.vision.data.visionset.VisionSet(tv_class, root, num_classes, default_norm, default_train_da=None, default_test_da=None, collate_fn=None, test_collate_fn=None)[source]¶
Bases:
object
Dataset abstraction for vision datasets. Use case
data = DATASET_REGISTER(“cifar10”).setup(…) for img, label in data.tran_loader:
…
- Parameters
tv_class (Type[homura.vision.data.visionset.VisionSetProtocol]) –
root (str) –
num_classes (int) –
default_norm (list) –
default_train_da (list) –
default_test_da (list) –
collate_fn (Callable) –
test_collate_fn (Callable) –
- Return type
None
- collate_fn: Callable = None¶
- default_norm: list¶
- default_test_da: list = None¶
- default_train_da: list = None¶
- get_dataloader(batch_size, train_da=None, test_da=None, norm=None, train_size=None, test_size=None, val_size=None, download=False, num_workers=1, test_num_workers=None, non_training_bs_factor=2, drop_last=False, pin_memory=True, return_num_classes=False, test_batch_size=None, pre_default_train_da=None, post_default_train_da=None, post_norm_train_da=None, prefetch_factor=2, persistent_workers=False, worker_init_fn=None, train_sampler=None, test_sampler=None, val_sampler=None, start_epoch=0)[source]¶
Get Dataloader. This will automatically handle distributed setting
- Parameters
batch_size (int) – Batch size
train_da (Optional[list]) – Data Augmentation for training
test_da (Optional[list]) – Data Augmentation for testing and validation
norm (Optional[list]) – Normalization after train_da and test_da
train_size (Optional[int]) – Size of training dataset. If None, full dataset will be available.
test_size (Optional[int]) – Size of test dataset. If None, full dataset will be available.
val_size (Optional[int]) – Size of validation dataset, randomly split from the training dataset.
download (bool) –
num_workers (int) –
test_num_workers (Optional[int]) –
drop_last (bool) –
pin_memory (bool) –
return_num_classes (bool) –
test_batch_size (Optional[int]) –
pre_default_train_da (Optional[list]) –
post_default_train_da (Optional[list]) –
post_norm_train_da (Optional[list]) –
prefetch_factor (int) –
persistent_workers (bool) –
worker_init_fn (Optional[Callable]) –
train_sampler (Optional[Callable]) –
test_sampler (Optional[Callable]) –
val_sampler (Optional[Callable]) –
start_epoch (bool) –
- Return type
tuple[torch.utils.data.dataloader.DataLoader, torch.utils.data.dataloader.DataLoader]
If None, None will be returned. :param download: If dataset needs downloading :param num_workers: Number of workers in data loaders :param non_training_bs_factor: Batch size scale factor during non training. For example, testing time requires no backward cache, so basically batch size can be doubled. :param drop_last: If drop last batch or not :param pin_memory: If pin memory or not :param return_num_classes: If return number of classes as the last return value :param test_batch_size: Test time batch size. If None, non_training_bs_factor * batch_size is used. :param pre_default_train_da: Data Augmentation before the default data augmentation :param post_default_train_da: Data Augmentation after the default data augmentation :param post_norm_train_da: Data Augmentation after normalization (i.e., norm) :param prefetch_factor: :param persistent_workers: :param worker_init_fn: :param train_sampler: None or function of dataset -> sampler :param test_sampler: None or function of dataset -> sampler :param val_sampler: None or function of dataset -> sampler :param start_epoch: Epoch at start time :return: train_loader, test_loader, [val_loader], [num_classes]
- get_dataset(train_size=None, test_size=None, val_size=None, train_da=None, test_da=None, norm=None, download=False, pre_train_da=None, post_train_da=None, post_norm_train_da=None)[source]¶
Get Dataset
- Parameters
train_size (Optional[int]) – Size of training dataset. If None, full dataset will be available.
test_size (Optional[int]) – Size of test dataset. If None, full dataset will be available.
val_size (Optional[int]) – Size of validation dataset, randomly split from the training dataset. If None, None will be returned.
train_da (Optional[list]) – Data Augmentation for training
test_da (Optional[list]) – Data Augmentation for testing and validation
norm (Optional[list]) – Normalization after train_da and test_da
download (bool) – If dataset needs downloading
pre_train_da (Optional[list]) – Data Augmentation before the default data augmentation
post_train_da (Optional[list]) – Data Augmentation after the default data augmentation
post_norm_train_da (Optional[list]) – Data Augmentation after normalization (i.e., norm)
- Returns
train_set, test_set, val_set
- Return type
tuple[homura.vision.data.visionset.VisionSetProtocol, homura.vision.data.visionset.VisionSetProtocol, homura.vision.data.visionset.VisionSetProtocol]
- num_classes: int¶
- root: str¶
- setup(batch_size, train_da=None, test_da=None, norm=None, train_size=None, test_size=None, val_size=None, download=False, num_workers=1, test_num_workers=None, non_training_bs_factor=2, drop_last=False, pin_memory=True, return_num_classes=False, test_batch_size=None, pre_default_train_da=None, post_default_train_da=None, post_norm_train_da=None, prefetch_factor=2, persistent_workers=False, worker_init_fn=None, train_sampler=None, test_sampler=None, val_sampler=None, start_epoch=0)[source]¶
- Parameters
batch_size (int) –
train_da (Optional[list]) –
test_da (Optional[list]) –
norm (Optional[list]) –
train_size (Optional[int]) –
test_size (Optional[int]) –
val_size (Optional[int]) –
download (bool) –
num_workers (int) –
test_num_workers (Optional[int]) –
drop_last (bool) –
pin_memory (bool) –
return_num_classes (bool) –
test_batch_size (Optional[int]) –
pre_default_train_da (Optional[list]) –
post_default_train_da (Optional[list]) –
post_norm_train_da (Optional[list]) –
prefetch_factor (int) –
persistent_workers (bool) –
worker_init_fn (Optional[Callable]) –
train_sampler (Optional[Callable]) –
test_sampler (Optional[Callable]) –
val_sampler (Optional[Callable]) –
start_epoch (bool) –
- Return type
- test_collate_fn: Callable = None¶
- property test_loader¶
- property test_set¶
- property train_loader¶
- property train_set¶
- tv_class: Type[homura.vision.data.visionset.VisionSetProtocol]¶
- property val_loader¶
- property val_set¶