homura.vision.data package

Submodules

homura.vision.data.classification module

class homura.vision.data.classification.ExtraSVHN(root, train=True, transform=None, download=False)[source]

Bases: object

class homura.vision.data.classification.ImageNet(root, train=True, transform=None, loader=None, download=False)[source]

Bases: torchvision.datasets.folder.ImageFolder

class homura.vision.data.classification.OriginalSVHN(root, train=True, transform=None, download=False)[source]

Bases: torchvision.datasets.svhn.SVHN

homura.vision.data.detection module

class homura.vision.data.detection.VOCDetection(root, train=True, transform=None, download=False)[source]

Bases: object

homura.vision.data.detection.det_collate_fn(batch)[source]
homura.vision.data.detection.remove_difficult(dataset)[source]
Parameters

dataset (homura.vision.data.detection._VOCDetection) –

homura.vision.data.segmentation module

class homura.vision.data.segmentation.ExtendedVOCSegmentation(root, train=True, transform=None, download=False)[source]

Bases: object

homura.vision.data.segmentation.cat_list(images, fill_value=0)[source]
homura.vision.data.segmentation.seg_collate_fn(batch)[source]

homura.vision.data.statistics module

class homura.vision.data.statistics.PerChannelStatistics(num_samples)[source]

Bases: object

estimates per channel image dataset statistics (mean, stdev)

>>> estimator = PerChannelStatistics(100)
>>> estimator.from_directory("data/images")
>>> estimator.from_batch(img_tensor)
>>> estimator.estimated
Parameters

num_samples (int) –

property estimated
from_batch(batch)[source]
Parameters

batch (Sequence[torch.Tensor]) –

from_directory(root)[source]
Parameters

root (pathlib.Path) –

property mean
property stdev

homura.vision.data.visionset module

class homura.vision.data.visionset.VisionSet(tv_class, root, num_classes, default_norm, default_train_da=None, default_test_da=None, collate_fn=None, test_collate_fn=None)[source]

Bases: object

Dataset abstraction for vision datasets. Use case

data = DATASET_REGISTER(“cifar10”).setup(…) for img, label in data.tran_loader:

Parameters
  • tv_class (Type[homura.vision.data.visionset.VisionSetProtocol]) –

  • root (str) –

  • num_classes (int) –

  • default_norm (list) –

  • default_train_da (list) –

  • default_test_da (list) –

  • collate_fn (Callable) –

  • test_collate_fn (Callable) –

Return type

None

collate_fn: Callable = None
default_norm: list
default_test_da: list = None
default_train_da: list = None
get_dataloader(batch_size, train_da=None, test_da=None, norm=None, train_size=None, test_size=None, val_size=None, download=False, num_workers=1, test_num_workers=None, non_training_bs_factor=2, drop_last=False, pin_memory=True, return_num_classes=False, test_batch_size=None, pre_default_train_da=None, post_default_train_da=None, post_norm_train_da=None, prefetch_factor=2, persistent_workers=False, worker_init_fn=None, train_sampler=None, test_sampler=None, val_sampler=None, start_epoch=0)[source]

Get Dataloader. This will automatically handle distributed setting

Parameters
  • batch_size (int) – Batch size

  • train_da (Optional[list]) – Data Augmentation for training

  • test_da (Optional[list]) – Data Augmentation for testing and validation

  • norm (Optional[list]) – Normalization after train_da and test_da

  • train_size (Optional[int]) – Size of training dataset. If None, full dataset will be available.

  • test_size (Optional[int]) – Size of test dataset. If None, full dataset will be available.

  • val_size (Optional[int]) – Size of validation dataset, randomly split from the training dataset.

  • download (bool) –

  • num_workers (int) –

  • test_num_workers (Optional[int]) –

  • drop_last (bool) –

  • pin_memory (bool) –

  • return_num_classes (bool) –

  • test_batch_size (Optional[int]) –

  • pre_default_train_da (Optional[list]) –

  • post_default_train_da (Optional[list]) –

  • post_norm_train_da (Optional[list]) –

  • prefetch_factor (int) –

  • persistent_workers (bool) –

  • worker_init_fn (Optional[Callable]) –

  • train_sampler (Optional[Callable]) –

  • test_sampler (Optional[Callable]) –

  • val_sampler (Optional[Callable]) –

  • start_epoch (bool) –

Return type

tuple[torch.utils.data.dataloader.DataLoader, torch.utils.data.dataloader.DataLoader]

If None, None will be returned. :param download: If dataset needs downloading :param num_workers: Number of workers in data loaders :param non_training_bs_factor: Batch size scale factor during non training. For example, testing time requires no backward cache, so basically batch size can be doubled. :param drop_last: If drop last batch or not :param pin_memory: If pin memory or not :param return_num_classes: If return number of classes as the last return value :param test_batch_size: Test time batch size. If None, non_training_bs_factor * batch_size is used. :param pre_default_train_da: Data Augmentation before the default data augmentation :param post_default_train_da: Data Augmentation after the default data augmentation :param post_norm_train_da: Data Augmentation after normalization (i.e., norm) :param prefetch_factor: :param persistent_workers: :param worker_init_fn: :param train_sampler: None or function of dataset -> sampler :param test_sampler: None or function of dataset -> sampler :param val_sampler: None or function of dataset -> sampler :param start_epoch: Epoch at start time :return: train_loader, test_loader, [val_loader], [num_classes]

get_dataset(train_size=None, test_size=None, val_size=None, train_da=None, test_da=None, norm=None, download=False, pre_train_da=None, post_train_da=None, post_norm_train_da=None)[source]

Get Dataset

Parameters
  • train_size (Optional[int]) – Size of training dataset. If None, full dataset will be available.

  • test_size (Optional[int]) – Size of test dataset. If None, full dataset will be available.

  • val_size (Optional[int]) – Size of validation dataset, randomly split from the training dataset. If None, None will be returned.

  • train_da (Optional[list]) – Data Augmentation for training

  • test_da (Optional[list]) – Data Augmentation for testing and validation

  • norm (Optional[list]) – Normalization after train_da and test_da

  • download (bool) – If dataset needs downloading

  • pre_train_da (Optional[list]) – Data Augmentation before the default data augmentation

  • post_train_da (Optional[list]) – Data Augmentation after the default data augmentation

  • post_norm_train_da (Optional[list]) – Data Augmentation after normalization (i.e., norm)

Returns

train_set, test_set, val_set

Return type

tuple[homura.vision.data.visionset.VisionSetProtocol, homura.vision.data.visionset.VisionSetProtocol, homura.vision.data.visionset.VisionSetProtocol]

num_classes: int
root: str
setup(batch_size, train_da=None, test_da=None, norm=None, train_size=None, test_size=None, val_size=None, download=False, num_workers=1, test_num_workers=None, non_training_bs_factor=2, drop_last=False, pin_memory=True, return_num_classes=False, test_batch_size=None, pre_default_train_da=None, post_default_train_da=None, post_norm_train_da=None, prefetch_factor=2, persistent_workers=False, worker_init_fn=None, train_sampler=None, test_sampler=None, val_sampler=None, start_epoch=0)[source]
Parameters
  • batch_size (int) –

  • train_da (Optional[list]) –

  • test_da (Optional[list]) –

  • norm (Optional[list]) –

  • train_size (Optional[int]) –

  • test_size (Optional[int]) –

  • val_size (Optional[int]) –

  • download (bool) –

  • num_workers (int) –

  • test_num_workers (Optional[int]) –

  • drop_last (bool) –

  • pin_memory (bool) –

  • return_num_classes (bool) –

  • test_batch_size (Optional[int]) –

  • pre_default_train_da (Optional[list]) –

  • post_default_train_da (Optional[list]) –

  • post_norm_train_da (Optional[list]) –

  • prefetch_factor (int) –

  • persistent_workers (bool) –

  • worker_init_fn (Optional[Callable]) –

  • train_sampler (Optional[Callable]) –

  • test_sampler (Optional[Callable]) –

  • val_sampler (Optional[Callable]) –

  • start_epoch (bool) –

Return type

homura.vision.data.visionset.VisionSet

test_collate_fn: Callable = None
property test_loader
property test_set
property train_loader
property train_set
tv_class: Type[homura.vision.data.visionset.VisionSetProtocol]
property val_loader
property val_set
class homura.vision.data.visionset.VisionSetProtocol(*args, **kwargs)[source]

Bases: Protocol

Module contents