Normalization for object detection

[pmeier]

Migrated from discuss.pytorch.org. Requests were made by @mattans.

📚 Documentation

The reference implementations for classification, segmentation, and video classification all use a normalization transform. In contrast, object detection does not use any normalization.

1. Consider explaining why the pretrained detection models are the only ones that don’t require image normalization (I understand that the training set was not normalized. But again, why?)
2. Worth mentioning that no normalization is needed. The classification, segmentation and detection pretrained models are trained on ImageNet, so one may think all of them require ImageNet normalization, when in fact only the classification and segmentation models require normalization. Perhaps it’s best to put this info in a table, since the pretrained video models also have a normalization, but different.

[fmassa]

Hey,

So, the issue is that we embed the normalization (and other transforms) inside the model itself, see

vision/torchvision/models/detection/transform.py

Lines 104 to 105 in e212cc8

	image = self.normalize(image)
	image, target_index = self.resize(image, target_index)

This inconsistency is unfortunate, but was kind of necessary in order to make it easier for users to use the detection models. My thinking was that we might want at some point in the future to make all the models have data transformations inside them, as the way you normalize the inputs is tied with the pre-trained weights that we provide.

For now, I think we might want to improve the documentation to potentially clarify any confusions

[mattans]

OK, thanks. I also think it's worth updating the docs.

[pmeier]

@mattans We are happy to accept a PR for that. Would you like to send one?

[mattans]

@mattans We are happy to accept a PR for that. Would you like to send one?

Yes, I will do it in the following days. Thank you very much.

[mattans]

Just to make sure: @fmassa , what will happen if I use the object detection models without pretraining? Will it still auto-normalize the inputs?
Also, does this auto-normalization apply for both training and inference?

[pmeier]

[W]hat will happen if I use the object detection models without pretraining? Will it still auto-normalize the inputs?

Yes. The normalization transform is "hard coded" into the models:

vision/torchvision/models/detection/faster_rcnn.py

Lines 227 to 233 in 131ba13

	if image_mean is None:
	image_mean = [0.485, 0.456, 0.406]
	if image_std is None:
	image_std = [0.229, 0.224, 0.225]
	transform = GeneralizedRCNNTransform(min_size, max_size, image_mean, image_std)
	super(FasterRCNN, self).__init__(backbone, rpn, roi_heads, transform)

KeypointRCNN and MaskRCNN inherit from FasterRCNN (shown above) and thus also have this behavior.

---

Also, does this auto-normalization apply for both training and inference?

Yes. The model is created the same for training and inference

vision/references/detection/train.py

Lines 95 to 98 in 131ba13

	print("Creating model")
	model = torchvision.models.detection.__dict__[args.model](num_classes=num_classes,
	pretrained=args.pretrained)
	model.to(device)

and the transform is also applied unconditionally:

vision/torchvision/models/detection/generalized_rcnn.py

Line 79 in 131ba13

images, targets = self.transform(images, targets)