Improve Validation Visualization

[thhart]

🚀 Feature

Improve validation output during training.

Motivation

IMHO it could be very helpful to improve the visualization output during training already. Either by distributing into Wandb/Tensorboard or by output image results of validation since these are calculated anyway. Of course this would increase storage consumption during training but the advantages are very big.

Beside of simple monitor the training process this would also help to isolate hardest training examples at earliest possible.

Is there such a mechanism switch already available which would enhance the training output already?
Do you have any hints what might be further beneficial to achieve the same goals other than post evaluating the model?

[glenn-jocher]

@thhart there is a W&B PR which more formally implements your suggestions, and also the validation labels and predictions are plotted for the first and last epochs by default and stored in your runs/train/exp directory. You can modify this very simply here to print these jpgs every epoch, and every batch if you'd like.

We will not enable this by default however due to the much slower performance and the much large storage requirement it would introduce on all users compared to the current implementation.

yolov5/test.py

Lines 208 to 214 in cd8ed35

	# Plot images
	if plots and batch_i < 3:
	f = save_dir / f'test_batch{batch_i}_labels.jpg' # labels
	Thread(target=plot_images, args=(img, targets, paths, f, names), daemon=True).start()
	f = save_dir / f'test_batch{batch_i}_pred.jpg' # predictions
	Thread(target=plot_images, args=(img, output_to_target(out), paths, f, names), daemon=True).start()

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.