CUDA out of memory in the middle of fine tuning

[hirunima]

First of all thank you for providing the trainig scripts. I am using four A4000 with 16G memory each. While trying to fine tune on COCO, CUDA produce an OutOfMemoryError after about 5 steps which can be expected due to total memory increment indicated in each step. Is there any reason for this memory increase during the training?