Understand torch profiler output for zero3 during fine-tuning of llama2 7B

[xwjiang2010]

Hi,
I am trying to understand the bottleneck of my fine-tuning job through torch profiler.
There should be 32 decoding layers for llama2 7B model.
I see somehow decoding layer 12 is seeing much longer time than other layers. Is there an explanation for it?

Thanks in advance!