Best boundingbox in output (by order or by confidence)

[XieKaiwen]

In theory, will the model output the boundingboxes in the array in order of their accuracy? Or do we have to argmax the confidence scores to then index out the best boundingboxes?

My finetuned model seems to do pretty well on a test dataset by just taking the first bounding box in the array everytime (each test caption only has 1 class), even when i add in the argmax procedure, over the entire dataset i seem to get around the same level of performance.

Hence i am questioning if argmax is needed? Or is the boundingboxes outputted by the model already ordered in some way that guarantees accuracy?