[Question] Why apply the causal mask to image tokens in the attention operations of LLM?

[yubin1219]

Question

In the attention operations of the LLM, text tokens must use the causal mask to predict the next text tokens for the answer. However, the image features are also masked by a causal mask rather than by a bidirectional mask in the attention operations of the LLM, even though the bidirectional mask allows access to more visual information than the causal mask. Is there a particular reason for this making?

[Yebulabula]

I have the same question.