guide for implementing multimodal models?

[amosyou]

I read this comment about how to adapt vllm implementations for sglang. I was wondering if there were any pointers on how to implement vision-language models for sglang? I've been reading through llava.py and it's not entirely clear to me what I need to change from a typical huggingface implementation.

Also, I was wondering why there's no check for ForwardMode.PREFILL in the forward function for LlavaLlamaForCausalLM?