Our goal is to solve multiple fundamental visual perception, visual reasoning, and multi-modal large langauge tasks using one model, which minimize handcraft designs and maximize the functionality and performance in one shot.

We present OMG-LLaVA, a new and elegant framework combining powerful pixel-level vision understanding with reasoning abilities. It can accept various visual and text prompts for flexible user interaction. Specifically, we use a universal segmentation method as the visual encoder, integrating image information, perception priors, and visual prompts into visual tokens provided to the LLM. The LLM is responsible for understanding the user's text instructions and providing text responses and pixel-level segmentation results based on the visual information.

OMG-LLaVA achieves image-level, object-level, and pixel-level reasoning and understanding in a single model, matching or surpassing the performance of specialized methods on multiple benchmarks. Rather than using LLM to connect each specialist, our work aims at end-to-end training on one encoder, one decoder, and one LLM.

We address various segmentation tasks, each traditionally tackled by distinct or partially unified models. We propose OMG-Seg, One Model that is Good enough to efficiently and effectively handle all the Segmentation tasks, including image semantic, instance, and panoptic segmentation, as well as their video counterparts, open vocabulary settings, prompt-driven, interactive segmentation like SAM, and video object segmentation. To our knowledge, this is the first model to fill all these tasks in one model and achieve good enough performance.

We show that OMG-Seg, a transformer-based encoder-decoder architecture with task-specific queries and outputs, can support over ten distinct segmentation tasks and yet significantly reduce computational and parameter overhead across various tasks and datasets. We rigorously evaluate the inter-task influences and correlations during co-training. Both the code and models will be publicly available.

Short introduction on VALSE of OMG-Seg with other SAM-like works, can be found here, in Chinese.

• 🔥2023-6-28, Release OMG-LLaVA test code and ckpt (7B) models, will be released soon.
• 🔥2023-4-06, Update the model trained with only one machine and demo scripts.
• 🔥2024-3-18, Training Code of OMG-Seg are released !! Stronger Performance using Object-365-instance segmentation pre-train !!
• 🔥2024-2-26, OMG-Seg is accepted by CVPR-2024 !!
• 🔥2024-1-19, Models and Test Code are released !!

• One model to perform image level, object level, pixel level understanding and reasoning.
• A new view for solving multiple referring segmentation, localization, grounding, and captioning tasks using only one encoder, one decoder and one LLMs.

• Our codebase supports joint multi dense prediction tasks co-training in one shot.
• The first open-sourced codebase for multiple multimodal understanding tasks, including training, inference and demo.

• A new unified solution for over ten different segmentation tasks: PS, IS, VSS, VIS, VPS, Open-Vocabulary Seg, and Interactive Segmentation.
• A novel unified view for solving multiple segmentation tasks in one model with extremely less parameters.

• OMG-Seg achieves good enough performance on in one shared architecture, on multiple datasets. (only 70M trainable parameters)

• Our codebase support joint image/video/multi-dataset co-training.
• The first open-sourced codebase, including training, inference and demo.

• OMG-Seg can be reproduced by only one 32GB V100 or 40GB A100 machine, which can be followed by Academic Labs.

• Add more easy-used tutorial. ()
• Release OMG-LLaVA Models. ()
• Release OMG-Seg Strong Models. (Done)
• Release OMG-Seg training code. (Done)
• Support HuggingFace. (Done)

For OMG-Seg, please see the OMG_Seg_README.md

For OMG-LLaVA, please see the OMG_LLaVA_README.md

If you think our codebases and works are useful for your research, please consider referring us:

@article{OMGLLaVA,
  title={OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understanding},
  author={Zhang, Tao and Li, Xiangtai and Fei, Hao and Yuan, Haobo and Wu, Shengqiong and Ji, Shunping and Chen, Change Loy and Yan, Shuicheng},
  journal={arXiv preprint},
  year={2024}
}

@inproceedings{OMGSeg,
  title={OMG-Seg: Is one model good enough for all segmentation?},
  author={Li, Xiangtai and Yuan, Haobo and Li, Wei and Ding, Henghui and Wu, Size and Zhang, Wenwei and Li, Yining and Chen, Kai and Loy, Chen Change},
  booktitle={CVPR},
  year={2024}
}

OMG-Seg follows the S-Lab LICENSE.

OMG-LLaVA follows the Apache-2.0 license, for the respect of both LLaVA and XTuner codebase.