OmniFusion — a multimodal model to communicate using text and images

Radiology Objects in COntext (ROCO): A Multimodal Image Dataset

The official start-up code for paper "FFA-IR: Towards an Explainable and Reliable Medical Report Generation Benchmark."

HI-ML toolbox for deep learning for medical imaging and Azure integration

The official code for "Towards Generalist Foundation Model for Radiology by Leveraging Web-scale 2D&3D Medical Data".

GLoRIA: A Multimodal Global-Local Representation Learning Framework forLabel-efficient Medical Image Recognition

A collection of resources on applications of multi-modal learning in medical imaging.

✨✨Latest Advances on Multimodal Large Language Models

CLIP (Contrastive Language-Image Pretraining), Predict the most relevant text snippet given an image

Official source code for the paper: "Reading Between the Frames Multi-Modal Non-Verbal Depression Detection in Videos"

Implementation of Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmetic

Source code and data used in the papers ViQuAE (Lerner et al., SIGIR'22), Multimodal ICT (Lerner et al., ECIR'23) and Cross-modal Retrieval (Lerner et al., ECIR'24)

tiny vision language model

Fine Tuning Multimodal LLM "Idefics 9B" on Pokemon Go Dataset available on Hugging Face.

Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning