Align and Prompt: Video-and-Language Pre-training with Entity Prompts

Official implementations for various pre-training models of ERNIE-family, covering topics of Language Understanding & Generation, Multimodal Understanding & Generation, and beyond.

PyTorch code for BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation

[CVPR 2022] Show Me What and Tell Me How: Video Synthesis via Multimodal Conditioning

Referring Video Object Segmentation / Multi-Object Tracking Repo

Code used in "Understanding Dimensional Collapse in Contrastive Self-supervised Learning" paper.

CLIP-GEN: Language-Free Training of a Text-to-Image Generator with CLIP

Official PyTorch implementation of "Large-scale Bilingual Language-Image Contrastive Learning" (ICLRW 2022)

PyTorch implementation of a 1.3B text-to-image generation model trained on 14 million image-text pairs

L-Verse: Bidirectional Generation Between Image and Text

Reference models and tools for Cloud TPUs.

[ICCV 2021- Oral] Official PyTorch implementation for Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers, a novel method to visualize any Transformer-…

This is the repo for my experiments with StyleGAN2. There are many like it, but this one is mine. Contains code for the paper Audio-reactive Latent Interpolations with StyleGAN.

DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generation Models (ICCV 2023)

MAGMA - a GPT-style multimodal model that can understand any combination of images and language. NOTE: The freely available model from this repo is only a demo. For the latest multimodal and multil…

PyTorch code for "Unifying Vision-and-Language Tasks via Text Generation" (ICML 2021)

[ICCV2021 & TPAMI2023] Vision-Language Transformer and Query Generation for Referring Segmentation

Code release for SLIP Self-supervision meets Language-Image Pre-training

CLIP (Contrastive Language-Image Pretraining), Predict the most relevant text snippet given an image

A concise but complete implementation of CLIP with various experimental improvements from recent papers

Repository for "Generating images from caption and vice versa via CLIP-Guided Generative Latent Space Search"

A pytorch Implementation of Open Vocabulary Object Detection with Pseudo Bounding-Box Labels

Open-Vocabulary Instance Segmentation via Robust Cross-Modal Pseudo-Labeling @ CVPR22

Research code for ECCV 2020 paper "UNITER: UNiversal Image-TExt Representation Learning"

official code repo for paper "CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformers"

WIT (Wikipedia-based Image Text) Dataset is a large multimodal multilingual dataset comprising 37M+ image-text sets with 11M+ unique images across 100+ languages.

Official implementation for "Blended Diffusion for Text-driven Editing of Natural Images" [CVPR 2022]

Pytorch implementation of Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors

[CVPR2022 oral] A Simple and Effective Baseline for Text-to-Image Synthesis