Skip to content

[A toolbox for fun.] Transform Image into Unique Paragraph with ChatGPT, BLIP2, OFA, GRIT, Segment Anything, ControlNet.

License

Notifications You must be signed in to change notification settings

showlab/Image2Paragraph

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

43 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Your Image Description

Image.txt: Transform Image Into Unique Paragraph

Hugging Face Spaces, , Open In Colab

Project Website

(huggingface sometimes may not work with safari, use chrome)

Demo

Your Image Description News

  • 17/April/2023. In addition to semantic segment anything, we use Edit Anything to get region-level semantic. Now all models takes less than 20s on 8G memory GPU card. (10times faster than previous version on cpu)
  • 17/April/2023. Our project is online on Huggingface. Have a try! huggingface
  • 14/April/2023. Our project is very popular in twitter. Looking the posted twitter for details.

(Can run on 8GB memory GPU within 20S!)

Your Image Description

Main Pipeline

Your Image Description

Reasoning Details

Your Image Description

To Do List

Done

  • GRIT example.
  • ControNet, BLIP2.
  • Semantic Segment Anything.
  • Segment Anything for fine-grained semantic.
  • Gradio.
  • Integrate GRIT into our code.
  • Support GPT4 API.
  • Notebook/Huggingface Space.
  • Region Semantic Classification from Edit-Anything.
  • Make the model lightweight.

Doing

  • Replace ChatGPT with own trained LLM.
  • Other grounding text2image model as instead of Canny ControlNet.
  • Show retrieval result in gradio.

Visualization

The text to image model is conrolnet with canny from diffuser.

Your Image Description

Your Image Description

Your Image Description

Installation

Please find installation instructions in install.md.

2. Start

Simple visualization

export OPENAI_KEY=[YOUR KEY HERE]
python main.py  --image_src [image_path] --out_image_name [out_file_name]

If your GPU memory smaller than 8 GPB.

python main.py --image_caption_device cpu --semantic_segment_device cpu

If you have no GPU available.

python main.py --image_caption_device cpu --semantic_segment_device cpu --dense_caption_device cpu  --contolnet_device cpu

like

python main.py --image_src "examples/3.jpg" --out_image_name "output/3_result.jpg"

Note: If you have GPU card with larger memory than 15GB. Set all device to GPU for fast inference.

The generated text and image are show in "output/".

Note: Use GPT4 for good result as GPT3.5 miss the position information sometime.

Use gradio directly

python main_gradio.py

If you have GPU Memory larger than 20GB. Use device='cuda' as default.

3. Visualization

Your Image Description A dog sitting on a porch with a bike. Your Image Description Your Image Description
Input BLIP2 Image Caption GRIT Dense Caption Semantic Segment Anything

The final generated paragraph with ChatGPT is:

  This image depicts a black and white dog sitting on a porch beside a red bike. The dense caption mentions other objects in the scene, such as a white car parked on the street and a red bike parked on the side of the road. The region semantic provides more specific information, including the porch, floor, wall, and trees. The dog can be seen sitting on the floor beside the bike, and there is also a parked bicycle and tree in the background. The wall is visible on one side of the image, while the street and trees can be seen in the other direction. 

4. Retrieval Result on COCO

Method Trainable Parameter Running Time IR@1 TR@1
Image-text 230M 9H 43.8 33.2
Generated Paragraph-text 0 5m 49.7 36.1

Interesting, we find compress image into paragraph. The retrieval result is even better than use source image.

Others

If you have more suggestions or functions need to be implemented in this codebase, feel free to drop me an email awinyimg dot gmail dot com or open an issue.

Acknowledgment

This work is based on ChatGPT, Edit_Anything, BLIP2, GRIT, OFA,Segment-Anything, Semantic-Segment-Anything, ControlNet.

About

[A toolbox for fun.] Transform Image into Unique Paragraph with ChatGPT, BLIP2, OFA, GRIT, Segment Anything, ControlNet.

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages