🍇 [Read our arXiv Paper] 🍎 [Try our Demo] 🍊 [Walk through Project Page]
We introduce FIND that can INterfacing Foundation models' embeDDings in an interleaved shared embedding space. Below is a brief introduction of all the generic and interleave tasks we can do!
- [2023.12.3] 🔥 We have a poster session@NeurIPS for SEEM, feel free to visit us during 5:00-7:00pm (CT)!
- [2023.12.2] We have released all the training, evaluation, and demo code!
- Demo Code
- Model Checkpoint
- Comprehensive User Guide
- Dataset
- Training Code
- Evaluation Code
entity_train2017.json | entity_val2017.json | entity_val2017_long.json |
---|---|---|
download | download | download |
COCO-Entity | COCO-Entity-Long | ||||||||
---|---|---|---|---|---|---|---|---|---|
cIoU | AP50 | IR@5 | IR@10 | cIoU | AP50 | IR@5 | IR@10 | ||
ImageBIND (H) | - | - | - | 51.4 | 61.3 | - | - | 58.7 | 68.9 |
Grounding-SAM (H) | - | 58.9 | 63.2 | - | - | 56.1 | 62.5 | - | - |
Focal-T | ckpt | 74.9 | 79.5 | 43.5 | 57.1 | 73.2 | 77.7 | 49.4 | 63.9 |
Focal-L | ckpt |
- Example Output
![Screenshot 2023-12-13 at 10 28 05 AM](https://private-user-images.githubusercontent.com/11957155/290260136-48d84fb9-160c-4113-b50b-e7872dcde544.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjIyMTU3ODEsIm5iZiI6MTcyMjIxNTQ4MSwicGF0aCI6Ii8xMTk1NzE1NS8yOTAyNjAxMzYtNDhkODRmYjktMTYwYy00MTEzLWI1MGItZTc4NzJkY2RlNTQ0LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA3MjklMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNzI5VDAxMTEyMVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTcxMjRmZjY1MzBhYjEyNjdjZDBmYjZlYWNmODU4NGNlZWUyZjZjNDZiZWMzYjAyMzE1YjRjZmFlMTQzNTg3ZGImWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.63XJWph08MQR3UEKYPzTcn0E4ABLqoChvkDOyBwJZaY)
![Screenshot 2023-12-13 at 10 31 36 AM](https://private-user-images.githubusercontent.com/11957155/290261080-b63582b2-45ca-4b3d-afd1-419770af2e2a.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjIyMTU3ODEsIm5iZiI6MTcyMjIxNTQ4MSwicGF0aCI6Ii8xMTk1NzE1NS8yOTAyNjEwODAtYjYzNTgyYjItNDVjYS00YjNkLWFmZDEtNDE5NzcwYWYyZTJhLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA3MjklMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNzI5VDAxMTEyMVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTZiOWIxZmJlMGNjZGYwMDNlMmEwMGNiNjI0ZjA3NjViNGI4NDdjODMxOWIzMDhjM2M1MTNiMGVkMTg3M2M1ZGYmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.Qom_H37oVUmJy9hysdi2CURJfuirikaVR5eoFH90SLA)