🍇 [Read our arXiv Paper] 🍎 [Try our Demo] 🍊 [Walk through Project Page]
We introduce FIND that can INterfacing Foundation models' embeDDings in an interleaved shared embedding space. Below is a brief introduction of all the generic and interleave tasks we can do!
- [2023.12.3] 🔥 We have a poster session@NeurIPS for SEEM, feel free to visit us during 5:00-7:00pm (CT)!
- [2023.12.2] We have released all the training, evaluation, and demo code!
- Demo Code
- Model Checkpoint
- Comprehensive User Guide
- Dataset
- Training Code
- Evaluation Code
entity_train2017.json | entity_val2017.json | entity_val2017_long.json |
---|---|---|
download | download | download |
COCO-Entity | COCO-Entity-Long | ||||||||
---|---|---|---|---|---|---|---|---|---|
cIoU | AP50 | IR@5 | IR@10 | cIoU | AP50 | IR@5 | IR@10 | ||
ImageBIND (H) | - | - | - | 51.4 | 61.3 | - | - | 58.7 | 68.9 |
Grounding-SAM (H) | - | 58.9 | 63.2 | - | - | 56.1 | 62.5 | - | - |
Focal-T | ckpt | 74.9 | 79.5 | 43.5 | 57.1 | 73.2 | 77.7 | 49.4 | 63.9 |
Focal-L | ckpt |
- Example Output
![Screenshot 2023-12-13 at 10 28 05 AM](https://private-user-images.githubusercontent.com/11957155/290260136-48d84fb9-160c-4113-b50b-e7872dcde544.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjE2NjgzMzgsIm5iZiI6MTcyMTY2ODAzOCwicGF0aCI6Ii8xMTk1NzE1NS8yOTAyNjAxMzYtNDhkODRmYjktMTYwYy00MTEzLWI1MGItZTc4NzJkY2RlNTQ0LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA3MjIlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNzIyVDE3MDcxOFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWY1OGNkNTg4YWY5M2I5MzUzNGQ1N2IxYWFhM2U1ZmQ5OTk3MmE3MzMwZDRmNDYxZWIzNGU4NmQ5ODVhYTg0ZTgmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.b1TW06CKBEeC7W4gbxSrgpU3IjZ6reMPTPjROMZayrg)
![Screenshot 2023-12-13 at 10 31 36 AM](https://private-user-images.githubusercontent.com/11957155/290261080-b63582b2-45ca-4b3d-afd1-419770af2e2a.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjE2NjgzMzgsIm5iZiI6MTcyMTY2ODAzOCwicGF0aCI6Ii8xMTk1NzE1NS8yOTAyNjEwODAtYjYzNTgyYjItNDVjYS00YjNkLWFmZDEtNDE5NzcwYWYyZTJhLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA3MjIlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNzIyVDE3MDcxOFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTIzYzZlYTg1Yzc3NzJhYjVjOWVlNTgzMzdiOTFkMDM4Mjk3MTJhMTQzYmZkZGJkMWU3N2UwMGFmZjM4MWNkY2UmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.5p4GpKIb-Kea2-xbZUhv0TeN5hvk4erN2ukAmXo9q3w)