Multimodal Needle in a Haystack: Benchmarking Long-Context Capability of Multimodal LLMs
H. Wang, H. Shi, S. Tan, W. Qin, W. Wang, T. Zhang, A. Nambi, T. Ganu, H. Wang
[Paper] [MMNeedle Dataset]
To use this benchmark, please download the MMNeedle dataset at this link. Alternatively, you could also construct your version of MMNeedle by following the instructions.
(2024-06-24) We released the leaderboard for Multimodal Long Context Understanding on Paper with Code!
(2024-06-17) We released the paper, code, and data for Multimodal Needle in a Haystack (MMNeedle) benchmark!
![Screen Shot 2024-06-17 at 7 38 45 PM](https://private-user-images.githubusercontent.com/30172609/340525052-cf481db4-ac83-4940-8897-e27d4faab4a8.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQzMjUxNzQsIm5iZiI6MTcyNDMyNDg3NCwicGF0aCI6Ii8zMDE3MjYwOS8zNDA1MjUwNTItY2Y0ODFkYjQtYWM4My00OTQwLTg4OTctZTI3ZDRmYWFiNGE4LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MjIlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODIyVDExMDc1NFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPThlNzA5M2M5ZDkwNjZlMjFkNjUzMmQ4MzQ3NWM3ZmFjMjc5NGVkZmU4NTNlYzVkOGFkOTU1ZDdjOGM3YmYwODMmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.Oc8wk5Evaoe49X3ssLrc1o4R_R4rP1QDdJiaOTLCGdE)
MMNeedle Evaluation Overview. Correct answers are marked with checkmark (
![Screen Shot 2024-06-17 at 7 39 52 PM](https://private-user-images.githubusercontent.com/30172609/340524882-e105e2f6-0585-4cbc-9e56-0f588134412d.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQzMjUxNzQsIm5iZiI6MTcyNDMyNDg3NCwicGF0aCI6Ii8zMDE3MjYwOS8zNDA1MjQ4ODItZTEwNWUyZjYtMDU4NS00Y2JjLTllNTYtMGY1ODgxMzQ0MTJkLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MjIlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODIyVDExMDc1NFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTdlNjVlYjE1YTc1OGIxMGQ3MTQxZTllNjQ4ZDVkN2EyNzI0ZDExODAwM2Q5OTQyZDA4NzJhOTU2YWQ1NWVhZTUmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.Qobx2m_ixnWKFWWanWuZkrEZkpSxx9FnBhH9WS13tro)
MMNeedle Evaluation Performance Comparison (Claude-3 refers to Claude 3 Opus, and Gemini-1.0/1.5 refers to Gemini Pro 1.0/1.5). The x-axis shows the results of different models, and the y-axis shows the results on various input image number M and stitching size N. For each row, i.e., setting (M,N), we show the average accuracy (%) of each model. For each stitched image, the color of row r, column c indicates the accuracy of predicting the exact position for samples with the "needle" sub-image in position (r,c) of the stitched image. For the M=10 setting, we show the average accuracy of each location (r,c) over 10 images. A redder cell indicates lower accuracy, while a greener cell indicates higher accuracy. The best result for each row is marked with underlining.