CUDA out of memory - minimum gpu size

[yscoffee]

Thanks for the amazing work.
Could you please suggest the minimum GPU size for doing offline Inference?
I tried with 24GB 4090 but it turns out not enought for running the offline inference example.

[BIGBALLON]

@yscoffee The total number of model parameters is 25.5B, consisting of the architecture of InternViT-6B-448px-V1-5 + MLP + InternLM2-Chat-20B.

• For bf16, at least 52GB of GPU memory is required.
• For int8, at least 26GB of GPU memory is needed. There is also some additional overhead. so 32GB memory is needed.

Examples, if you can understand Chinese, you can refer to this article:

• 1 GPU for int8 model:
• 2 GPU for bf16 model:
• 4 GPU for bf16 model:

[yscoffee]

thank you for the detail explanation!