-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Qwen-72B-Chat-Int4推理时间 #783
Comments
输出有多长呢?参考README中速度说明看看 https://github.com/QwenLM/Qwen#inference-performance (额外考虑下transformers推理的话,多卡速度要比单卡慢。) |
输出也不是很长: 在一个寒冷的冬日,小明走在回家的路上。他看见一只小鸟掉在了地上,冻得瑟瑟发抖。小明心疼极了,他把小鸟捧在手心里,用自己的体温温暖它。 大概153token |
遇到了同样的问题,4张v100,让他讲个故事花了10多分钟...,该如何解决 |
一样,docker部署的72b-int4模型,单卡和双卡推理都非常慢 |
@sheiy @zhudongwork @BUJIDAOVS @boquanzhou 您好,如果您是在Docker中部署72B量化版本模型的话,推理速度变慢是因为之前docker镜像中的auto-gptq版本存在问题(可参考此issue) |
@fyabc 感谢 |
是否已有关于该错误的issue或讨论? | Is there an existing issue / discussion for this?
该问题是否在FAQ中有解答? | Is there an existing answer for this in FAQ?
当前行为 | Current Behavior
Qwen-72B-Chat-Int4使用A100 40G*2 进行推理,时间长达257s,这种情况正常么
期望行为 | Expected Behavior
No response
复现方法 | Steps To Reproduce
运行环境 | Environment
备注 | Anything else?
No response
The text was updated successfully, but these errors were encountered: