Add x86 information #1130
Merged
Add x86 information #1130
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jklj077
reviewed
Mar 12, 2024
| @@ -354,6 +354,9 @@ If you suffer from lack of GPU memory and you would like to run the model on mor | |||
|
|
|||
| However, though this method is simple, the efficiency of the native pipeline parallelism is low. We advise you to use vLLM with FastChat and please read the section for deployment. | |||
|
|
|||
| ### x86 Platforms | |||
| When deploy on Core™/Xeon® Scalable Processors or with Arc™ GPU, [OpenVINO™ Toolkit](https://docs.openvino.ai/2023.3/gen_ai_guide.html) is recommended. You can install and run this [example notebook] ( https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/254-llm-chatbot). | |||
|
|
|||
There was a problem hiding this comment.
would you mind adding instructions for users to contact you if they met problems? for example: For related issues, you are welcome to file an issue at <...>.
There was a problem hiding this comment.
Add update information for issue track, please check.
jklj077
reviewed
Mar 12, 2024
| @@ -347,6 +347,10 @@ model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen-7B-Chat", device_map="cp | |||
|
|
|||
| 尽管这个方法很简单，但它的效率相对较低。我们建议使用vLLM和FastChat并请阅读部署章节。 | |||
|
|
|||
| ### x86 平台 | |||
| 在 酷睿™/至强® 可扩展处理器或 Arc™ GPU 上部署量化模型时，建议使用 [OpenVINO™ Toolkit](https://docs.openvino.ai/2023.3/gen_ai_guide.html) 以充分利用硬件，实现更好的推理性能。您可以安装并运行此[example notebook] (https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/254-llm-chatbot)。 | |||
|
|
|||
There was a problem hiding this comment.
Add update information for issue track, please check.
Add issue contact information
Add issue contact information.
Add issue support information for openvino
jklj077
approved these changes
Mar 13, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add Intel HW and SW supporting information. Thanks.