Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Qualcomm mobile SoC native backend for GGML #771

Closed
zhouwg opened this issue Mar 25, 2024 · 1 comment
Closed

Add Qualcomm mobile SoC native backend for GGML #771

zhouwg opened this issue Mar 25, 2024 · 1 comment

Comments

@zhouwg
Copy link
Contributor

zhouwg commented Mar 25, 2024

  1. Apple's dedicated machine learning acceleration library(ANE) is very important for GGML/llama.cpp's performance on iOS/Mac

we should/might study something about Qualcomm's dedicated/proprietary machine learning acceleration library accordingly.

Screenshot from 2024-03-10 21-57-20

  1. background:
  1. A small R&D team from Intel China branch in Shanghai has provided ggml with the sycl native backend based on Intel chips.
@zhouwg zhouwg changed the title Adding native backend for Qualcomm mobile SoC Add Qualcomm mobile SoC native backend for GGML/whispercpp/llamacpp Mar 29, 2024
@zhouwg zhouwg changed the title Add Qualcomm mobile SoC native backend for GGML/whispercpp/llamacpp Add Qualcomm mobile SoC native backend for GGML Mar 29, 2024
@zhouwg
Copy link
Contributor Author

zhouwg commented Apr 17, 2024

It seems done(although lack of implementation of other GGML OPs using QNN API ) and now it's PR stage.

Implementation of GGML_OP_MUL_MAT using QNN API and GGML_OP_ADD using QNN API and GGML_OP_MUL using QNN API could be found in ggml-qnn.cpp. other implementation of GGML OPs using QNN API is very similar to GGML_OP_ADD / GGML_OP_MUL / GGML_OP_MULMAT in ggml-qnn.cpp. hope community could help to do it in upstream whiserp.cpp after the PR is accepted by upstream whisper.cpp

so I'd like to close this opening issue accordingly.

@zhouwg zhouwg closed this as completed Apr 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant