Skip to content

Commit

Permalink
ggml : add RPC backend (llama/6829)
Browse files Browse the repository at this point in the history
* ggml : add RPC backend

The RPC backend proxies all operations to a remote server which runs a
regular backend (CPU, CUDA, Metal, etc).

* set TCP_NODELAY

* add CI workflows

* Address review comments

* fix warning

* implement llama_max_devices() for RPC

* Address review comments

* Address review comments

* wrap sockfd into a struct

* implement get_alignment and get_max_size

* add get_device_memory

* fix warning

* win32 support

* add README

* readme : trim trailing whitespace

* Address review comments

* win32 fix

* Address review comments

* fix compile warnings on macos
  • Loading branch information
rgerganov authored and ggerganov committed May 14, 2024
1 parent cc3c140 commit 48bb90e
Show file tree
Hide file tree
Showing 2 changed files with 1,047 additions and 0 deletions.
Loading

0 comments on commit 48bb90e

Please sign in to comment.