Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

llama : ggml-backend integration #4766

Merged
merged 39 commits into from
Jan 12, 2024
Merged

llama : ggml-backend integration #4766

merged 39 commits into from
Jan 12, 2024

Commits on Jan 6, 2024

  1. llama : ggml-backend integration

    @slaren
    slaren committed Jan 6, 2024
    Configuration menu
    Copy the full SHA
    33f0761 View commit details
    Browse the repository at this point in the history

  2. ggml-backend : add names to buffers

    @slaren
    slaren committed Jan 6, 2024
    Configuration menu
    Copy the full SHA
    6483328 View commit details
    Browse the repository at this point in the history

  3. fix unmap after loading

    @slaren
    slaren committed Jan 6, 2024
    Configuration menu
    Copy the full SHA
    a1ab35c View commit details
    Browse the repository at this point in the history

  4. batched-bench : add tensor_split param

    @ggerganov @slaren
    ggerganov authored and slaren committed Jan 6, 2024
    Configuration menu
    Copy the full SHA
    1fa7ee2 View commit details
    Browse the repository at this point in the history

  5. llama : check for null tensor_split

    @slaren
    slaren committed Jan 6, 2024
    Configuration menu
    Copy the full SHA
    863ef45 View commit details
    Browse the repository at this point in the history

  6. ggml-backend : increase GGML_MAX_BACKENDS

    @slaren
    slaren committed Jan 6, 2024
    Configuration menu
    Copy the full SHA
    d107459 View commit details
    Browse the repository at this point in the history

  7. improve graph splitting, partial fix for --no-kv-offload

    @slaren
    slaren committed Jan 6, 2024
    Configuration menu
    Copy the full SHA
    ece0b0d View commit details
    Browse the repository at this point in the history

  8. cuda : add ggml-backend split buffer support

    @slaren
    slaren committed Jan 6, 2024
    Configuration menu
    Copy the full SHA
    2f2c367 View commit details
    Browse the repository at this point in the history

  9. cuda : do not create buffer types for devices that don't exist (fixes… 

    … usage without CUDA devices available)
    @slaren
    slaren committed Jan 6, 2024
    Configuration menu
    Copy the full SHA
    72b74f3 View commit details
    Browse the repository at this point in the history

Commits on Jan 7, 2024

  1. ggml : fix null backend dereference (#4807) 

    * ggml : fix null backend dereference

* ggml : also check ggml_backend_is_cpu
    @ggerganov
    ggerganov committed Jan 7, 2024
    Configuration menu
    Copy the full SHA
    f77c72f View commit details
    Browse the repository at this point in the history

  2. test-backend-ops : check buffer allocation failures

    @slaren
    slaren committed Jan 7, 2024
    Configuration menu
    Copy the full SHA
    7c16cf1 View commit details
    Browse the repository at this point in the history

  3. Merge remote-tracking branch 'origin/master' into sl/backend-sched

    @slaren
    slaren committed Jan 7, 2024
    Configuration menu
    Copy the full SHA
    87c8207 View commit details
    Browse the repository at this point in the history

  4. llama : add cparam (split_mode) and command line argument (--split-mo… 

    …de, -sm) to configure the split mode (none, layer or row)
    @slaren
    slaren committed Jan 7, 2024
    Configuration menu
    Copy the full SHA
    5e879c9 View commit details
    Browse the repository at this point in the history

Commits on Jan 8, 2024

  1. ggml : fix mul_mat_id work size

    @slaren
    slaren committed Jan 8, 2024
    Configuration menu
    Copy the full SHA
    ac145fd View commit details
    Browse the repository at this point in the history

  2. llama : rewrite session kv load/set without graphs

    @slaren
    slaren committed Jan 8, 2024
    Configuration menu
    Copy the full SHA
    444b975 View commit details
    Browse the repository at this point in the history

  3. minor

    @slaren
    slaren committed Jan 8, 2024
    Configuration menu
    Copy the full SHA
    d41cef9 View commit details
    Browse the repository at this point in the history

  4. llama : only initialize used backends, free backends on context free

    @slaren
    slaren committed Jan 8, 2024
    Configuration menu
    Copy the full SHA
    5a62db3 View commit details
    Browse the repository at this point in the history

  5. llama : abort ctx if cuda backend init fails

    @slaren
    slaren committed Jan 8, 2024
    Configuration menu
    Copy the full SHA
    4813e17 View commit details
    Browse the repository at this point in the history

  6. llama : rewrite lora with ggml-backend and compute on CPU 

    ggml-ci
    @slaren
    slaren committed Jan 8, 2024
    Configuration menu
    Copy the full SHA
    11583c1 View commit details
    Browse the repository at this point in the history

  7. llama : only map to a backend buffer the region of the file mapping c… 

    …ontaining the tensors used in the buffer
    @slaren
    slaren committed Jan 8, 2024
    Configuration menu
    Copy the full SHA
    4ed5f62 View commit details
    Browse the repository at this point in the history

Commits on Jan 9, 2024

  1. opencl : add ggml-backend buffer type

    @slaren
    slaren committed Jan 9, 2024
    Configuration menu
    Copy the full SHA
    fa76201 View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'origin/master' into sl/backend-sched 

    ggml-ci
    @slaren
    slaren committed Jan 9, 2024
    Configuration menu
    Copy the full SHA
    2e7814a View commit details
    Browse the repository at this point in the history

Commits on Jan 10, 2024

  1. cuda : only use batched_cublas with batched mat muls (fixes fp16 tg p… 

    …erf)
    @slaren
    slaren committed Jan 10, 2024
    Configuration menu
    Copy the full SHA
    5d2dffc View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'origin/master' into sl/backend-sched

    @slaren
    slaren committed Jan 10, 2024
    Configuration menu
    Copy the full SHA
    3cb1c1f View commit details
    Browse the repository at this point in the history

  3. llama : on Metal, by default offload the full model 

    ggml-ci
    @ggerganov
    ggerganov committed Jan 10, 2024
    Configuration menu
    Copy the full SHA
    07a1b05 View commit details
    Browse the repository at this point in the history

  4. metal : page align the data ptr (#4854)

    @ggerganov
    ggerganov committed Jan 10, 2024
    Configuration menu
    Copy the full SHA
    3cd0cbb View commit details
    Browse the repository at this point in the history

  5. Apply suggestions from code review 

    Co-authored-by: Johannes Gäßler <[email protected]>
    @slaren @JohannesGaessler
    slaren and JohannesGaessler committed Jan 10, 2024
    Configuration menu
    Copy the full SHA
    74066f8 View commit details
    Browse the repository at this point in the history

  6. cuda : fix split buffer free

    @slaren
    slaren committed Jan 10, 2024
    Configuration menu
    Copy the full SHA
    c522c11 View commit details
    Browse the repository at this point in the history

Commits on Jan 11, 2024

  1. address review comments

    @slaren
    slaren committed Jan 11, 2024
    Configuration menu
    Copy the full SHA
    9d4ba6e View commit details
    Browse the repository at this point in the history

  2. llama-bench : add split-mode parameter

    @slaren
    slaren committed Jan 11, 2024
    Configuration menu
    Copy the full SHA
    d83c084 View commit details
    Browse the repository at this point in the history

  3. fix whitespace

    @slaren
    slaren committed Jan 11, 2024
    Configuration menu
    Copy the full SHA
    6dcc42b View commit details
    Browse the repository at this point in the history

  4. opencl : fix double initialization

    @slaren
    slaren committed Jan 11, 2024
    Configuration menu
    Copy the full SHA
    42aa835 View commit details
    Browse the repository at this point in the history

  5. Merge remote-tracking branch 'origin/master' into sl/backend-sched

    @slaren
    slaren committed Jan 11, 2024
    Configuration menu
    Copy the full SHA
    c3681af View commit details
    Browse the repository at this point in the history

  6. server : add --split-mode parameter

    @slaren
    slaren committed Jan 11, 2024
    Configuration menu
    Copy the full SHA
    c486719 View commit details
    Browse the repository at this point in the history

Commits on Jan 12, 2024

  1. use async copy and compute to improve multi-gpu performance 

    ggml-ci
    @slaren
    slaren committed Jan 12, 2024
    Configuration menu
    Copy the full SHA
    23c14ef View commit details
    Browse the repository at this point in the history

  2. use async memcpys to copy the graph outputs to the CPU

    @slaren
    slaren committed Jan 12, 2024
    Configuration menu
    Copy the full SHA
    e73009e View commit details
    Browse the repository at this point in the history

  3. fix opencl

    @slaren
    slaren committed Jan 12, 2024
    Configuration menu
    Copy the full SHA
    1e7694e View commit details
    Browse the repository at this point in the history

  4. Merge remote-tracking branch 'origin/master' into sl/backend-sched

    @slaren
    slaren committed Jan 12, 2024
    Configuration menu
    Copy the full SHA
    458674c View commit details
    Browse the repository at this point in the history

  5. use a host buffer for the cpu compute buffer for faster copies to the… 

    … gpu
    @slaren
    slaren committed Jan 12, 2024
    Configuration menu
    Copy the full SHA
    53ae0dd View commit details
    Browse the repository at this point in the history