Skip to content
This repository has been archived by the owner on Jun 24, 2024. It is now read-only.

#107 is broken on macOS Apple Silicon #109

Closed
philpax opened this issue Apr 4, 2023 · 0 comments
Closed

#107 is broken on macOS Apple Silicon #109

philpax opened this issue Apr 4, 2023 · 0 comments
Labels
issue:bug Something isn't working

Comments

@philpax
Copy link
Collaborator

philpax commented Apr 4, 2023

Inference with both ggml and ggmf is broken on macOS with #107. It generates garbage:

llama-rs # head -c 4 ../llama-models/ggml-alpaca-7b-q4.bin
lmgg

# previous commit
llama-rs # git log --oneline -n 1
6dde2b6 (HEAD) Merge pull request #100 from rustformers/document_working_models

llama-rs # RUST_LOG=error cargo run --release --bin llama-cli -- -m ../llama-models/ggml-alpaca-7b-q4.bin -p "You are a son of a " --top-k 1
    Finished release [optimized] target(s) in 0.14s
     Running `target/release/llama-cli -m ../llama-models/ggml-alpaca-7b-q4.bin -p 'You are a son of a ' --top-k 1`
You are a son of a 70 year old man who has been diagnosed with terminal cancer. He is in the process of making his will and^C

# main
llama-rs # git checkout main
Previous HEAD position was 6dde2b6 Merge pull request #100 from rustformers/document_working_models
Switched to branch 'main'
Your branch is up to date with 'upstream/main'.
llama-rs # git log --oneline -n 1
f103ae9 (HEAD -> main, upstream/main) Merge pull request #104 from rustformers/fix/num_cpu_cli_arg

llama-rs # RUST_LOG=error cargo run --release --bin llama-cli -- -m ../llama-models/ggml-alpaca-7b-q4.bin -p "You are a son of a " --top-k 1
    Finished release [optimized] target(s) in 4.27s
     Running `target/release/llama-cli -m ../llama-models/ggml-alpaca-7b-q4.bin -p 'You are a son of a ' --top-k 1`
You are a son of a , and. (! R Al to in/ ///-0 E)[ c. that…7 C. your you from6Al. , t. …
 - – & back2M !!!!!!!^C

This seems to work fine on Windows x86-64 (me) and Linux x86-64 (setzer), but macOS M1 (me and tonloc on the Discord) have garbage generated.

@philpax philpax added the issue:bug Something isn't working label Apr 4, 2023
@philpax philpax mentioned this issue Apr 5, 2023
4 tasks
@philpax philpax closed this as completed in b71051a Apr 5, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
issue:bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant