Prepare for release

[philpax]

Mostly housekeeping.

Closes #57.

Still to do:

• Document llama-rs
• Update to latest llama.cpp #62
• Use usize over i32 where possible #18
• Use RMSNorm for normalization #80
• Make InferenceSession Clone-able #48
• API footgun: infer_next_token still works after end of text #44
• Fix performance on Apple Silicon

[philpax]

This is good to go aside from the AS fix (need to test on work laptop).

Something I noticed is that InferenceParameters contains play_back_previous_tokens (only relevant to inference_with_prompt) and inference_with_prompt takes maximum_token_count as a parameter - we should consider moving these into a specific struct for parameters for inference_with_prompt only, like

    pub fn inference_with_prompt<E: std::error::Error + 'static>(
        &mut self,
        model: &Model,
        vocab: &Vocabulary,
        inference_params: &InferenceParameters,
        inference_with_prompt_params: &InferenceWithPromptParameters,
        prompt: &str,
        rng: &mut impl rand::Rng,
        callback: impl Fn(&str) -> Result<(), E>,
    ) -> Result<InferenceStats, InferenceError> {
}

We could also potentially move the &mut impl rand::Rng into InferenceParameters. I'm not sure if this is a good idea yet, but it should let us simplify the number of arguments some more.

Is there a context in which you would use the Vocabulary of a model without the model itself? I'm leaning towards moving that into Model, too.

[KerfuffleV2]

We could also potentially move the &mut impl rand::Rng into InferenceParameters.

You'd have to add a lifetime to the struct and maybe it would need to be mutable also, so probably not worth it. (Also, that's a lot of tasks checked off. Nice work!)

[hhamud]

I'm on apple silicon and I haven't felt any performance issues versus the original implementation of llama.cpp in C. What are the performance issues?

[philpax]

We don't compile with any flags with ARM, so the compiler won't use NEON etc. Easy enough to fix, just haven't yet

[setzer22]

Thanks a lot for the changes! :) Everything looks good, there's just a very minor nitpick comment from my end.

Also, why does ggml.rs appear as a removed file? The code seems to still be using it, and I don't see where that code has moved to 🤔

[philpax]

Thanks a lot for the changes! :) Everything looks good, there's just a very minor nitpick comment from my end.

Also, why does ggml.rs appear as a removed file? The code seems to still be using it, and I don't see where that code has moved to 🤔

It's moved to ggml/src/lib.rs. Not sure why Git can't detect the move.

[setzer22]

LGTM! 😄