Nomic Embed integration via llama.cpp BERT implementation

[cebtenzzre]

Backend changes

• Removed bert.cpp
• Implemented LLamaModel::embed on top of new llama.cpp BERT implementation, which intelligently splits and batches inputs depending on length and n_ctx
• Added BERT and Nomic BERT to llama.cpp arch whitelist
• Explicitly blacklist old MiniLM GGUF
• Dynamic supportsCompletion/supportsEmbedding based on model arch
• Implemented compatibility with llama : fix embeddings ggerganov/llama.cpp#5796
• Implemented Matryoshka dimensionality reduction for nomic-embed-text-v1.5
• Set n_batch to trained context length for embedding models
• Implemented compatibility with Nomic Atlas API:
  • Built-in support for prefix to add to each embedding input
  • Hardcoded map of model names to query/storage prefixes
  • Change overlap to 8 tokens
  • Use the "nomic empty" placeholder when embedding an empty string
  • Add a mode where the input is truncated instead of averaging the embeddings
  • Optional length limits that match Atlas
  • L2 normalize before and after averaging embeddings
• Return proper error strings or print warnings in certain cases instead of opaque failure or incorrect results:
  • Error when setting dimensionality on a model that isn't nomic-embed-text-v1.5
  • Error when setting dimensionality to larger than n_embd
  • Error when using a prefix that is not known to the model (if in hardcoded map)
  • Error when n_ctx - prefix is smaller than overlap
  • Warn when text tokenizes to an empty result
• Added MiniLM, Nomic Embed v1 and v1.5 to models3.json
• Implement embeddingModel key in models3.json so they can be identified before downloading

Python bindings changes

• Embed4All now accepts a list of texts to embed
• Added prefix, domensionality, do_mean, and atlas parameters to Embed4All
• Dimensionality argument is pre-checked for sanity

Chat UI changes

• Identify embedding models based on GGUF metadata, and remove special-casing of MiniLM GGUF
• Use appropriate task type automatically in the chat UI, for any known embedding model
• Restricted to use MiniLM for local embeddings for now, but the whitelist can easily be removed

TODO

• Update models3.json based on what GPT4All release this will be included in
• Add example usage to python docs
• Implement Nomic python client integration with local/dynamic mode

[manyoso]

Please address comments.