Beyond containers, expand containerd to manage LLMs on Apple-Silicon Macs

[yeahdongcn]

I've been experimenting with Ollama and discovered that its CLI is quite similar to Docker and its models are OCI artifacts. As such, I wondered if it would be possible to separate the OCI image management from the LLM runtime, just like how containerd replaces Docker in Kubernetes.
After making some hack changes, I was able to decouple the two: a new snapshotter in containerd and wrap llama.cpp as an LLM runtime (a single binary on Mac without extra dependencies).
This allows me to join my Mac to our existing Kubernetes cluster and apply a deployment YAML file to spin up multiple replicas of the LLM service. Even more exciting is that this setup allows me to deploy Linux containers for upper-level applications, giving me access to the LLM service from various environments.
Please check out https://github.com/makllama/makllama for more information. In the demo video, a standard Kubernetes deployment (tinyllama) with 2 replicas is deployed on Mac and can be accessed by a Linux pod.