Skip to content
/ cortex Public
forked from janhq/cortex.cpp

Drop-in, local AI alternative to the OpenAI stack. Multi-engine (llama.cpp, TensorRT-LLM). Powers 👋 Jan

nitro.jan.ai/

License

AGPL-3.0 license
0 stars 116 forks Branches Tags Activity
Star
Notifications You must be signed in to change notification settings

kollya/cortex

BranchesTags
 
 

Repository files navigation

Nitro - Embeddable AI

nitrologo

Documentation - API Reference - Changelog - Bug reports - Discord

⚠️ Nitro is currently in Development: Expect breaking changes and bugs!

Features

  • Fast Inference: Built on top of the cutting-edge inference library llama.cpp, modified to be production ready.
  • Lightweight: Only 3MB, ideal for resource-sensitive environments.
  • Easily Embeddable: Simple integration into existing applications, offering flexibility.
  • Quick Setup: Approximately 10-second initialization for swift deployment.
  • Enhanced Web Framework: Incorporates drogon cpp to boost web service efficiency.

About Nitro

Nitro is a high-efficiency C++ inference engine for edge computing, powering Jan. It is lightweight and embeddable, ideal for product integration.

The binary of nitro after zipped is only ~3mb in size with none to minimal dependencies (if you use a GPU need CUDA for example) make it desirable for any edge/server deployment 👍.

Read more about Nitro at https://nitro.jan.ai/

Repo Structure

.
├── controllers
├── docs 
├── llama.cpp -> Upstream llama C++
├── nitro_deps -> Dependencies of the Nitro project as a sub-project
└── utils

Quickstart