Website | Docs | Tutorials | Playground | Blog | Discord

NOS is a fast and flexible PyTorch inference server that runs on any cloud or AI HW.

• 👩‍💻 Easy-to-use: Built for PyTorch and designed to optimize, serve and auto-scale Pytorch models in production without compromising on developer experience.
• 🥷 Multi-modal & Multi-model: Serve multiple foundational AI models (LLMs, Diffusion, Embeddings, Speech-to-Text and Object Detection) simultaneously, in a single server.
• ⚙️ HW-aware Runtime: Deploy PyTorch models effortlessly on modern AI accelerators (NVIDIA GPUs, AWS Inferentia2, AMD - coming soon, and even CPUs).
• ☁️ Cloud-agnostic Containers: Run on any cloud (AWS, GCP, Azure, Lambda Labs, On-Prem) with our ready-to-use inference server containers.

• [Feb 2024] ✍️ [blog] Introducing the NOS Inferentia2 (inf2) runtime.
• [Jan 2024] ✍️ [blog] Serving LLMs on a budget with SkyServe.
• [Jan 2024] 📚 [docs] NOS x SkyPilot Integration page!
• [Jan 2024] ✍️ [blog] Getting started with NOS tutorials is available here!
• [Dec 2023] 🛝 [repo] We open-sourced the NOS playground to help you get started with more examples built on NOS!

We highly recommend that you go to our quickstart guide to get started. To install the NOS client, you can run the following command:

conda create -n nos python=3.8 -y
conda activate nos
pip install torch-nos

Once the client is installed, you can start the NOS server via the NOS serve CLI. This will automatically detect your local environment, download the docker runtime image and spin up the NOS server:

nos serve up --http --logging-level INFO

You are now ready to run your first inference request with NOS! You can run any of the following commands to try things out. You can set the logging level to DEBUG if you want more detailed information from the server.

NOS provides an OpenAI-compatible server with streaming support so that you can connect your favorite OpenAI-compatible LLM client to talk to NOS.

from nos.client import Client

client = Client()

model = client.Module("TinyLlama/TinyLlama-1.1B-Chat-v1.0")
response = model.chat(message="Tell me a story of 1000 words with emojis", _stream=True)

curl \
-X POST https://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
    "model": "TinyLlama/TinyLlama-1.1B-Chat-v1.0",
    "messages": [{
        "role": "user",
        "content": "Tell me a story of 1000 words with emojis"
    }],
    "temperature": 0.7,
    "stream": true
  }'

Build MidJourney discord bots in seconds.

from nos.client import Client

client = Client()

sdxl = client.Module("stabilityai/stable-diffusion-xl-base-1-0")
image, = sdxl(prompts=["hippo with glasses in a library, cartoon styling"],
              width=1024, height=1024, num_images=1)

curl \
-X POST https://localhost:8000/v1/infer \
-H 'Content-Type: application/json' \
-d '{
    "model_id": "stabilityai/stable-diffusion-xl-base-1-0",
    "inputs": {
        "prompts": ["hippo with glasses in a library, cartoon styling"],
        "width": 1024, "height": 1024,
        "num_images": 1
    }
}'

Build scalable semantic search of images/videos in minutes.

from nos.client import Client

client = Client()

clip = client.Module("openai/clip-vit-base-patch32")
txt_vec = clip.encode_text(texts=["fox jumped over the moon"])

curl \
-X POST https://localhost:8000/v1/infer \
-H 'Content-Type: application/json' \
-d '{
    "model_id": "openai/clip-vit-base-patch32",
    "method": "encode_text",
    "inputs": {
        "texts": ["fox jumped over the moon"]
    }
}'

Perform real-time audio transcription using Whisper.

from pathlib import Path
from nos.client import Client

client = Client()

model = client.Module("openai/whisper-small.en")
with client.UploadFile(Path("audio.wav")) as remote_path:
  response = model(path=remote_path)
# {"chunks": ...}

curl \
-X POST https://localhost:8000/v1/infer/file \
-H 'accept: application/json' \
-H 'Content-Type: multipart/form-data' \
-F 'model_id=openai/whisper-small.en' \
-F '[email protected]'

Run classical computer-vision tasks in 2 lines of code.

from pathlib import Path
from nos.client import Client

client = Client()

model = client.Module("yolox/medium")
response = model(images=[Image.open("image.jpg")])

curl \
-X POST https://localhost:8000/v1/infer/file \
-H 'accept: application/json' \
-H 'Content-Type: multipart/form-data' \
-F 'model_id=yolox/medium' \
-F '[email protected]'

Want to run models not supported by NOS? You can easily add your own models following the examples in the NOS Playground.

This project is licensed under the Apache-2.0 License.

NOS collects anonymous usage data using Sentry. This is used to help us understand how the community is using NOS and to help us prioritize features. You can opt-out of telemetry by setting NOS_TELEMETRY_ENABLED=0.

We welcome contributions! Please see our contributing guide for more information.

• 💬 Send us an email at [email protected] or join our Discord for help.
• 📣 Follow us on Twitter, and LinkedIn to keep up-to-date on our products.