Aphrodite is the official backend engine for PygmalionAI. It is designed to serve as the inference endpoint for the PygmalionAI website, and to allow serving the Pygmalion models to a large number of users with blazing fast speeds (thanks to vLLM's Paged Attention).

Aphrodite builds upon and integrates the exceptional work from various projects.

The compute necessary for Aphrodite's development is provided by Arc Compute.

• Continuous Batching
• Efficient K/V management with PagedAttention from vLLM
• Optimized CUDA kernels for improved inference
• Quantization support via AQLM, AWQ, Bitsandbytes, EXL2, GGUF, GPTQ, QuIP#, Smoothquant+, and SqueezeLLM
• Distributed inference
• Variety of sampling methods (Mirostat, Locally Typical Sampling, Tail-Free Sampling, etc)
• 8-bit KV Cache for higher context lengths and throughput, at both FP8 and INT8 formats.

pip install aphrodite-engine

python -m aphrodite.endpoints.openai.api_server --model PygmalionAI/pygmalion-2-7b

This will create a OpenAI-compatible API server that can be accessed at port 2242 of the localhost. You can plug in the API into a UI that supports OpenAI, such as SillyTavern.

You can play around with the engine in the demo here:

We provide the flexibility of a Docker container, enabling it to interact with another Docker container (Worker) through a specific Docker network via internal port 7860. Additionally, it has the capability to communicate with the host via port 2242, if the worker is not containerized.

docker network create <network_name>

To deploy Aphrodite Engine using Docker, execute the following command:

docker run -it -p 2242:7860 --network <networ_name> --gpus "all" --shm-size 8g --name <container_name> -e MODEL_NAME="PygmalionAI/pygmalion-2-7b" -e KOBOLD_API="true" alpindale/aphrodite-engine

This command launches the Aphrodite Engine Docker container with the following configurations:

• It joins the Docker network named <network_name>.
• Utilizes all available GPUs for processing. Alternatively, you can specify a specific GPU card number (e.g., '1' to select GPU number 1). Sets the shared memory size to 8 GB.
• Maps port 2242 on the host to port 7860 internally within the container. The internal port should correspond to the port specified in the worker's bridgeData.yaml if it is containerized.
• Starts the container in interactive mode (-it) with the name <container_name>.
• Specifies environment variables MODEL_NAME and KOBOLD_API for the Aphrodite Engine.
• Uses the Docker image alpindale/aphrodite-engine.

This command will download the Aphrodite Engine image (approximately 9GiB) and then launch the engine with the Pygmalion-2-7b model on port 7860 (internal) and port 2242 (external). For the complete list of environment variables, please refer to here

See here for the Compose file to use with Docker Compose.

• Operating System: Linux (or WSL for Windows)
• Python: at least 3.8

For windows users, it's recommended to use tabbyAPI instead, if you do not need batching support.

• CUDA >= 11

For supported GPUs, see here. Generally speaking, all semi-modern GPUs are supported - down to Pascal (GTX 10xx, P40, etc.)

• Using pip
• Build from source

For usage, please refer to the wiki page for detailed instructions. Aphrodite provides many different options for LLM inference, so please read through the list of options here.

Speeds vary with different GPUs, model sizes, quantization schemes, batch sizes, etc. Here are some baseline benchmarks conducted by requesting as many completions as possible from the API server.

These are the speeds a user would normally get if they request a single output with a sizable prompt and output length. Essentially, normal chatting experience.

The following results were gathered by sending a request with 8192 prompt tokens and requesting 1024 tokens with ignore_eos=True.

GPU: NVIDIA A40, Mistral 7B. Baseline is the same model loaded with text-generation-webui in FP16.

This table is outdated, will be replaced soon.

1. By design, Aphrodite takes up 90% of your GPU's VRAM. If you're not serving an LLM at scale, you may want to limit the amount of memory it takes up. You can do this in the API example by launching the server with the --gpu-memory-utilization 0.6 (0.6 means 60%).

2. You can view the full list of commands by running python -m aphrodite.endpoints.openai.api_server --help.

3. Context Length extension via the RoPE method is supported for most models. Use the command-line flag --max-model-len to specify a desired context length and the engine will adjust the RoPE scaling accordingly.

4. Please refer to the FAQ & Issues if you run into problems. If you don't find an answer there, please make an issue.

Aphrodite Engine would have not been possible without the phenomenal work of other open-source projects. Credits go to:

• vLLM (CacheFlow)
• TensorRT-LLM
• xFormers
• AutoAWQ
• AutoGPTQ
• SqueezeLLM
• Exllamav2
• TabbyAPI
• AQLM
• KoboldAI
• Text Generation WebUI
• Megatron-LM
• Ray

Everyone is welcome to contribute. You can support the project by opening Pull Requests for new features, fixes, or general UX improvements.