A Gradio web UI for Large Language Models.

Its goal is to become the AUTOMATIC1111/stable-diffusion-webui of text generation.

• 3 interface modes: default (two columns), notebook, and chat
• Multiple model backends: Transformers, llama.cpp (through llama-cpp-python), ExLlama, ExLlamaV2, AutoGPTQ, AutoAWQ, GPTQ-for-LLaMa, CTransformers
• Dropdown menu for quickly switching between different models
• LoRA: load and unload LoRAs on the fly, train a new LoRA using QLoRA
• Precise instruction templates for chat mode, including Llama-2-chat, Alpaca, Vicuna, WizardLM, StableLM, and many others
• 4-bit, 8-bit, and CPU inference through the transformers library
• Use llama.cpp models with transformers samplers (llamacpp_HF loader)
• Multimodal pipelines, including LLaVA and MiniGPT-4
• Extensions framework
• Custom chat characters
• Markdown output with LaTeX rendering, to use for instance with GALACTICA
• OpenAI-compatible API server with Chat and Completions endpoints -- see the examples

To learn how to use the various features, check out the Documentation:

https://github.com/oobabooga/text-generation-webui/wiki

1. Clone or download the repository.
2. Run the start_linux.sh, start_windows.bat, start_macos.sh, or start_wsl.bat script depending on your OS.
3. Select your GPU vendor when asked.
4. Have fun!

The script creates a folder called installer_files where it sets up a Conda environment using Miniconda. The installation is self-contained: if you want to reinstall, just delete installer_files and run the start script again.

To launch the webui in the future after it is already installed, run the same start script.

Run update_linux.sh, update_windows.bat, update_macos.sh, or update_wsl.bat.

If you ever need to install something manually in the installer_files environment, you can launch an interactive shell using the cmd script: cmd_linux.sh, cmd_windows.bat, cmd_macos.sh, or cmd_wsl.bat.

To define persistent command-line flags like --listen or --api, edit the CMD_FLAGS.txt file with a text editor and add them there. Flags can also be provided directly to the start scripts, for instance, ./start-linux.sh --listen.

• There is no need to run any of those scripts as admin/root.
• For additional instructions about AMD setup, WSL setup, and nvcc installation, consult the documentation.
• The installer has been tested mostly on NVIDIA GPUs. If you can find a way to improve it for your AMD/Intel Arc/Mac Metal GPU, you are highly encouraged to submit a PR to this repository. The main file to be edited is one_click.py.
• For automated installation, you can use the GPU_CHOICE, USE_CUDA118, LAUNCH_AFTER_INSTALL, and INSTALL_EXTENSIONS environment variables. For instance: GPU_CHOICE=A USE_CUDA118=FALSE LAUNCH_AFTER_INSTALL=FALSE INSTALL_EXTENSIONS=FALSE ./start_linux.sh.

Recommended if you have some experience with the command-line.

https://docs.conda.io/en/latest/miniconda.html

On Linux or WSL, it can be automatically installed with these two commands (source):

curl -sL "https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh" > "Miniconda3.sh"
bash Miniconda3.sh

conda create -n textgen python=3.11
conda activate textgen

The up-to-date commands can be found here: https://pytorch.org/get-started/locally/.

For NVIDIA, you may also need to manually install the CUDA runtime libraries:

conda install -y -c "nvidia/label/cuda-12.1.0" cuda-runtime

git clone https://github.com/oobabooga/text-generation-webui
cd text-generation-webui
pip install -r <requirements file according to table below>

Requirements file to use:

1. Use requirements_cpu_only.txt or requirements_cpu_only_noavx2.txt in the command above.

2. Manually install llama-cpp-python using the appropriate command for your hardware: Installation from PyPI.

   • Use the LLAMA_HIPBLAS=on toggle.
   • Note the Windows remarks.

3. Manually install AutoGPTQ: Installation.

   • Perform the from-source installation - there are no prebuilt ROCm packages for Windows.

4. Manually install ExLlama by simply cloning it into the repositories folder (it will be automatically compiled at runtime after that):

cd text-generation-webui
git clone https://github.com/turboderp/exllama repositories/exllama

1. For Kepler GPUs and older, you will need to install CUDA 11.8 instead of 12:

pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
conda install -y -c "nvidia/label/cuda-11.8.0" cuda-runtime

2. bitsandbytes >= 0.39 may not work. In that case, to use --load-in-8bit, you may have to downgrade like this:
   • Linux: pip install bitsandbytes==0.38.1
   • Windows: pip install https://github.com/jllllll/bitsandbytes-windows-webui/raw/main/bitsandbytes-0.38.1-py3-none-any.whl

The requirements*.txt above contain various precompiled wheels. If you wish to compile things manually, or if you need to because no suitable wheels are available for your hardware, you can use requirements_nowheels.txt and then install your desired loaders manually.

ln -s docker/{nvidia/Dockerfile,docker-compose.yml,.dockerignore} .
cp docker/.env.example .env
# Edit .env and set: 
#   TORCH_CUDA_ARCH_LIST based on your GPU model
#   APP_RUNTIME_GID      your host user's group id (run `id -g` in a terminal)
#   BUILD_EXTENIONS      optionally add comma separated list of extensions to build
docker compose up --build

• You need to have Docker Compose v2.17 or higher installed. See this guide for instructions.
• For additional docker files, check out this repository.

From time to time, the requirements*.txt changes. To update, use these commands:

conda activate textgen
cd text-generation-webui
pip install -r <requirements file that you've used> --upgrade

Models should be placed in the text-generation-webui/models folder. They are usually downloaded from Hugging Face.

• Transformers or GPTQ models are made of several files and must be placed in a subfolder. Example:

text-generation-webui
├── models
│   ├── lmsys_vicuna-33b-v1.3
│   │   ├── config.json
│   │   ├── generation_config.json
│   │   ├── pytorch_model-00001-of-00007.bin
│   │   ├── pytorch_model-00002-of-00007.bin
│   │   ├── pytorch_model-00003-of-00007.bin
│   │   ├── pytorch_model-00004-of-00007.bin
│   │   ├── pytorch_model-00005-of-00007.bin
│   │   ├── pytorch_model-00006-of-00007.bin
│   │   ├── pytorch_model-00007-of-00007.bin
│   │   ├── pytorch_model.bin.index.json
│   │   ├── special_tokens_map.json
│   │   ├── tokenizer_config.json
│   │   └── tokenizer.model

• GGUF models are a single file and should be placed directly into models. Example:

text-generation-webui
├── models
│   ├── llama-2-13b-chat.Q4_K_M.gguf

In both cases, you can use the "Model" tab of the UI to download the model from Hugging Face automatically. It is also possible to download via the command-line with python download-model.py organization/model (use --help to see all the options).

python download-model.py EleutherAI/gpt-j-6B --text-only

conda activate textgen
cd text-generation-webui
python server.py

Then browse to

https://localhost:7860/?__theme=dark

Optionally, you can use the following command-line flags:

⚠️ Requires minimum compute of 7.0 on Windows at the moment.

https://colab.research.google.com/github/oobabooga/text-generation-webui/blob/main/Colab-TextGen-GPU.ipynb

If you would like to contribute to the project, check out the Contributing guidelines.

• Subreddit: https://www.reddit.com/r/oobabooga/
• Discord: https://discord.gg/jwZCF2dPQN

In August 2023, Andreessen Horowitz (a16z) provided a generous grant to encourage and support my independent work on this project. I am extremely grateful for their trust and recognition, which will allow me to dedicate more time towards realizing the full potential of text-generation-webui.