This repository is intended as a minimal, hackable and readable example to load LLaMA (arXiv) models and run inference. In order to download the checkpoints and tokenizer, fill this google form

For windows user you will need to clone this repository in a WSL environement. You might have to turn on Virtualisation in the BIOS

install WSL and Cuda

wsl --install

start wsl
wsl -d Ubuntu

Install Cuda in wsl https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=x86_64&Distribution=WSL-Ubuntu&target_version=2.0&target_type=deb_local

Once this is done clone the repos in wls

In a conda env with pytorch / cuda available, run

pip install -r requirements.txt

Then

pip install -e .

pip install gradio

You will need create a folder celled "models " in the root of this app on put tokenizer_checklist.ch, tokenizer.model and 7B folder inside

To start ths app type

./run.sh

Once your request is approved, you will receive links to download the tokenizer and model files. Edit the download.sh script with the signed url provided in the email to download the model weights and tokenizer.

The provided example.py can be run on a single or multi-gpu node with torchrun and will output completions for two pre-defined prompts. Using TARGET_FOLDER as defined in download.sh:

torchrun --nproc_per_node MP example.py --ckpt_dir $TARGET_FOLDER/model_size --tokenizer_path $TARGET_FOLDER/tokenizer.model

Different models require different MP values:

• 1. The download.sh script doesn't work on default bash in MacOS X
• 2. Generations are bad!
• 3. CUDA Out of memory errors
• 4. Other languages

LLaMA: Open and Efficient Foundation Language Models -- https://arxiv.org/abs/2302.13971

@article{touvron2023llama,
  title={LLaMA: Open and Efficient Foundation Language Models},
  author={Touvron, Hugo and Lavril, Thibaut and Izacard, Gautier and Martinet, Xavier and Lachaux, Marie-Anne and Lacroix, Timoth{\'e}e and Rozi{\`e}re, Baptiste and Goyal, Naman and Hambro, Eric and Azhar, Faisal and Rodriguez, Aurelien and Joulin, Armand and Grave, Edouard and Lample, Guillaume},
  journal={arXiv preprint arXiv:2302.13971},
  year={2023}
}

See MODEL_CARD.md

See the LICENSE file.