- โWhat is
ilab
- ๐ Requirements
- โ Getting started
- ๐ป Creating new knowledge or skills and training the model
- ๐ Submit your new knowledge or skills
- ๐ฌ Contributing to Instruct-Lab CLI
InstructLab ๐ถ uses a novel synthetic data-based alignment tuning method for Large Language Models (LLMs.) The "lab" in InstructLab ๐ถ stands for Large-Scale Alignment for ChatBots [1].
[1] Shivchander Sudalairaj*, Abhishek Bhandwaldar*, Aldo Pareja*, Kai Xu, David D. Cox, Akash Srivastava*. "LAB: Large-Scale Alignment for ChatBots", arXiv preprint arXiv: 2403.01081, 2024. (* denotes equal contributions)
ilab
is a Command-Line Interface (CLI) tool that allows you to:
- Download a pre-trained Large Language Model (LLM).
- Chat with the LLM.
To add new knowledge and skills to the pre-trained LLM you have to add new information to the companion taxonomy repository. After that is done, you can:
- Use
ilab
to generate new synthetic training data based on the changes in your localtaxonomy
repository. - Re-train the LLM with the new training data.
- Chat with the re-trained LLM to see the results.
The full process is described graphically in the workflow diagram.
- ๐ Apple M1/M2/M3 Mac or ๐ง Linux system (tested on Fedora). We anticipate support for more operating systems in the future.
- C++ compiler
- Python 3.9+ (<3.12 for PyTorch JIT)
- Approximately 60GB disk space (entire process)
NOTE: PyTorch 2.2.1 does not support
torch.compile
with Python 3.12. On Fedora 39+, installpython3.11-devel
and create the virtual env withpython3.11
if you wish to use PyTorch's JIT compiler.
-
If you are on Fedora Linux, install C++, Python 3.9+, and other necessary tools by running the following command:
sudo dnf install g++ gcc make pip python3 python3-devel python3-GitPython
Optional: If g++ is not found, try 'gcc-c++' by running the following command:
sudo dnf install gcc-c++ gcc make pip python3 python3-devel python3-GitPython
-
Create a new directory called
instructlab
to store the files theilab
CLI needs when running andcd
into the directory by running the following command:mkdir instructlab cd instructlab
NOTE: The following steps in this document use Python venv for virtual environments. However, if you use another tool such as pyenv or Conda Miniforge for managing Python environments on your machine continue to use that tool instead. Otherwise, you may have issues with packages that are installed but not found in
venv
. -
Install and activate your
venv
environment by running the following command:python3 -m venv venv source venv/bin/activate pip install git+ssh:https://[email protected]/instructlab/instructlab.git@stable
NOTE: โณ
pip install
may take some time, depending on your internet connection. -
From your
venv
environment, verifyilab
is installed correctly, by running theilab
command.ilab
(venv) $ ilab Usage: ilab [OPTIONS] COMMAND [ARGS]... CLI for interacting with InstructLab. If this is your first time running `ilab`, it's best to start with `ilab init` to create the environment Options: --config PATH Path to a configuration file. [default: config.yaml] --help Show this message and exit. Commands: chat Run a chat using the modified model check Check that taxonomy is valid convert Converts model to GGUF download Download the model(s) to train generate Generates synthetic data to enhance your example data init Initializes environment for InstructLab list Lists taxonomy files that have changed since a reference commit (default origin/main) serve Start a local server test Runs basic test to ensure model correctness train Takes synthetic data generated locally with `ilab generate`...
IMPORTANT: every
ilab
command needs to be run from within your Python virtual environment. To enter the Python environment, run the following command:source venv/bin/activate
-
Initialize
ilab
by running the following command:ilab init
Welcome to InstructLab CLI. This guide will help you set up your environment. Please provide the following values to initiate the environment [press Enter for defaults]: Path to taxonomy repo [taxonomy]: <ENTER>
-
When prompted by the interface, press Enter to add a new default
config.yaml
file. -
When prompted, clone the
[email protected]:instructlab/taxonomy.git
repository into the current directory by typing y.Optional: If you want to point to an existing local clone of the
taxonomy
repository, you can pass the path interactively or alternatively with the--taxonomy-path
flag.(venv) $ ilab init Welcome to InstructLab CLI. This guide will help you set up your environment. Please provide the following values to initiate the environment [press Enter for defaults]: Path to taxonomy repo [taxonomy]: <ENTER> `taxonomy` seems to not exists or is empty. Should I clone [email protected]:instructlab/taxonomy.git for you? [y/N]: y Cloning [email protected]:instructlab/taxonomy.git... Generating `config.yaml` in the current directory... Initialization completed successfully, you're ready to start using `lab`. Enjoy!
ilab
will use the default configuration file unless otherwise specified. You can override this behavior with the--config
parameter for anyilab
command.
-
Run the
ilab download
command.ilab download
ilab download
will download a pre-trained model (~4.4G) from HuggingFace and store it in amodels
directory:(venv) $ ilab download Downloading model from instructlab/merlinite-7b-lab-GGUF@main to models... (venv) $ ls models merlinite-7b-lab-Q4_K_M.gguf
NOTE โณ This command can take few minutes or immediately depending on your internet connection or model is cached. If you have issues connecting to Hugging Face, refer to the Hugging Face discussion forum for more details.
-
Serve the model by running the following command:
ilab serve
Once the model is served and ready, you'll see the following output:
(venv) $ ilab serve INFO 2024-03-02 02:21:11,352 lab.py:201 Using model 'models/ggml-merlinite-7b-lab-Q4_K_M.gguf' with -1 gpu-layers and 4096 max context size. Starting server process After application startup complete see https://127.0.0.1:8000/docs for API. Press CTRL+C to shut down the server.
NOTE: If multiple
ilab
clients try to connect to the same InstructLab server at the same time, the 1st will connect to the server while the others will start their own temporary server. This will require additional resources on the host machine.
Because you're serving the model in one terminal window, you will have to create a new window and re-activate your Python virtual environment to run ilab chat
command:
source venv/bin/activate
ilab chat
Before you start adding new skills and knowledge to your model, you can check its baseline performance by asking it a question such as what is the capital of Canada?
.
NOTE: the model needs to be trained with the generated synthetic data to use the new skills or knowledge
(venv) $ ilab chat
โญโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ system โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ Welcome to Chat CLI w/ GGML-MERLINITE-7B-lab-Q4_K_M (type /h for help) โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
>>b> what is the capital of Canada [S][default]
โญโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ ggml-merlinite-7b-lab-Q4_K_M โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ The capital city of Canada is Ottawa. It is located in the province of Ontario, on the southern banks of the Ottawa River in the eastern portion of southern Ontario. The city serves as the political center for Canada, as it is home to โ
โ Parliament Hill, which houses the House of Commons, Senate, Supreme Court, and Cabinet of Canada. Ottawa has a rich history and cultural significance, making it an essential part of Canada's identity. โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ elapsed 12.008 seconds โโฏ
>>> [S][default]
Contribute new knowledge or compositional skills to your local taxonomy repository.
Detailed contribution instructions can be found in the taxonomy repository.
-
List your new data by running the following command:
ilab diff
-
To ensure
ilab
is registering your new knowledge or skills, you can run theilab diff
command. The following is the expected result after adding the new compositional skill foo-lang:(venv) $ ilab diff compositional_skills/writing/freeform/foo-lang/foo-lang.yaml Taxonomy in /taxonomy/ is valid :)
-
To generate a synthetic dataset based on your newly added knowledge or skill set in taxonomy repository, run the following command:
ilab generate
NOTE: โณ This can take from 15 minutes to 1+ hours to complete, depending on your computing resources.
(venv) $ ilab generate INFO 2024-02-29 19:09:48,804 lab.py:250 Generating model 'ggml-merlinite-7b-lab-Q4_K_M' using 10 CPUs, taxonomy: '/home/username/instructlab/taxonomy' and seed 'seed_tasks.json' 0%|##########| 0/100 Cannot find prompt.txt. Using default prompt. 98%|##########| 98/100 INFO 2024-02-29 20:49:27,582 generate_data.py:428 Generation took 5978.78s
The synthetic data set will be three files in the newly created
generated
directory namedgenerated*.json
,test*.jsonl
, andtrain*.jsonl
. -
Verify the files have been created by running the
ls generated
command.(venv) $ ls generated/ 'generated_ggml-merlinite-7b-lab-0226-Q4_K_M_2024-02-29T19 09 48.json' 'train_ggml-merlinite-7b-lab-0226-Q4_K_M_2024-02-29T19 09 48.jsonl' 'test_ggml-merlinite-7b-lab-0226-Q4_K_M_2024-02-29T19 09 48.jsonl'
Optional: It is also possible to run the generate step against a different model via an OpenAI-compatible API. For example, the one spawned by
ilab serve
or any remote or locally hosted LLM (e.g. via ollama, LM Studio, etc.). Run the following command:ilab generate --endpoint-url https://localhost:8000/v1
There are three options to train the model on your synthetic data-enhanced dataset.
Note: Every
ilab
command needs to be run from within your Python virtual environment.
ilab train
NOTE: โณ This step can take several hours to complete depending on your computing resources.
ilab train
outputs a brand-new model that can be served in the models
directory called ggml-model-f16.gguf
.
(venv) $ ls models
ggml-merlinite-7b-lab-Q4_K_M.gguf ggml-model-f16.gguf
NOTE:
ilab train
ships with experimental support for GPU acceleration with Nvidia CUDA or AMD ROCm. See the GPU acceleration documentation for more details.
To train the model locally on your M-Series Mac is as easy as running:
ilab train
Note: โณ This process will take a little while to complete (time can vary based on hardware and output of
ilab generate
but on the order of 20 minutes to 1+ hours)
ilab train
outputs a brand-new model that is saved in the <model_name>-mlx-q
directory called adapters.npz
(in Numpy's compressed array format). For example:
(venv) $ ls instructlab-merlinite-7b-lab-mlx-q
adapters-010.npz adapters-050.npz adapters-090.npz config.json tokenizer.model
adapters-020.npz adapters-060.npz adapters-100.npz model.safetensors tokenizer_config.json
adapters-030.npz adapters-070.npz adapters.npz special_tokens_map.json
adapters-040.npz adapters-080.npz added_tokens.json tokenizer.jso
Follow the instructions in Training.
โณ Approximate amount of time taken on each platform:
- Google Colab: 0.5-2.5 hours with a T4 GPU
- Kaggle: ~8 hours with a P100 GPU.
After that's done, you can play with your model directly in the Google Colab or Kaggle notebook. Model trained on the cloud will be saved on the cloud. The model can also be downloaded and served locally.
-
Run the following command to test the model:
ilab test
NOTE: ๐ This step is only implemented for macOS with M-series chips (for now)
The output from the command will consist of a series of outputs from the model before and after training.
-
Stop the server you have running by entering
ctrl+c
keys in the terminal running the server.IMPORTANT:
-
๐ This step is only implemented for macOS with M-series chips (for now).
-
Before serving the newly trained model you must convert it to work with the
ilab
cli. Theilab convert
command converts the new model into quantized GGUF format which is required by the server to host the model in theilab serve
command.
-
-
Convert the newly trained model by running the following command:
ilab convert
-
Serve the newly trained model locally via
ilab serve
command with the--model
argument to specify your new model:ilab serve --model-path <New model name>
Which model shouold you select to serve? After running the
ilab convert
command, a few files and directories are generated. The one you will want to serve will end in.gguf
and will exist in a directory with the suffixfused-pt
. For example:instructlab-merlinite-7b-lab-mlx-q-fused-pt/ggml-model-Q4_K_M.gguf
-
Try the fine-tuned model out live using the chat interface, and see if the results are better than the untrained version of the model with chat by running the following command:
ilab chat -m <New model name>
If you are interested in optimizing the quality of the model's responses, please see TROUBLESHOOTING.md
Of course, the final step is, if you've improved the model, to open a pull-request in the taxonomy repository that includes the files (e.g. qna.yaml
) with your improved data.
Check out our contributing guide to learn how to contribute.