This repo contains the data preparation, tokenization, training and inference code for BLOOMChat-176B-v1. BLOOMChat is a 176 billion parameter multilingual chat model. It is instruction tuned from BLOOM (176B) on assistant-style conversation datasets and supports conversation, question answering and generative answers in multiple languages.

We trained BLOOMChat on SambaNova DataScale systems using SambaNova’s unique Reconfigurable Dataflow Architecture The training data used to train BLOOMChat originated from OIG dataset from OpenChatKit, Dolly 2.0, and OASST1.

• Blog Post: Link
• Discord: Link
• HF Hosting: Chat with me!
• BLOOMChat-176B-v1: Link

• Clone SambaNova's Generation Data Preparation repo
• Create a virtual environment
• Set up environment using the above repo's instructions
• Run this command pip install datasets

Further preprocessing had been done on the original datasets. You can find the relevant code under data prep.

To run these files:

1. cd data_prep
2. python prepare_oa_dolly.py
3. python subsample_openchatkit.py

After running these commands there should be 2 files under the data_prep directory:

• oasst1_dolly.jsonl
• bloom_ock_100K_each.jsonl

NOTE these files are referenced in the tokenization code, so when running the tokenization scripts they need to be done within tokenization_prep otherwise the file paths will need to be changed.

DISCLAIMER: The OIG dataset preprocessed for BLOOMChat might not be 100% reproducible using OIG dataset from OpenChatKit. We will update soon about the other steps necessary to reproduce the dataset.

The next step after preprocessing the data is to tokenize the data using SambaNova's Generation Data Preparation repo. The scripts that utilize this public repository can be found under tokenization prep.

Follow the instructions on how to set up Generation Data Preparation.

For the bash scripts they take in one argument which is the absolute path to the generation data preparation repo.

Example of running the script:

1. cd tokenization_prep
2. bash tokenization_oa_dolly.sh /home/.../generative_data_prep
3. bash tokenization_openchatkit.sh /home/.../generative_data_prep

After running these scripts there should be 2 directories under the tokenization_prep directory:

• oasst1_dolly_out
• bloom_ock_100k_out

These directories contain two directories:

• hdf5
• splits

The splits directory contains the original text, the hdf5 directory contains the tokenized text which will be fed into the model.

As our models were trained on SambaNova's in-house Reconfigurable Dataflow Unit (RDU), our scripts will not work for training on GPU; however, we want to give better insight and transparency on the hyper-parameters that were used to train this model. The scripts that were ran on the RDU can be found under training. The model is ran directly from HuggingFace, using a built-in wrapper provided by SambaFlow, our SDK for the RDU. For those interested in running models on RDUs, please feel free to get in touch.

NOTE: BLOOMChat is a two step process:

1. First train a model using OIG dataset from OpenChatKit
2. Second train the above model using Dolly 2.0 and OASST1 (Final Model)

• OIG dataset from OpenChatKit
• Dolly 2.0
• OASST1

First create a python virtual environment for these packages

python3 -m venv bloomchat_venv
source bloomchat_venv/bin/activate
pip install --upgrade pip

pip install flask flask_api gunicorn pydantic accelerate huggingface_hub>=0.9.0 deepspeed>=0.7.3 deepspeed-mii==0.0.2

And then

pip install transformers==4.27.0

You will see messages like this

ERROR: deepspeed-mii 0.0.2 has requirement transformers==4.21.2, but you'll have transformers 4.27.0 which is incompatible.
Installing collected packages: transformers
  Found existing installation: transformers 4.21.2
    Uninstalling transformers-4.21.2:
      Successfully uninstalled transformers-4.21.2
Successfully installed transformers-4.27.0

Now let's git clone the huggingface/transformers-bloom-inference repo.

git clone https://github.com/huggingface/transformers-bloom-inference.git
cd transformers-bloom-inference/

And then you need to modify two files in this transformers-bloom-inference repo:

• Modifying inference_server/models/hf_accelerate.py
  • This is because for our testing of this repo we used 4 80GB A100 GPUs and would run into memory issues
• Modifying inference_server/cli.py
  • This is because the model was trained using specific human, bot tags
  • Trailing spaces may lead to subpar performance

Modifications for inference_server/models/hf_accelerate.py:

diff --git a/inference_server/models/hf_accelerate.py b/inference_server/models/hf_accelerate.py
index 9be3c3f..a8ecb1d 100644
--- a/inference_server/models/hf_accelerate.py
+++ b/inference_server/models/hf_accelerate.py
@@ -1,4 +1,5 @@
 from argparse import Namespace
+from accelerate.utils.modeling import get_max_memory
 
 import torch
 
@@ -12,6 +13,12 @@ class HFAccelerateModel(Model):
 
         kwargs = {"pretrained_model_name_or_path": args.model_name, "device_map": "auto"}
 
+        original_max_memory_dict = get_max_memory()
+
+        reduce_max_memory_dict = {device_key: int(original_max_memory_dict[device_key] * 0.85) for device_key in original_max_memory_dict}
+
+        kwargs["max_memory"] = reduce_max_memory_dict
+
         if get_world_size() > 1:
             kwargs["device_map"] = "balanced_low_0"

Modifications for inference_server/cli.py:

diff --git a/inference_server/cli.py b/inference_server/cli.py
index fc903d5..5450236 100644
--- a/inference_server/cli.py
+++ b/inference_server/cli.py
@@ -22,6 +22,9 @@ def main() -> None:
     while True:
         input_text = input("Input text: ")
 
+        input_text = input_text.strip()
+        modified_input_text = f"<human>: {input_text}\n<bot>:"
+
         if input("change generate_kwargs? [y/n] ") == "y":
             while True:
                 try:
@@ -33,7 +36,7 @@ def main() -> None:
                     print("message =", e_message)
                     continue
 
-        response = model.generate(text=[input_text], generate_kwargs=generate_kwargs)
+        response = model.generate(text=[modified_input_text], generate_kwargs=generate_kwargs)
 
         print_rank_0("Output text:", response.text[0])
         print_rank_0("Generated tokens:", response.num_generated_tokens[0])

And now you are good to go!

Running command for bf16, NO sampling

python -m inference_server.cli --model_name sambanovasystems/BLOOMChat-176B-v1 --model_class AutoModelForCausalLM --dtype bf16 --deployment_framework hf_accelerate --generate_kwargs '{"do_sample": false, "max_new_tokens": 512}'

Running command for bf16, YES sampling

python -m inference_server.cli --model_name sambanovasystems/BLOOMChat-176B-v1 --model_class AutoModelForCausalLM --dtype bf16 --deployment_framework hf_accelerate --generate_kwargs '{"do_sample": true, "temperature": 0.8, "repetition_penalty": 1.2, "top_p": 0.9, "max_new_tokens": 512}'

Running command for int8 (sub optimal performance, but fast inference time) NO sampling:

python -m inference_server.cli --model_name sambanovasystems/BLOOMChat-176B-v1 --model_class AutoModelForCausalLM --dtype int8 --deployment_framework hf_accelerate --generate_kwargs '{"do_sample": false, "max_new_tokens": 512}'

Running command for int8 (sub optimal performance, but fast inference time) YES sampling:

python -m inference_server.cli --model_name sambanovasystems/BLOOMChat-176B-v1 --model_class AutoModelForCausalLM --dtype int8 --deployment_framework hf_accelerate --generate_kwargs '{"do_sample": true, "temperature": 0.8, "repetition_penalty": 1.2, "top_p": 0.9, "max_new_tokens": 512}'

DISCLAIMER: When using int8, the results will be subpar compared to bf16 as the model is being quantized.

• Temperature: 0.8
• Repetition penalty: 1.2
• Top-p: 0.9
• Max generated tokens: 512

Input text: Write a script in which Bob accidentally breaks his dad's guitar

Input text: Create an itemized list of tasks to complete to start a clothing brand

Input text: 十七岁的风是什么颜色的?

The inference code to run the model can be found under RDU quick start. This code requires the SambaFlow SDK to execute. For those interested in running models on RDUs, please feel free to get in touch.

We would like to extend our gratitude to Together for their insightful technical discussions on overall project planning, data processing, model training, human evaluation experiment design, open-source endeavors, and their contributions on data processing code on OpenChatKit, OASST1, and Dolly 2.0.

We would also like to extend our gratitude to Jue Wang who was the main contributor from Together on this collaboration.

We are grateful to the various researchers and open-source projects that have contributed to the development of BLOOMChat. We thank BigScience for providing the BLOOM model, which served as the base for our instruction tuning. We also thank LAION for their OIG dataset, OpenAssistant Conversations Dataset (OASST1) and also thank Databricks for providing Dolly 2.0, to provide the dataset that we instruction tuned on.

@software{bloomchat,
  title = {{BLOOMChat: a New Open Multilingual Chat LLM}},
  author = {SambaNova Systems, Together Computer},
  url = {https://huggingface.co/sambanovasystems/BLOOMChat-176B-v1}
  month = {5},
  year = {2023},
  version = {1.0},
}