Repository for AI model benchmarking on accelerator hardware.

First, create either a Python virtual environment with PyBuda installed or execute from a Docker container with PyBuda installed.

Installation instructions can be found at Install TT-BUDA.

Next, install the model requirements:

pip install -r requirements.txt

If your system contains a GPU device, you can use the requirements-cuda.txt file to install the correct dependencies.

To install packages, make sure your virtual environment is active.

pip install -r requirements-cuda.txt

To access the benchmarking datasets, follow these steps to set up your access to the HuggingFace Hub:

1. Create a HuggingFace Account:

   • Visit Hugging Face and create an account if you haven't already.

2. Generate User Access Token:

   • Follow the steps outlined in the HuggingFace Docs - Security Tokens to generate a User Access Token.

3. Install huggingface_hub Library:

   • Install the huggingface_hub library by running:

     pip install huggingface_hub

4. Login to HuggingFace CLI:

   • Login to the HuggingFace CLI using your User Access Token:

     huggingface-cli login

   • Enter your User Access Token when prompted.

5. Validate Setup:

   • Run the following command to verify your login status:

     huggingface-cli whoami

   • If your username is displayed, it means you are successfully logged in.

6. Dataset Access:

   • Visit HuggingFace Datasets - ImageNet-1k and follow the instructions to grant access to the ImageNet-1k dataset.

After completing the setup process, ensure that everything is working correctly:

1. Verify Hugging Face Hub Login:

   • Run the following command to verify that you are logged in to the Hugging Face Hub:

     huggingface-cli whoami

   • If your username is displayed, it means you are successfully logged in.

2. Check Dataset Access:

   • Visit the HuggingFace Datasets - ImageNet-1k page.
   • Make sure you can view and access the dataset details without any permission errors.

3. Accept Dataset Access (If Required):

   • If you encounter any permission errors while accessing the ImageNet-1k dataset, ensure that you follow the instructions provided on the dataset page to grant access.

To set up the three required datasets for running benchmarking tests within this repository, follow these steps for each dataset:

1. HuggingFace datasets: will download into ${HF_HOME}/datasets/ once HuggingFace Hub access is set up.

2. COCO Dataset: You can automatically download the COCO dataset from here. No login is required, and the dataset will be cached in ~/.cache/coco.

   To download the COCO dataset, follow these steps:

   # use another location for MLDATA_DIR if desired, below is default
   # Create the `coco` directory inside the cache directory:
   mkdir -p ~/.cache/mldata/coco
   
   # Navigate to the `coco` directory:
   cd ~/.cache/mldata/coco
   
   # Create the `images` directory:
   mkdir images
   cd images
   
   # Download the COCO validation images:
   wget http:https://images.cocodataset.org/zips/val2017.zip
   
   # Unzip the downloaded file:
   unzip val2017.zip
   
   # Move back to the `coco` directory:
   cd ..
   
   # Download the COCO train/val annotations:
   wget http:https://images.cocodataset.org/annotations/annotations_trainval2017.zip
   
   # Unzip the downloaded file:
   unzip annotations_trainval2017.zip

3. LGG Segmentation Dataset: must be manually downloaded from https://www.kaggle.com/datasets/mateuszbuda/lgg-mri-segmentation, this requires a Kaggle login. The commands below are to extract the downloaded archive into the correct location: ~/.cache/mldata/lgg_segmentation/kaggle_3m.

   # use another location for MLDATA_DIR if desired, below is default;
   # Download and move the downloaded archive and unzip within the llgg_segmentation folder.
   mkdir -p ~/.cache/mldata/lgg_segmentation
   cd ~/.cache/mldata/lgg_segmentation
   # download and move the archive here then unzip
   unzip archive.zip
   # the dataset appears to have two copies that are equivalent, remove the extra one
   rm -r lgg-mri-segmentation

As part of the environment setup, you may need to add the root to PYTHONPATH:

export PYTHONPATH=.

benchmark.py allows easy way to benchmark performance of a support model, while varying configurations and compiler options. The script will measure real end-to-end time on host, starting post-compile at first input pushed, and ending at last output received.

To specify the device to run on ("tt", "cpu", or "cuda"), include the -d argument flag.

The script optionally outputs a .json file with benchmark results and options used to run the test. If the file already exists, the results will be appended, allowing the user to run multiple benchmark back-to-back, like:

python benchmark.py -d tt -m bert -c base --task text_classification --save_output

or

python benchmark.py -d cuda -m bert -c base --task text_classification --save_output

To see which models and configurations are currently supported, run:

benchmark.py --list

You can find example commands for various conditions in the file:

• run_benchmark_tt_perf for TT and run_benchmark_cuda GPU & CPU devices

We are excited to move our development to the public, open-source domain. However, we are not adequately staffed to review contributions in an expedient and manageable time frame at this time. In the meantime, please review the contributor's guide for more information about contribution standards.

If you would like to formally propose a new feature, report a bug, or have issues with permissions, please file through GitHub issues.

Please access the Discord community forum for updates, tips, live troubleshooting support, and more!