• Table of Contents
• About
• Roadmap and New Features List
• Try our public demo!
• Installing VoucherVision
  • Prerequisites
  • Installation - Cloning the VoucherVision Repository
  • About Python Virtual Environments
  • Installation - Windows 10+ (using pip)
  • Installation - Windows 10+ (using conda)
    • Virtual Environment
    • Installing Packages
  • Troubleshooting CUDA
• Create a Desktop Shortcut to Launch VoucherVision GUI (MacOS)
• Create a Desktop Shortcut to Launch VoucherVision GUI (Windows)
• Run VoucherVision
  • Setting up API key
  • Check GPU
  • Run Tests
  • Starting VoucherVision
  • Azure Instances of OpenAI
• Custom Prompt Builder
  • Load, Build, Edit
  • Instructions
  • Defining Column Names Field-Specific Instructions
  • Prompting Structure
  • Mapping Columns for VoucherVisionEditor
• Expense Reporting
  • Expense Report Dashboard
• User Interface Images

For inquiries, feedback (or if you want to get involved!) please complete our form.

Initiated by the University of Michigan Herbarium, VoucherVision harnesses the power of large language models (LLMs) to transform the transcription process of natural history specimen labels. Our workflow is as follows:

• Text extraction from specimen labels with LeafMachine2.
• Text interpretation using Google Vision OCR.
• LLMs, including GPT-3.5, GPT-4, PaLM 2, and Azure instances of OpenAI models, standardize the OCR output into a consistent spreadsheet format. This data can then be integrated into various databases like Specify, Symbiota, and BRAHMS.

For ensuring accuracy and consistency, the VoucherVisionEditor serves as a quality control tool.

• Update all GPT models
• Option to zip output files for simpler import into VVE
• Expense tracking
  • Dashboard
  • More granular support for different GPT versions
  • Project-based and cummulative tracking
• Hugging Face Spaces
  • Working and refactored
  • Unified with local version
  • Visualize locations on a map (verbatim and decimal)
  • Tested with batch of 300 images
  • Optimize for +300 images at a time (not recommended unless paying for HF persistent storage)
• Modular Prompt Builder
  • Build, save, load, submit to VV library
  • Test whether order of column matters
  • Provide 3 standard prompts: long, medium, short
  • Test shorter prompt effectiveness
• API Availability Test/Visualization
  • Lock-out unsupported methods
• Option to load existing OCR into VoucherVision workflow
• Save/load VV settings
• LLM Cost Calculator
• FAQs page
• Detect and display system hardware
• Filename Prefix Handling (works, but needs more options and a GUI test)
• Validation Tools
  • JSON formatting
  • World Flora Online
  • Wikipedia links
  • USDA GRIN links
  • Plants of the World Online links

• OpenAI
  • GPT 4
  • GPT 4 Turbo 1106-preview
  • GPT 4 Turbo 0125-preview
  • GPT 4 32k
  • GPT 3.5
  • GPT 3.5 Instruct
• OpenAI (Microsoft Azure Endpoints)
  • GPT 4
  • GPT 4 Turbo 1106-preview
  • GPT 4 Turbo 0125-preview
  • GPT 4 32k
  • GPT 3.5
  • GPT 3.5 Instruct
• MistralAI
  • Mistral Tiny
  • Mistral Small
  • Mistral Medium
• Google PaLM2
  • text-bison@001
  • text-bison@002
  • text-unicorn@001
• Google Gemini
  • Gemini-Pro
• LOCAL LLMs
  • GPU -- Mixtral 8x7B Instruct v0.1
  • GPU -- Mistral 7B Instruct v0.2
  • CPU -- Mistral 7B Instruct v0.2 GGUF

• MistralAI (24GB+ VRAM GPU Required)
  • Mixtral 8x7B Instruct v0.1
  • Mixtral 7B Instruct v0.2
• MistralAI (CPU Inference) ((can run on almost computer!))
  • Mixtral 7B Instruct v0.2 GGUF via llama.cpp
• Meta-Llama2 7B
  • Llama2 7B chat hf

• Streamline the startup procedure
• Add configurable dropdown menus for certain fields
• Make sure that VVE can accomodate arbitrary column names
• Remove legacy support (version 1 prompts)
• Taxonomy validation helper
• Visualize locations on a map (verbatim and decimal)
• More support for datum and verbatim coordinates
• Compare raw OCR to values in form to flag hallucinations/generated content
• Accept zipped folders as input
• Flag user when multiple people/names/determinations are present

The main VoucherVision tool and the VoucherVisionEditor are packaged separately. This separation ensures that lower-performance computers can still install and utilize the editor. While VoucherVision is optimized to function smoothly on virtually any modern system, maximizing its capabilities (like using LeafMachine2 label collages or running Retrieval Augmented Generation (RAG) prompts) mandates a GPU.

NOTE: You can absolutely run VoucherVision on computers that do not have a GPU, but the LeafMachine2 collage will run slower.

Our public demo, while lacking several quality control and reliability features found in the full VoucherVision module, provides an exciting glimpse into its capabilities. Feel free to upload your herbarium specimen and see what happens! VoucherVision Demo

• Python 3.10.4 or later
• Optional: an Nvidia GPU + CUDA for running LeafMachine2

1. First, install Python 3.10, or greater, on your machine of choice. We have validated up to Python 3.11.
   • Make sure that you can use pip to install packages on your machine, or at least inside of a virtual environment.
   • Simply type pip into your terminal or PowerShell. If you see a list of options, you are all set. Otherwise, see either this PIP Documentation or this help page
2. Open a terminal window and cd into the directory where you want to install VoucherVision.
3. In the Git BASH terminal, clone the VoucherVision repository from GitHub by running the command:

   git clone https://github.com/Gene-Weaver/VoucherVision.git

4. Move into the VoucherVision directory by running cd VoucherVision in the terminal.
5. To run VoucherVision we need to install its dependencies inside of a python virtual environmnet. Follow the instructions below for your operating system.

A virtual environment is a tool to keep the dependencies required by different projects in separate places, by creating isolated python virtual environments for them. This avoids any conflicts between the packages that you have installed for different projects. It makes it easier to maintain different versions of packages for different projects.

For more information about virtual environments, please see Creation of virtual environments

Installation should basically be the same for Linux.

1. Still inside the VoucherVision directory, show that a venv is currently not active

   python --version

2. Then create the virtual environment (venv_VV is the name of our new virtual environment)

   python3 -m venv venv_VV

   Or depending on your Python version...

   python -m venv venv_VV

3. Activate the virtual environment

   .\venv_VV\Scripts\activate

4. Confirm that the venv is active (should be different from step 1)

   python --version

5. If you want to exit the venv later for some reason, deactivate the venv using

   deactivate

1. Install the required dependencies to use VoucherVision
   cd into VoucherVision

pip install -r requirements.txt

If you do NOT have a GPU, then you are all set. Otherwise...

2. Make sure that your GPU can be recognized. While in the terminal/powershell, type

   python

   This opens a Python script. Import torch

   import torch

   Make sure the GPU is found

   torch.cuda.is_available()

   Exit the Python instance

   exit()

3. If torch.cuda.is_available() returned True, then you should be set. Otherwise, you need to make sure that your CUDA version is compatible with the PyTorch version. It's usually a good idea to leave the CUDA drivers alone and find the right PyTorch version since installing/updating CUDA can be non-trivial.

   • Example: If torch.cuda.is_available() returned False, I would first check my CUDA version. In a terminal, type

   nvidia-smi

   • If this throws an error, then you do not have CUDA installed. Please see the troubleshooting steps below.

   • Otherwise, look for CUDA Version: XX.X. In this example, we saw CUDA Version: 12.1

   • Go to https://pytorch.org/get-started/previous-versions/, search for 12.1 (or your CUDA version) and find the conda installation version. There are MacOS options too.

   • We need a PyTorch version greater than 2.X.X. If none exists, then your CUDA version may be too old.

   • When I searched for 12.1, I found this: pip install torch==2.1.2 torchvision==0.16.2 torchaudio==2.1.2 --index-url https://download.pytorch.org/whl/cu121

   • Install your matching version

   • Cheat sheet:

     • CUDA 11.8

     pip install torch==2.1.2 torchvision==0.16.2 torchaudio==2.1.2 --index-url https://download.pytorch.org/whl/cu118

     • CUDA 12.1

     pip install torch==2.1.2 torchvision==0.16.2 torchaudio==2.1.2 --index-url https://download.pytorch.org/whl/cu121

   • Verify the installation

   • Now we should have the right PyTorch version. Check to see if torch.cuda.is_available() returns True by following the same procedure as above

If your CUDA version < 11.8, then you should probably upgrade to 12.1 If you need help, please submit an inquiry in the form at LeafMachine.org

1. First, install Anaconda using default settings
   • Anaconda
2. Open the Anaconda Powershell Prompt (Windows) or the terminal (macOS/Linux)
3. Install Mamba in the base environment. We will use Mamba because it it much faster!

   conda install mamba -n base -c conda-forge

4. Make sure Conda and Mamba are up to date

   conda update conda

   conda update mamba -c conda-forge

   mamba update --all

   mamba clean --all

5. Create a new Conda environment using Mamba

   mamba create --name vouchervision python=3.10 git -c conda-forge

6. Activate the Conda environment

   conda activate vouchervision

7. Use cd to move to the directory where you want VoucherVision to live
8. Clone the VoucherVision repository

   git clone https://github.com/Gene-Weaver/VoucherVision.git

   Move into the VoucherVision home directory

   cd VoucherVision

10. Then we need to install some packages

    pip install -r requirements.txt

If you do NOT have a GPU, then you are all set. Otherwise...

11. Make sure that your GPU can be recognized. While in the terminal/powershell, type

    python

    This opens a Python script. Import torch

    import torch

    Make sure the GPU is found

    torch.cuda.is_available()

    Exit the Python instance

    exit()

12. If torch.cuda.is_available() returned True, then you should be set. Otherwise, you need to make sure that your CUDA version is compatible with the PyTorch version. It's usually a good idea to leave the CUDA drivers alone and find the right PyTorch version since installing/updating CUDA can be non-trivial.

    • Example: If torch.cuda.is_available() returned False, I would first check my CUDA version. In a terminal, type

    nvidia-smi

    • If this throws an error, then you do not have CUDA installed. Please see the troubleshooting steps below.

    • Otherwise, look for CUDA Version: XX.X. In this example, we saw CUDA Version: 11.7

    • Go to https://pytorch.org/get-started/previous-versions/, search for 11.7 (or your CUDA version) and find the conda installation version. There are MacOS options too.

    • We need a PyTorch version greater than 2.X.X. If none exists, then your CUDA version may be too old.

    • When I searched for 11.7, I found this: mamba install pytorch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 pytorch-cuda=11.7 -c pytorch -c nvidia

    • Install your matching version (conda install can be very slow)

    • Cheat sheet:

      • CUDA 11.7

      mamba install pytorch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 pytorch-cuda=11.7 -c pytorch -c nvidia

      • CUDA 11.8

      mamba install pytorch==2.1.2 torchvision==0.16.2 torchaudio==2.1.2 pytorch-cuda=11.8 -c pytorch -c nvidia

      • CUDA 12.1

      mamba install pytorch==2.1.2 torchvision==0.16.2 torchaudio==2.1.2 pytorch-cuda=12.1 -c pytorch -c nvidia

    • Verify the installation

    mamba list cudnn

    • Now we should have the right PyTorch version. Check to see if torch.cuda.is_available() returns True by following the same procedure as above
    • Check torch.cuda.is_available() one more time just to be sure.

If your CUDA version < 11.8, then you should probably upgrade to 12.1 If you need help, please submit an inquiry in the form at LeafMachine.org

We can create a desktop shortcut to launch VoucherVision. In the ../VoucherVision/ directory is a file called create_desktop_shortcut.py. In the terminal, move into the ../VoucherVision/ directory and type:

python create_desktop_shortcut.py

Or...

python3 create_desktop_shortcut.py

Follow the instructions, select where you want the shortcut to be created, then where the virtual environment is located.

Note If you ever see an error that says that a "port is not available", open run.py in a plain text editor and change the --port value to something different but close, like 8502. Sometimes the connection may not close properly. Also make sure that the previous terminal is closed before re-launching.

We can create a desktop shortcut to launch VoucherVision. In the ../VoucherVision/ directory is a file called create_desktop_shortcut_mac.py. In the terminal, cd into the ../VoucherVision/ directory and type:

python create_desktop_shortcut_mac.py

Or...

python3 create_desktop_shortcut_mac.py

Now go look in the ../VoucherVision/ directory. You will see a new file called VoucherVision.app. Drag this file into the Applications folder so that you can open VoucherVisionEditor just like any other app.

Note If you ever see an error that says that a "port is not available", open run.py in a plain text editor and change the --port value to something different but close, like 8502. Sometimes the connection may not close properly. Also make sure that the previous terminal is closed before re-launching.

NOTE: The instructions below have not been updated to reflect the new code as of Feb. 14, 2024. Stay tuned for updated instructions

1. In the terminal, make sure that you cd into the VoucherVision directory and that your virtual environment is active (you should see venv_VV on the command line).
2. Type:

   python run_VoucherVision.py

   or depending on your Python installation:

   python3 run_VoucherVision.py

3. If you ever see an error that says that a "port is not available", open run_VoucherVision.py in a plain text editor and change the --port value to something different but close, like 8502.

VoucherVision requires access to Google Vision OCR and at least one of the following LLMs: OpenAI API, Google PaLM 2, a private instance of OpenAI through Microsoft Azure. On first startup, you will see a page with instructions on how to get these API keys. Nothing will work until you get at least the Google Vision OCR API key and one LLM API key.

Press the "Check GPU" button to see if you have a GPU available. If you know that your computer has an Nvidia GPU, but the check fails, then you need to install an different version of PyTorch in the virtual environment.

Once you have provided API keys, you can test all available prompts and LLMs by pressing the test buttons. Every combination of LLM, prompt, and LeafMachine2 collage will run on the image in the ../VoucherVision/demo/demo_images folder. A grid will appear letting you know which combinations are working on your system.

1. "Run name" - Set a run name for your project. This will be the name of the new folder that contains the output files.

2. "Output directory" - Paste the full file path of where you would like to save the folder that will be created in step 1.

3. "Input images directory" - Paste the full file path of where the input images are located. This folder can only have JPG or JPEG images inside of it.

4. "Select an LLM" - Pick the LLM you want to use to parse the unstructured OCR text.

   • As of Nov. 1, 2023 PaLM 2 is free to use.

5. "Prompt Version" - Pick your prompt version. We recommend "Version 2" for production use, but you can experiment with our other prompts.

6. "Cropped Components" - Check the box to use LeafMachine2 collage images as the input file. LeafMachine2 can often find small handwritten text that may be missed by Google Vision OCR's text detection algorithm. But, the difference in performance is not that big. You will still get good performance without using the LeafMachine2 collage images.

7. "Domain Knowledge" is only used for "Version 1" prompts.

8. "Component Detector" sets basic LeafMachine2 parameters, but the default is likely good enough.

9. "Processing Options"

   • The image file name defines the row name in the final output spreadsheet.
   • We provide some basic options to clean/parse the image file name to produce the desired output.
   • For example, if the input image name is MICH-V-3819482.jpg but the desired name is just 3819482 you can add MICH-V- to the "Remove prefix from catalog number" input box. Alternatively, you can check the "Require Catalog..." box and achieve the same result.

10. Finally you can press the start processing button.

If your institution has an enterprise instance of OpenAI's services, like at the University of Michigan, you can use Azure instead of the OpenAI servers. Your institution should be able to provide you with the required keys (there are 5 required keys for this service).

VoucherVision empowers individual institutions to customize the format of the LLM output. Using our pre-defined prompts you can transcribe the label text into 20 columns, but using our Prompt Builder you can load one of our default prompts and adjust the output to meet your needs. More instructions will come soon, but for now here are a few more details.

The Prompt Builder creates a prompt in the structure that VoucherVision expects. This information is stored as a configuration yaml file in ../VoucherVision/custom_prompts/. We provide a few versions to get started. You can load one of our examples and then use the Prompt Builder to edit or add new columns.

Right now, the prompting instructions are not configurable, but that may change in the future.

The central JSON object shows the structure of the columns that you are requesting the LLM to create and populate with information from the specimen's labels. These will become the rows in the final xlsx file the VoucherVision generates. You can pick formatting instructions, set default values, and give detailed instructions.

Note: formatting instructions are not always followed precisely by the LLM. For example, GPT-4 is capable of granular instructions like converting ALL CAPS TEXT to sentence-case, but GPT-3.5 and PaLM 2 might not be capable of following that instruction every time (which is why we have the VoucherVisionEditor and are working to link these instructions so that humans editing the output can quickly/easily fix these errors).

The rightmost JSON object is the entire prompt structure. If you load the required_structure.yaml prompt, you will wee the bare-bones version of what VoucherVision expects to see. All of the parts are there for a reason. The Prompt Builder UI may be a little unruly right now thanks to quirks with Streamlit, but we still recommend using the UI to build your own prompts to make sure that all of the required components are present.

Finally, we need to map columns to a VoucherVisionEditor category.

VoucherVision logs the number of input and output tokens (using tiktoken) from every call. We store the publicly listed prices of the LLM APIs in ../VoucherVision/api_cost/api_cost.yaml. Then we do some simple math to estimage the cost of run, which is stored inside of your project's output directory ../run_name/Cost/run_name.csv and all runs are accumulated in a csv file stored in ../VoucherVision/expense_report/expense_report.csv. VoucherVision only manages expense_report.csv, so if you want to split costs by month/quarter then copy and rename expense_report.csv. Deleting expense_report.csv will let you accumulate more stats.

This should be treated as an estimate. The true cost may be slightly different.

This is an example of the stats that we track:

The sidebar in VoucherVision displays summary stats taken from expense_report.csv.

Validation test when the OpenAI key is not provided, but keys for PaLM 2 and Azure OpenAI are present:

Validation test when all versions of the OpenAI keys are provided:

A successful GPU test:

Successful PaLM 2 test: