Skip to content

utensil/llm-playground

Repository files navigation

LLM Playground

This is a new era, with so many low-hanging fruits on the new magical tree, and so many new tools to reach them emerging every day. Things are changing so fast to play catch-up. I need a place to fix the settings and results of my experiments, this is it.

Focus Matrix

I need to focus since I don't have unlimited time and resources.

It's a focus tensor actually: tools, models, datasets, data sources, hardware, metrics, hows.

  • Tools:
    • For inference and training, use text-generation-webui if applicable, and others otherwise.
      • Focus on 4bit LoRA training with the biggest possible foundation models
      • Also try some fully-fledged training on toy models and domain datasets
    • For creating, retrieving, and preprocessing datasets
  • Models:
    • Cover the mainstream ones with a focus on the ones that don't have legal issues for commercial use.
    • Try to cover more types of LLM models to evaluate the differences.
  • Datasets:
    • Focus on the high-quality ones that enhance coding, math, reasoning, and instruction-following capacities and have no legal issues for commercial use.
    • Natural languages include English, Chinese, and other languages I like.
    • Programming languages include Python, C++, JavaScript, Rust, Lean 4, Julia, etc.
  • Data sources:
    • The data source types that can be loaded with LlamaIndex/LangChain and injected into prompts.
    • Other types I need.
  • Hardware:
    • Edge: ~8 GB RAM
    • CPU-only: ~32 GB RAM
    • Consumer-grade GPU: RTX 3090Ti/A6000 ~48G VRAM
    • Training-grade GPU: ~A100 80G VRAM
    • Mostly use cloud services to switch between hardware easily
  • Metrics:
    • Light-weight metrics to evaluate datasets for quality and diversity
    • A small set of instructions and interactions for quick sanity-check on models and easy to run like unit tests
    • Light-weight metrics to evaluate models for general and domain-specific tasks
  • Hows:
    • Experiments and visualizations to better understand why and how LLMs work

The settings

  • Base on a bare-bone docker image like nvidia/cuda:11.8.0-devel-ubuntu22.04 so I can switch between different cloud services easily
  • These files are at the root or they could be folder-specific
    • .env for environment variables
    • packages.txt for apt packages
    • requirements.txt for Python packages
    • datasets.txt for datasets
    • models.txt for models
  • The Jupyter notebooks should
    • Clone this repo and use helper scripts to fix other dependencies like text-generation-webui, GPTQ-for-LLaMa, alpaca_lora_4bit, etc.
    • Use helper scripts to download datasets and models
    • Upload results and models to HuggingFace for persistence
    • Rely on credentials for Github, HuggingFace, etc.