SambaNova AI Starter Kits are a collection of open-source examples and guides to facilitate the deployment of AI-driven use cases in the enterprise.

To run these examples, you need access to a SambaStudio environment with your models deployed to endpoints. Most code examples are written in Python, though the concepts can be applied in any language.

Questions? Just message us on Discord or create an issue in GitHub. We're happy to help live!

• 🔍 Data Extraction
• 📊 Web-crawled Data Retriever
• 🧠 Synthetic Q&A Pair Creation

• 🚀 YoDA - Your Data, Your Model Recipe
• 🎯 Q&A Fine-tuning
• 💾 SQL Model Fine-tuning
• 🔬 Fine-tuning Embeddings

• 📚 Enterprise Knowledge Retriever
• 🔎 Search Assistant
• 🖼️ Image Search
• 🖼️ Multi-Modal Knowledge Retriever

• 🧭 CoE Routing
• 💡 Prompt Engineering
• 🔀 Performance Benchmarking
• 🔧 Function Calling

• 🖥️ Endpoint Creation
• 🔗 Samba Apps Integration
• 🐳 Docker Images

Sambaverse allows you to interact with multiple open-source models. You can view the list of available models and interact with them in the playground.

Please note that Sambaverse's free offering is performance-limited. Companies that are ready to evaluate the production tokens-per-second performance, volume throughput, and 10x lower total cost of ownership (TCO) of SambaNova should contact us for a non-limited evaluation instance.

Begin by creating a Sambaverse account, then get your API key from the username button. Use the available models.

Begin by deploying your LLM of choice (e.g. Llama 2 13B chat, etc) to an endpoint for inference in SambaStudio. Use either the GUI or CLI, as described in the SambaStudio endpoint documentation.

Integrate your LLM deployed on SambaStudio with this AI starter kit in two simple steps:

  git clone https://github.com/sambanova/ai-starter-kit.git

These are represented as configurable variables in the environment variables file in sn-ai-starter-kit/.env.

For example, enter an endpoint with the URL "https://api-stage.sambanova.net/api/predict/nlp/12345678-9abc-def0-1234-56789abcdef0/456789ab-cdef-0123-4567-89abcdef0123" in the env file (with no spaces) as:

SAMBASTUDIO_BASE_URL="https://api-stage.sambanova.net"
SAMBASTUDIO_BASE_URI="api/predict/nlp"
SAMBASTUDIO_PROJECT_ID="12345678-9abc-def0-1234-56789abcdef0"
SAMBASTUDIO_ENDPOINT_ID="456789ab-cdef-0123-4567-89abcdef0123"
SAMBASTUDIO_API_KEY="89abcdef-0123-4567-89ab-cdef01234567"

Enter a Sambaverse API key, for example "456789ab-cdef-0123-4567-89abcdef0123", in the env file (with no spaces) as:

SAMBAVERSE_API_KEY="456789ab-cdef-0123-4567-89abcdef0123"

You can use SambaStudio E5 embedding model endpoint instead of using default in cpu HugginFace embeddings to increase inference speed, follow this guide to deploy your SambaStudio embedding model

be sure to set batch size model parameter to 32

Update API information for the SambaNova embedding endpoint. These are represented as configurable variables in the environment variables file in the root repo directory sn-ai-starter-kit/.env. For example, an endpoint with the URL "https://api-stage.sambanova.net/api/predict/nlp/12345678-9abc-def0-1234-56789abcdef0/456789ab-cdef-0123-4567-89abcdef0123" would be entered in the env file (with no spaces) as:

SAMBASTUDIO_EMBEDDINGS_BASE_URL="https://api-stage.sambanova.net"
SAMBASTUDIO_EMBEDDINGS_BASE_URI="api/predict/nlp"
SAMBASTUDIO_EMBEDDINGS_PROJECT_ID="12345678-9abc-def0-1234-56789abcdef0"
SAMBASTUDIO_EMBEDDINGS_ENDPOINT_ID="456789ab-cdef-0123-4567-89abcdef0123"
SAMBASTUDIO_EMBEDDINGS_API_KEY="89abcdef-0123-4567-89ab-cdef01234567"

Go to the README.md of the starter kit you want to use and follow the instructions. See Available AI Starter Kits.

Set your environment as shown in integrate your model.

1. Import the samabanova_endpoint langchain wrapper in your project and define your Sambaverse LLM:

from langchain_community.llms.sambanova import Sambaverse

load_dotenv('.env')

llm = Sambaverse(
    sambaverse_model_name="Meta/Meta-Llama-3-8B-Instruct",
    model_kwargs={
      "do_sample": False,
      "temperature": 0.0,
      "max_tokens_to_generate": 512,
      "select_expert": "Meta-Llama-3-8B-Instruct",
      "process_prompt": "False"
      },
)

2. Use the model

llm.invoke("your prompt")

1. Import the samabanova_endpoint langchain wrapper in your project and define your *SambaStudio LLM:

• If using a CoE endpont

from langchain_community.llms.sambanova import SambaStudio

load_dotenv('.env')

llm = SambaNovaEndpoint(
    model_kwargs={
      "do_sample": False,
      "max_tokens_to_generate": 512,
      "temperature": 0.0,
      "select_expert": "Meta-Llama-3-8B-Instruct",
      "process_prompt": "False"
      },
)

• If using a single model endpont

from langchain_community.llms.sambanova import SambaStudio

load_dotenv('.env')

llm = SambaNovaEndpoint(
    model_kwargs={
      "do_sample": False,
      "max_tokens_to_generate": 512,
      "temperature": 0.0,
      "process_prompt": "False"
      },
)

2. Use the model

llm.invoke("your prompt")

See utils/usage.ipynb for an example.

1. Import the sambastudio_endpoint langchain wrapper in your project and define your SambaStudioEmbeddings embedding:

• If using a CoE endpont

from langchain_community.embeddings import SambaStudioEmbeddings

load_dotenv('.env')

embedding = SambaStudioEmbeddings(
              batch_size=1,
              model_kwargs = {
                  "select_expert":e5-mistral-7b-instruct
                  }
              )

• If using a single embedding model endpont

from langchain_community.embeddings import SambaStudioEmbeddings

load_dotenv('.env')

embedding = SambaStudioEmbeddings(batch_size=32)

Note that using different embedding models (cpu or sambastudio) may change the results, and change the way they are set and their parameters

2. Use your embedding model in your langchain pipeline

See utils/usage.ipynb for an example.

There are two approaches to setting up your environment for the AI Starter Kits:

1. Individual Kit Setup (Traditional Method)
2. Base Environment Setup (WIP)

Each starter kit has its own README.md and requirements.txt file. You can set up a separate virtual environment for each kit by following the instructions in their respective directories. This method is suitable if you're only interested in running a single kit or prefer isolated environments for each project.

To use this method:

1. Navigate to the specific kit's directory
2. Create a virtual environment
3. Install the requirements
4. Follow the kit-specific instructions

For users who plan to work with multiple kits or prefer a unified development environment, we recommend setting up a base environment. This approach uses a Makefile to automate the setup of a consistent Python environment that works across all kits.

Benefits of the base environment approach:

• Consistent Python version across all kits
• Centralized dependency management
• Simplified setup process
• Easier switching between different kits

• pyenv: The Makefile will attempt to install pyenv if it's not already installed.
• Docker: (Optional) If you want to use the Docker-based setup, ensure Docker is installed on your system.

1. Installs pyenv and Poetry if they are not already installed.
2. Sets up a Python virtual environment using a specified Python version (default is 3.11.3).
3. Installs all necessary dependencies for the base environment.
4. Sets up the parsing service required by some kits.
5. Installs system dependencies like Tesseract OCR and Poppler.
6. Provides Docker-based setup options for consistent environments across different systems.

1. Install and Set Up the Base Environment:

make all

This command will set up the base ai-starter-kit environment, including installing all necessary tools and dependencies.

2. Activate the Base Environment:

source .venv/bin/activate

3. Navigate to Your Chosen Starter Kit:

cd path/to/starter_kit

Within the starter kit there will be instructions on how to start the kit. You can skip the virtual environment creation part in the kits README.md as we've done it here.

For certain kits, we utilise a standard parsing service. By Default it's started automatically with the base environment. To work with this service in isolation, following the steps in this section.

• Start Parsing Service:

make start-parsing-service

• Stop Parsing Service:

make stop-parsing-service

• Check Parsing Service Status:

make parsing-status

• View Parsing Service Logs:

make parsing-log

To use the Docker-based setup:

1. Ensure Docker is installed on your system.
2. Build the Docker image:

make docker-build

3. Run a specific kit in the Docker container:

make docker-run-kit KIT=<kit_name>

Replace <kit_name> with the name of the starter kit you want to run (e.g., function_calling).

4. To open a shell in the Docker container:

make docker-shell

To clean up all virtual environments created by the makefile and stop parsing services run the following command:

make clean

This command removes all virtual environments created with the makefile, stops the parsing service, and cleans up any temporary files.

If you encounter issues while setting up or running the AI Starter Kit, here are some common problems and their solutions:

If you're having problems with Python versions:

1. Ensure you have pyenv installed: make ensure-pyenv
2. Install the required Python versions: make install-python-versions
3. If issues persist, check your system's Python installation and PATH settings.

If you're experiencing dependency conflicts:

1. Try cleaning your environment: make clean
2. Update the lock file: poetry lock --no-update
3. Reinstall dependencies: make install

If you encounter an error while installing pikepdf, such as:

ERROR: Failed building wheel for pikepdf
Failed to build pikepdf

This is likely due to missing qpdf dependency. The Makefile should automatically install qpdf for you, but if you're still encountering issues:

1. Ensure you have proper permissions to install system packages.
2. If you're on macOS, you can manually install qpdf using Homebrew:

   brew install qpdf

3. On Linux, you can install it using your package manager, e.g., on Ubuntu:

   sudo apt-get update && sudo apt-get install -y qpdf
   

4. After installing qpdf, try running make install again.

If you continue to face issues, please ensure your system meets all the requirements for building pikepdf and consider checking the pikepdf documentation for more detailed installation instructions.

If the parsing service isn't starting or is behaving unexpectedly:

1. Check its status: make parsing-status
2. View its logs: make parsing-log
3. Try stopping and restarting it: make stop-parsing-service followed by make start-parsing-service

If you encounter issues related to Tesseract OCR or Poppler:

1. Ensure the Makefile has successfully installed these dependencies.
2. On macOS, you can manually install them using Homebrew:

  brew install tesseract poppler

3. On Linux (Ubuntu/Debian), you can install them manually:

  sudo apt-get update && sudo apt-get install -y tesseract-ocr poppler-utils

4. On Windows, you may need to install these dependencies manually and ensure they are in your system PATH.

If you're using the Docker-based setup and encounter issues:

1. Ensure Docker is properly installed and running on your system.
2. Try rebuilding the Docker image: make docker-build
3. Check Docker logs for any error messages.
4. Ensure your firewall or antivirus is not blocking Docker operations.

1. Ensure all prerequisites (Python, pyenv, Poetry) are correctly installed.
2. Try cleaning and rebuilding the environment: make clean all
3. Check for any error messages in the console output and address them specifically.
4. Ensure your .env file is correctly set up in the ai-starter-kit root with all necessary environment variables.

If you continue to experience issues, please open an issue with details about your environment, the full error message, and steps to reproduce the problem.

• Ensure you have sufficient permissions to install software on your system.
• The setup process may take several minutes, especially when installing Python versions or large dependencies.
• If you encounter any issues during setup, check the error messages and ensure your system meets all prerequisites.
• Always activate the base environment before navigating to and running a specific starter kit.
• Some kits may require additional setup steps. Always refer to the specific README of the kit you're using.

Note: These AI Starter Kit code samples are provided "as-is," and are not production-ready or supported code. Bugfix/support will be on a best-effort basis only. Code may use third-party open-source software. You are responsible for performing due diligence per your organization policies for use in your applications.