The only AI assistant that investigates incidents like a human does - by looking at alerts and fetching missing data until it finds the root cause. Powered by OpenAI or any tool-calling LLM of your choice, including open source models.
- Investigate Incidents (AIOps) from PagerDuty/OpsGenie/Prometheus/Jira/more
- Bidirectional Integrations see investigation results inside your existing ticketing/incident management system
- Automated Triage: Use HolmesGPT as a first responder. Flag critical alerts and prioritize them for your team to look at
- Alert Enrichment: Automatically add context to alerts - like logs and microservice health info - to find root causes faster
- Identify Cloud Problems by asking HolmesGPT questions about unhealthy infrastructure
- Runbook Automation in Plain English: Speed up your response to known issues by investigating according to runbooks you provide
Kubernetes Troubleshooting
holmes ask "what pods are unhealthy in my cluster and why?"Prometheus Alert RCA (root cause analysis)
Investigate Prometheus alerts right from Slack with the official Robusta integration.
Or run HolmesGPT from the cli:
kubectl port-forward alertmanager-robusta-kube-prometheus-st-alertmanager-0 9093:9093 &
holmes investigate alertmanager --alertmanager-url https://localhost:9093Note - if on Mac OS and using the Docker image, you will need to use https://docker.for.mac.localhost:9093 instead of https://localhost:9093
Log File Analysis
Attach files to the HolmesGPT session with -f:
sudo dmesg > dmesg.log
poetry run python3 holmes.py ask "investigate errors in this dmesg log" -f dmesg.logJira Ticket Investigation
holmes investigate jira --jira-url https://<PLACEDHOLDER>.atlassian.net --jira-username <PLACEHOLDER_EMAIL> --jira-api-key <PLACEHOLDER_API_KEY>By default results are displayed in the CLI . Use --update to get the results as a comment in the Jira ticket.
GitHub Issue Investigation
holmes investigate github --github-url https://<PLACEHOLDER> --github-owner <PLACEHOLDER_OWNER_NAME> --github-repository <PLACEHOLDER_GITHUB_REPOSITORY> --github-pat <PLACEHOLDER_GITHUB_PAT>By default results are displayed in the CLI. Use --update to get the results as a comment in the GitHub issue.
OpsGenie Alert Investigation
holmes investigate opsgenie --opsgenie-api-key <PLACEHOLDER_APIKEY>By default results are displayed in the CLI . Use --update --opsgenie-team-integration-key <PLACEHOLDER_TEAM_KEY> to get the results as a comment in the OpsGenie alerts. Refer to the CLI help for more info.
PagerDuty Incident Investigation
holmes investigate pagerduty --pagerduty-api-key <PLACEHOLDER_APIKEY>By default results are displayed in the CLI. Use --update --pagerduty-user-email <PLACEHOLDER_EMAIL> to get the results as a comment in the PagerDuty issue. Refer to the CLI help for more info.
Like what you see? Checkout other use cases or get started by installing HolmesGPT.
- Connects to Existing Observability Data: Find correlations you didn’t know about. No need to gather new data or add instrumentation.
- Compliance Friendly: Can be run on-premise with your own LLM (or in the cloud with OpenAI or Azure)
- Transparent Results: See a log of the AI’s actions and what data it gathered to understand how it reached conclusions
- Extensible Data Sources: Connect the AI to custom data by providing your own tool definitions
- Runbook Automation: Optionally provide runbooks in plain English and the AI will follow them automatically
- Integrates with Existing Workflows: Connect Slack and Jira to get results inside your existing tools
First you will need an OpenAI API key, or the equivalent for another model. Then install with one of the below methods:
Brew (Mac/Linux)
- Add our tap:
brew tap robusta-dev/homebrew-holmesgpt- Install holmesgpt:
brew install holmesgpt- Check that installation was successful. This will take a few seconds on the first run - wait patiently.:
holmes --help- Run holmesgpt:
holmes ask "what issues do I have in my cluster"Prebuilt Docker Container
Run the prebuilt Docker container docker.pkg.dev/genuine-flight-317411/devel/holmes, with extra flags to mount relevant config files (so that kubectl and other tools can access AWS/GCP resources using your local machine's credentials)
docker run -it --net=host -v ~/.holmes:/root/.holmes -v ~/.aws:/root/.aws -v ~/.config/gcloud:/root/.config/gcloud -v $HOME/.kube/config:/root/.kube/config us-central1-docker.pkg.dev/genuine-flight-317411/devel/holmes ask "what pods are unhealthy and why?"From Source (Python Poetry)
First install poetry (the python package manager)
git clone https://github.com/robusta-dev/holmesgpt.git
cd holmesgpt
poetry install --no-root
poetry run python3 holmes.py ask "what pods are unhealthy and why?"
From Source (Docker)
Clone the project from github, and then run:
cd holmesgpt
docker build -t holmes .
docker run -it --net=host -v -v ~/.holmes:/root/.holmes -v ~/.aws:/root/.aws -v ~/.config/gcloud:/root/.config/gcloud -v $HOME/.kube/config:/root/.kube/config holmes ask "what pods are unhealthy and why?"HolmesGPT requires an API Key to function. Follow one of the instructions below.
OpenAI
To work with OpenAI’s GPT 3.5 or GPT-4 models you need a paid OpenAI API key.
Note: This is different from being a “ChatGPT Plus” subscriber.
Pass your API key to holmes with the --api-key cli argument:
holmes ask --api-key="..." "what pods are crashing in my cluster and why?"
If you prefer not to pass secrets on the cli, set the OPENAI_API_KEY environment variable or save the API key in a HolmesGPT config file.
Azure OpenAI
To work with Azure AI, you need the Azure OpenAI.
holmes ask "what pods are unhealthy and why?" --llm=azure --api-key=<PLACEHOLDER> --azure-endpoint='<PLACEHOLDER>'The --azure-endpoint should be a URL in the format "https://some-azure-org.openai.azure.com/openai/deployments/gpt4-1106/chat/completions?api-version=2023-07-01-preview"
If you prefer not to pass secrets on the cli, set the AZURE_OPENAI_API_KEY environment variable or save the API key in a HolmesGPT config file.
Using a self-hosted LLM
You will need an LLM with support for function-calling (tool-calling). To use it, set the OPENAI_BASE_URL environment variable and run holmes with a relevant model name set using --model.
Important: Please verify that your model and inference server support function calling! HolmesGPT is currently unable to check if the LLM it was given supports function-calling or not. Some models that lack function-calling capabilities will hallucinate answers instead of reporting that they are unable to call functions. This behaviour depends on the model.
In particular, note that vLLM does not yet support function calling, whereas llama-cpp does support it.
HolmesGPT is usually used for incident response, but it can function as a general-purpose DevOps assistant too. Here are some examples:
Ask Questions About Your Cloud
holmes ask "what services does my cluster expose externally?"Ticket Management - Automatically Respond to Jira tickets related to DevOps tasks
holmes investigate jira --jira-url https://<PLACEDHOLDER>.atlassian.net --jira-username <PLACEHOLDER_EMAIL> --jira-api-key <PLACEHOLDER_API_KEY>Find the right configuration to change in big Helm charts
LLM uses the built-in Helm toolset to gather information.
holmes ask "what helm value should I change to increase memory request of the my-argo-cd-argocd-server-6864949974-lzp6m pod"Optimize Docker container size
LLM uses the built-in Docker toolset to gather information.
holmes ask "Tell me what layers of my pavangudiwada/robusta-ai docker image consume the most storage and suggest some fixes to it"HolmesGPT can investigate many issues out of the box, with no customization or training.
That said, we provide several extension points for teaching HolmesGPT to investigate your issues, according to your best practices. The two main extension points are:
- Custom Tools - give HolmesGPT access to data that it can't otherwise access - e.g. traces, APM data, or custom APIs
- Custom Runbooks - give HolmesGPT instructions for investigating specific issues it otherwise wouldn't know how to handle
Add Custom Tools
The more data you give HolmesGPT, the better it will perform. Give it access to more data by adding custom tools.
New tools are loaded using -t from custom toolset files or by adding them to the ~/.holmes/config.yaml with the setting custom_toolsets: ["/path/to/toolset.yaml"].
Add Custom Runbooks
HolmesGPT can investigate by following runbooks written in plain English. Add your own runbooks to provided the LLM specific instructions.
New runbooks are loaded using -r from custom runbook files or by adding them to the ~/.holmes/config.yaml with the custom_runbooks: ["/path/to/runbook.yaml"].
Reading settings from a config file
You can customize HolmesGPT's behaviour with command line flags, or you can save common settings in config file for re-use.
You can view an example config file with all available settings here.
By default, without specifying --config the agent will try to read ~/.holmes/config.yaml. When settings are present in both config file and cli, the cli option takes precedence.
Custom Toolsets
You can define your own custom toolsets to extend the functionality of your setup. These toolsets can include querying company-specific data, fetching logs from observability tools, and more.
# Add paths to your custom toolsets here
# Example: ["path/to/your/custom_toolset.yaml"]
#custom_toolsets: ["examples/custom_toolset.yaml"]Alertmanager Configuration
Configure the URL for your Alertmanager instance to enable alert management and notifications.
# URL for the Alertmanager
#alertmanager_url: "https://localhost:9093"Jira Integration
Integrate with Jira to automate issue tracking and project management tasks. Provide your Jira credentials and specify the query to fetch issues and optionally update their status.
# Jira credentials and query settings
#jira_username: "[email protected]"
#jira_api_key: "..."
#jira_url: "https://your-company.atlassian.net"
#jira_query: "project = 'Natan Test Project' and Status = 'To Do'"- jira_username: The email you use to log into your Jira account. Eg:
[email protected] - jira_api_key: Follow these instructions to get your API key.
- jira_url: The URL of your workspace. For example: https://workspace.atlassian.net (Note: schema (https) is required)
- project: Name of the project you want the Jira tickets to be created in. Go to Project Settings -> Details -> Name.
- status: Status of a ticket. Example:
To Do,In Progress
GitHub Integration
Integrate with GitHub to automate issue tracking and project management tasks. Provide your GitHub PAT (personal access token) and specify the owner/repository.
# GitHub credentials and query settings
#github_owner: "robusta-dev"
#github_pat: "..."
#github_url: "https://api.github.com" (default)
#github_repository: "holmesgpt"
#github_query: "is:issue is:open"- github_owner: The repository owner. Eg:
robusta-dev - github_pat: Follow these instructions to get your GitHub pat (personal access token).
- github_url: The URL of your GitHub API. For example: https://api.github.com (Note: schema (https) is required)
- github_repository: Name of the repository you want the GitHub issues to be scanned. Eg:
holmesgpt.
PagerDuty Integration
Integrate with PagerDuty to automate incident tracking and project management tasks. Provide your PagerDuty credentials and specify the user email to update the incident with findings.
pagerduty_api_key: "..."
pagerduty_user_email: "[email protected]"
pagerduty_incident_key: "..."- pagerduty_api_key: The PagerDuty API key. This can be found in the PagerDuty UI under Integrations > API Access Key.
- pagerduty_user_email: When --update is set, which user will be listed as the user who updated the incident. (Must be the email of a valid user in your PagerDuty account.)
- pagerduty_incident_key: If provided, only analyze a single PagerDuty incident matching this key
OpsGenie Integration
Integrate with OpsGenie to automate alert investigations. Provide your OpsGenie credentials and specify the query to fetch alerts.
opsgenie_api_key : "..."
opsgenie-team-integration-key: "...."
opsgenie-query: "..."- opsgenie_api_key: The OpsGenie API key. Get it from Settings > API key management > Add new API key
- opsgenie-team-integration-key: OpsGenie Team Integration key for writing back results. (NOT a normal API Key.) Get it from Teams > YourTeamName > Integrations > Add Integration > API Key. Don't forget to turn on the integration and add the Team as Responders to the alert.
- opsgenie-query: E.g. 'message: Foo' (see https://support.atlassian.com/opsgenie/docs/search-queries-for-alerts/)
Slack Integration
Configure Slack to send notifications to specific channels. Provide your Slack token and the desired channel for notifications.
# Slack token and channel configuration
#slack_token: "..."
#slack_channel: "#general"- slack-token: The Slack API key. You can generate with
pip install robusta-cli && robusta integrations slack - slack-channel: The Slack channel where you want to receive the findings.
Custom Runbooks
Define custom runbooks to give explicit instructions to the LLM on how to investigate certain alerts. This can help in achieving better results for known alerts.
# Add paths to your custom runbooks here
# Example: ["path/to/your/custom_runbook.yaml"]
#custom_runbooks: ["examples/custom_runbooks.yaml"]Choose between OpenAI or Azure for integrating large language models. Provide the necessary API keys and endpoints for the selected service.
OpenAI
# Configuration for OpenAI LLM
#llm: "openai"
#api_key: "your-secret-api-key"Azure
# Configuration for Azure LLM
#llm: "azure"
#api_key: "your-secret-api-key"
#azure_endpoint: "https://some-azure-org.openai.azure.com/openai/deployments/gpt4-1106/chat/completions?api-version=2023-07-01-preview"Distributed under the MIT License. See LICENSE.txt for more information.
If you have any questions, feel free to message us on robustacommunity.slack.com
To contribute to HolmesGPT, first follow the Installation instructions for running HolmesGPT from source using Poetry. Then follow an appropriate guide below, or ask us for help on Slack
Adding new runbooks
You can contribute knowledge on solving common alerts and HolmesGPT will use this knowledge to solve related issues. To do so, add a new file to ./holmes/plugins/runbooks - or edit an existing runbooks file in that same directory.
Note: if you prefer to keep your runbooks private, you can store them locally and pass them to HolmesGPT with the -r flag. However, if your runbooks relate to common problems that others may encounter, please consider opening a PR and making HolmesGPT better for everyone!
Adding new toolsets
You can add define new tools in YAML and HolmesGPT will use those tools in it's investigation. To do so, add a new file to ./holmes/plugins/toolsets - or edit an existing toolsets file in that same directory.
Note: if you prefer to keep your tools private, you can store them locally and pass them to HolmesGPT with the -t flag. However, please consider contributing your toolsets! At least one other community member will probably find them useful!
Modifying the default prompts (prompt engineering)
The default prompts for HolmesGPT are located in ./holmes/plugins/prompts. Most holmes commands accept a --system-prompt flag that you can use to override this.
If you find a scenario where the default prompts don't work, please consider letting us know by opening a GitHub issue or messaging us on Slack! We have an internal evaluation framework for benchmarking prompts on many troubleshooting scenarios and if you share a case where HolmesGPT doesn't work, we will be able to add it to our test framework and fix the performance on that issue and similar ones.
Adding new data sources
If you want HolmesGPT to investigate external tickets or alert, you can add a new datasource. This requires modifying the source code and opening a PR. You can see an example PR like that here, which added support for investigating GitHub issues.