🪢 Open source LLM engineering platform: LLM Observability, metrics, evals, prompt management, playground, datasets. Integrates with LlamaIndex, Langchain, OpenAI SDK, LiteLLM, and more. 🍊YC W23
-
Updated
Aug 30, 2024 - TypeScript
🪢 Open source LLM engineering platform: LLM Observability, metrics, evals, prompt management, playground, datasets. Integrates with LlamaIndex, Langchain, OpenAI SDK, LiteLLM, and more. 🍊YC W23
Test your prompts, agents, and RAGs. Red teaming, pentesting, and vulnerability scanning for LLMs. Compare performance of GPT, Claude, Gemini, Llama, and more. Simple declarative configs with command line and CI/CD integration.
🐢 Open-Source Evaluation & Testing for LLMs and ML models
The LLM Evaluation Framework
🧊 Open source LLM-Observability Platform for Developers. One-line integration for monitoring, metrics, evals, agent tracing, prompt management, playground, etc. Supports OpenAI SDK, Vercel AI SDK, Anthropic SDK, LiteLLM, LLamaIndex, LangChain, and more. 🍓 YC W23
The all-in-one LLM developer platform: prompt management, evaluation, human feedback, and deployment all in one place.
Data-Driven Evaluation for LLM-Powered Applications
Awesome-LLM-Eval: a curated list of tools, datasets/benchmark, demos, leaderboard, papers, docs and models, mainly for Evaluation on LLMs. 一个由工具、基准/数据、演示、排行榜和大模型等组成的精选列表,主要面向基础大模型评测,旨在探求生成式AI的技术边界.
Prompty makes it easy to create, manage, debug, and evaluate LLM prompts for your AI applications. Prompty is an asset class and format for LLM prompts designed to enhance observability, understandability, and portability for developers.
Awesome papers involving LLMs in Social Science.
The official evaluation suite and dynamic data release for MixEval.
Python SDK for running evaluations on LLM generated responses
Connect agents to live web environments evaluation.
A list of LLMs Tools & Projects
Superpipe - optimized LLM pipelines for structured data
Framework for LLM evaluation, guardrails and security
Evaluating LLMs with CommonGen-Lite
Initiative to evaluate and rank the most popular LLMs across common task types based on their propensity to hallucinate.
A simple GPT-based evaluation tool for multi-aspect, interpretable assessment of LLMs.
LeanEuclid is a benchmark for autoformalization in the domain of Euclidean geometry, targeting the proof assistant Lean.
Add a description, image, and links to the llm-evaluation topic page so that developers can more easily learn about it.
To associate your repository with the llm-evaluation topic, visit your repo's landing page and select "manage topics."