An effective open-source system for classifying conversation flows in large language models (LLMs) using FastAPI. This system incorporates two models to detect typical events and patterns in interactions with LLMs, such as recognizing an apology where the LLM admits a mistake, or identifying a complaint when a user shows dissatisfaction. These labels serve as foundational elements for sophisticated LLM analytics 📊.

The models have been optimized for rapid CPU-based inference using ONNX, enabling efficient performance ⚡. Furthermore, this system is designed for deployment on serverless platforms.

This project functions as the backend supporting the flowdetect-plugin designed for use with PostHog-LLM.

git clone https://github.com/minuva/llm-flow-classification.git
cd llm-flow-classification.git
pip install -r requirements.txt

Run the following command to start the server (from the root directory):

chmod +x ./run.sh
./run.sh

Check config.py for more configuration options.

Run the following command to start the server (the root directory):

docker build --tag llmflow .
docker run --network=postlang --network-alias=flowdetect -p 9612:9612 -it llmflow

The network and the network alias are used to allow PostHog-LLM to communicate with the flowdetect classification service. Since PostHog-LLM is running in a docker container, we connect the two services by adding them to the same network for fast and reliable communication.

curl -X 'POST' \
  'http:https://127.0.0.1:9612/conversation_flow_plugin' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "llm_input": "That is fine, can you expand on that list",
  "llm_output": "My apologies."
}'

And returns

{
  "user_flow": "more_listing_or_expand",
  "agent_flow": "agent_apology_error_mistake"
}