A simple, multi-user, multi-conversation, web-based pybot.

Pybot incorporates OpenID Connect for user identification. It relies on an external OAuth Client oidc-authservice to handle authentication and set a trusted userid Header to the downstream services. Alternatively, oauth2-proxy, which is more actively maintained, can be used in place of oidc-authservice.

Pybot supports multiple conversations. Each conversation is identified by a unique conversationId. Conversations consists of a sequence of messages as well as metadata such as title, updatedAt. Message persistance is handled by langchain's RedisChatMessageHistory module, which leverages Redis for storing chat history. Metadata persistance is handled separately by redis-om, an object mapping library for Redis from Redis Labs. This separation of message content and metadata storage provides modularity and flexibility in the pybot's underlying persistence architecture.

Pybot supports streaming LLM outputs to the user in real-time. Streaming messages are delivered via WebSockets, which enables bidirectional, full-duplex communication channels between the server and client.

On the LLM side, Pybot uses Text Generation Inference (TGI), an open source library from HuggingFace, to host large language models for text generation. TGI provides out-of-the-box support for continuous batching, streaming inference, and other useful features for deploying production-ready LLMs. Using TGI eliminates the need to build complex serving infrastructure from scratch. Its continuous batching allows the pybot to achieve high throughput by batching requests. Streaming inference enables the pybot to return partial results instantly rather than waiting for the full output.

See deployment instructions