A Practical Toolkit for Structured LLM Data Handling
island-ai is a TypeScript toolkit for development, integration, and evaluation of structured outputs from Large Language Models (LLMs). Our toolkit includes packages for schema-validation, streaming JSON data, evaluating AI-generated content, and universal LLM client interfaces. Below are the core packages that make up Island AI.
zod-stream is designed to define structured response models for OpenAI or Anyscale completions using Zod schemas. It enables partial streaming of JSON so that it can be used safely and immediately.
- Define structured response models.
- Enable partial JSON streaming.
- Safely parse and use responses right away.
# with pnpm
$ pnpm add zod-stream zod openai
# with npm
$ npm install zod-stream zod openai
# with bun
$ bun add zod-stream zod openaiimport { OAIStream } from "zod-stream";
import { withResponseModel } from "zod-stream";
import OpenAI from "openai";
import { z } from "zod";
const oai = new OpenAI({
apiKey: process.env["OPENAI_API_KEY"] ?? undefined,
organization: process.env["OPENAI_ORG_ID"] ?? undefined
});
// Define a response model using Zod
const schema = z.object({
content: z.string(),
users: z.array(z.object({
name: z.string(),
})),
});
// API Route Example (Next.js)
export async function POST(request: Request) {
const { messages } = await request.json();
const params = withResponseModel({
response_model: { schema: schema, name: "Users extraction and message" },
params: {
messages,
model: "gpt-4",
},
mode: "TOOLS",
});
const extractionStream = await oai.chat.completions.create({
...params,
stream: true,
});
return new Response(OAIStream({ res: extractionStream }));
}schema-stream is a utility for parsing streams of JSON data. It provides safe-to-read-from stubbed versions of data before the stream has fully completed.
- Stream JSON data parsing with partial data availability.
- Zod schema validation for robust data handling.
- Incremental model hydration for real-time data processing.
npm install schema-stream zodimport { SchemaStream } from "schema-stream";
import { z } from "zod";
const schema = z.object({
someString: z.string(),
someNumber: z.number(),
});
const response = await getSomeStreamOfJson();
const parser = new SchemaStream(schema, {
someString: "default string",
});
const streamParser = parser.parse({});
response.body?.pipeThrough(parser);
const reader = streamParser.readable.getReader();
const decoder = new TextDecoder();
let result = {};
while (!done) {
const { value, done: doneReading } = await reader.read();
done = doneReading;
if (done) {
console.log(result);
break;
}
const chunkValue = decoder.decode(value);
result = JSON.parse(chunkValue);
}stream-hooks provides React hooks for consuming streams of JSON data, particularly from LLMs. It integrates seamlessly with Zod schemas to ensure structured data handling.
- React hooks for consuming streaming JSON data.
- Seamlessly integrates with Zod for structured data validation.
- Hooks facilitate the incorporation of live data feeds into user interfaces.
# with pnpm
$ pnpm add stream-hooks zod zod-stream
# with npm
$ npm install stream-hooks zod zod-stream
# with bun
$ bun add stream-hooks zod zod-streamimport { useJsonStream } from "stream-hooks";
import { z } from "zod";
const schema = z.object({
content: z.string(),
});
export function Test() {
const { loading, startStream, stopStream, data } = useJsonStream({
schema,
onReceive: data => {
console.log("incremental update to final response model", data);
},
});
const submit = async () => {
try {
await startStream({
url: "/api/ai/chat",
method: "POST",
body: { messages: [{ content: "yo", role: "user" }] },
});
} catch (e) {
console.error(e);
}
};
return (
<div>
{data.content}
<button onClick={submit}>start</button>
<button onClick={stopStream}>stop</button>
</div>
);
}llm-polyglot is a universal LLM client that provides support for various LLM providers, ensuring a consistent API interface.
- Extends the official OpenAI SDK.
- Supports providers like Anthropic, Together, OpenAI, Microsoft, Anyscale, and Anthropic.
- Universal SDK for multiple LLMs with consistent API.
# with pnpm
$ pnpm add llm-polyglot openai
# with npm
$ npm install llm-polyglot openai
# with bun
$ bun add llm-polyglot openaiimport { createLLMClient } from "llm-polyglot";
const anthropicClient = createLLMClient({
provider: "anthropic",
});
const completion = await anthropicClient.chat.completions.create({
model: "claude-3-opus-20240229",
max_tokens: 1000,
messages: [{ role: "user", content: "hey how are you" }],
});llm-polyglot is a universal LLM client that provides support for various LLM providers, ensuring a consistent API interface.
Key Features:
- Extends the official OpenAI SDK.
- Supports providers like Anthropic, Together, OpenAI, Microsoft, Anyscale, and Anthropic.
- Universal SDK for multiple LLMs with consistent API.
# with pnpm
$ pnpm add llm-polyglot openai
# with npm
$ npm install llm-polyglot openai
# with bun
$ bun add llm-polyglot openaiimport { createLLMClient } from "llm-polyglot";
const anthropicClient = createLLMClient({
provider: "anthropic",
});
const completion = await anthropicClient.chat.completions.create({
model: "claude-3-opus-20240229",
max_tokens: 1000,
messages: [{ role: "user", content: "hey how are you" }],
});evalz is a package designed to facilitate both model-graded, accuracy, and context-based evaluations with a focus on structured output.
Key Features:
- Structured evaluation models using Zod schemas.
- Flexible evaluation strategies including score-based and binary evaluations.
- Integration with OpenAI for structured model-graded evaluations.
- Supports accuracy evaluations using Levenshtein distance or semantic embeddings.
- Context-based evaluations measuring precision, recall, and entities recall.
npm install evalz openai zod @instructor-ai/instructorimport { createEvaluator } from "evalz";
import OpenAI from "openai";
const oai = new OpenAI({
apiKey: process.env["OPENAI_API_KEY"],
organization: process.env["OPENAI_ORG_ID"],
});
// Define a relevance evaluator
function relevanceEval() {
return createEvaluator({
client: oai,
model: "gpt-4-turbo",
evaluationDescription: "Rate the relevance from 0 to 1.",
});
}
// Conducting an evaluation
const evaluator = relevanceEval();
const result = await evaluator({ data: yourResponseData });
console.log(result.scoreResults);