[v3.1] Messages from `useChat` hook are of type `Message` instead of `CoreMessage` for multimodal use #2039

omzi · 2024-06-20T22:05:26Z

omzi
Jun 20, 2024

👋 Hello!

I'm working a GPT-4o multimodal chatbot. Here's my Next.js API route:

import rateLimit from '#/lib/rateLimit';
import { openai } from '@ai-sdk/openai';
import { NextRequest, NextResponse } from 'next/server';
import { edgestoreBackendClient } from '#/lib/edgestoreServer';
import { type CoreMessage, streamText, CoreUserMessage, UserContent } from 'ai';

export const runtime = 'edge';
export const maxDuration = 30;

export const POST = async (req: NextRequest, res: NextResponse) => {
  const isRateLimited = await rateLimit(req, res);
  if (isRateLimited) {
    return new Response('Rate limited!', { status: 429 });
  }
  
  const { messages, chatImageUrl }: { messages: CoreMessage[], chatImageUrl?: string } = await req.json();
  const newMessage = messages.pop();
  const modifiedMessage = { role: 'user', content: [] } as CoreUserMessage;

  if (chatImageUrl) {
    await edgestoreBackendClient.chatFiles.confirmUpload({ url: chatImageUrl });
    modifiedMessage.content = [...modifiedMessage.content, { type: 'image', image: chatImageUrl }] as UserContent
  }

  if (newMessage && typeof newMessage.content === 'string') {
    modifiedMessage.content = [...modifiedMessage.content, { type: 'text', text: newMessage.content }] as UserContent
  }

  const allMessages = [ ...messages, modifiedMessage ];
  const system = `...`; 
  
  const model = openai.chat('gpt-4o');
  const result = await streamText({
    model,
    system,
    messages: allMessages,
    onFinish: async (event) => {
      console.log('Completion Text :>>', event.text);
    }
  });

  return result.toAIStreamResponse();
}

In this route, I format a user message to be able to add in an image URL. The route works alright but the useChat hook somehow removes the image and sends only a string message.

Here's my useChat hook:

const { messages, error, handleSubmit, isLoading, input, setInput, handleInputChange, stop, reload } = useChat({
	api: '/api/chat',
	initialMessages: [],
	body: { chatImageUrl },
	onError: (error) => {
		toast.error(error.message);
	}
});

The messages array is of type Message[] instead of CoreMessage[] (which supports image type messages) as I use on the route. I've also tried logging the content of the messages array to be sure but I get a string message still.

How does one work with images with the useChat hook? Could you share a sample please? If the useChat hook is not ready for multimodal models, is there a recent solution you can recommend?

PS: I haven't seen an official example for multimodal chatbots 🤔 (except the one on https://sdk.vercel.ai which is not open-source ;/).