Using message file attachments with assistants

[dmeyeroc]

Are user-uploaded attachments supported yet using assistants 2.0?

Our use-case is that a user uploads a file and asks a question about the text content of the file, and the AI model produces a result based on the text content of the file and the user's question.

I've been going through the provided examples and sifting through the code, but I don't see a way of referencing/including uploaded files in an assistant thread message. I only see examples of uploading files to a newly created assistant, or using images as message content, neither of which are what we are trying to do.

...
RunCreationOptions runCreationOptions = new RunCreationOptions()
{
    InstructionsOverride = "Sample role",
    ModelOverride = "gpt-4o"
}

runCreationOptions.ToolsOverride.Add(new FileSearchToolDefinition());

ThreadCreationOptions threadCreationOptions = new()
{
    InitialMessages =
    {
        new ThreadInitializationMessage(new List<MessageContent>
        {
            "Hello, assistant! Please tell me something about the contents of this file",
            //MessageContent.FromImageFileId(pictureOfAppleFile.Id), // don't need this
            //MessageContent.FromImageUrl(linkToPictureOfOrange), // don't need this
            // Is there a different way to get it to reference a file that is not an image?
        })
    },
};

ThreadRun threadRun = await _assistantsClient.CreateThreadAndRunAsync(_assistantId, threadCreationOptions, runCreationOptions)
...

[trrwilson]

Hello, @dmeyeroc!

As also outlined in the Assistants API reference (c.f. content and attachments under messages), message content parts and message attachments are distinct and parallel concepts -- and the SDK does support both. Content parts only support images, as you saw, but attachments support general file IDs for consumption by the specified tools.

Here's a quick snippet walking through (1) step-by-step creation of a thread that has a starting message that includes an attachment; and (2) adding another message with both an attachment and mixed content items.

// Content for the message
MessageContent contentForMessage
    = MessageContent.FromText("Calculate ANOVA for data points in the attached file.");

// Attachment for the message
MessageCreationAttachment attachmentForCodeInterpreter = new(
    fileId: "your-data-points-file-id",
    tools: [new CodeInterpreterToolDefinition()]);

// Message for the thread
ThreadInitializationMessage messageForNewThread = new([contentForMessage]);
messageForNewThread.Attachments.Add(attachmentForCodeInterpreter);

// Options for creating the thread
ThreadCreationOptions optionsForNewThread = new()
{
    InitialMessages = { messageForNewThread },
};

// Creating the thread
AssistantThread thread = client.CreateThread(optionsForNewThread);

// Adding another message, with everything in one shot
ThreadMessage additionalMessage = client.CreateMessage(
    thread,
    content:
    [
        "Also, what's this image and how does it relate to a data plot of this additional attached file?",
        MessageContent.FromImageUrl(imageUri),
    ],
    options: new MessageCreationOptions()
    {
        Attachments =
        {
            new MessageCreationAttachment("another-file-id", [new CodeInterpreterToolDefinition()]);
        },
    });

Does that additional Attachments property in MessageCreationOptions/ThreadInitializationMessage line up with your needs? Also, if there's anything that could make this clearer, please share your feedback! It's a sophisticated API with a lot going on all at once, so we want to smooth out as many avoidable rough edges as we can.

[dmeyeroc]

Thanks for your quick response @trrwilson!

Yes, this appears to be exactly what we are looking for. We'll put it through its paces and if I hit any more roadblocks I'll bring it back to this thread to hopefully help educate others.

I'd recommend adding this example to those that appear in the readme, as I'm sure others may be looking for it there.

Thank you again for your responsiveness and detailed answer.

[devsienko]

Hi, I see MessageContent.FromImageUrl(imageUri), but it requires Uri class instanse I far as I understand I cannot to use data url like that:

"image_url": {
    "url": f"data:image/jpeg;base64,{base64_image}"
},

but I can do it by Open AI API, is there any way to do it by your library?

[crickman]

@devsienko - #135