Based on OpenAI-DotNet
A OpenAI package for the Unity to use though their RESTful API. Independently developed, this is not an official library and I am not affiliated with OpenAI. An OpenAI API account is required.
All copyrights, trademarks, logos, and assets are the property of their respective owners.
This repository is available to transfer to the OpenAI organization if they so choose to accept it.
Requires Unity 2021.3 LTS or higher.
The recommended installation method is though the unity package manager and OpenUPM.
- Open your Unity project settings
- Add the OpenUPM package registry:
- Name:
OpenUPM
- URL:
https://package.openupm.com
- Scope(s):
com.openai
com.utilities
- Name:
- Open the Unity Package Manager window
- Change the Registry from Unity to
My Registries
- Add the
OpenAI
package
- Open your Unity Package Manager
- Add package from git url:
https://github.com/RageAgainstThePixel/com.openai.unity.git#upm
Note: this repo has dependencies on other repositories! You are responsible for adding these on your own.
- Authentication
- Azure OpenAI
- OpenAI API Proxy
- Models
- Assistants 🆕
- Threads 🆕
- Chat
- Audio
- Images
- Files
- Fine Tuning
- Embeddings
- Moderations
Completions⚠️ DeprecatedStreaming⚠️ Deprecated
Edits⚠️ DeprecatedCreate Edit⚠️ Deprecated
There are 4 ways to provide your API keys, in order of precedence:
- Pass keys directly with constructor
⚠️ - Unity Scriptable Object
⚠️ - Load key from configuration file
- Use System Environment Variables
You use the OpenAIAuthentication
when you initialize the API as shown:
var api = new OpenAIClient("sk-apiKey");
Or create a OpenAIAuthentication
object manually
var api = new OpenAIClient(new OpenAIAuthentication("sk-apiKey", "org-yourOrganizationId"));
You can save the key directly into a scriptable object that is located in the Assets/Resources
folder.
You can create a new one by using the context menu of the project pane and creating a new OpenAIConfiguration
scriptable object.
Attempts to load api keys from a configuration file, by default .openai
in the current directory, optionally traversing up the directory tree or in the user's home directory.
To create a configuration file, create a new text file named .openai
and containing the line:
Organization entry is optional.
{
"apiKey": "sk-aaaabbbbbccccddddd",
"organization": "org-yourOrganizationId"
}
OPENAI_KEY=sk-aaaabbbbbccccddddd
ORGANIZATION=org-yourOrganizationId
You can also load the configuration file directly with known path by calling static methods in OpenAIAuthentication
:
- Loads the default
.openai
config in the specified directory:
var api = new OpenAIClient(new OpenAIAuthentication().LoadFromDirectory("path/to/your/directory"));
- Loads the configuration file from a specific path. File does not need to be named
.openai
as long as it conforms to the json format:
var api = new OpenAIClient(new OpenAIAuthentication().LoadFromPath("path/to/your/file.json"));
Use your system's environment variables specify an api key and organization to use.
- Use
OPENAI_API_KEY
for your api key. - Use
OPENAI_ORGANIZATION_ID
to specify an organization.
var api = new OpenAIClient(new OpenAIAuthentication().LoadFromEnvironment());
You can also choose to use Microsoft's Azure OpenAI deployments as well.
You can find the required information in the Azure Playground by clicking the View Code
button and view a URL like this:
https://{your-resource-name}.openai.azure.com/openai/deployments/{deployment-id}/chat/completions?api-version={api-version}
your-resource-name
The name of your Azure OpenAI Resource.deployment-id
The deployment name you chose when you deployed the model.api-version
The API version to use for this operation. This follows the YYYY-MM-DD format.
To setup the client to use your deployment, you'll need to pass in OpenAISettings
into the client constructor.
var auth = new OpenAIAuthentication("sk-apiKey");
var settings = new OpenAISettings(resourceName: "your-resource-name", deploymentId: "deployment-id", apiVersion: "api-version");
var api = new OpenAIClient(auth, settings);
Authenticate with MSAL as usual and get access token, then use the access token when creating your OpenAIAuthentication
. Then be sure to set useAzureActiveDirectory to true when creating your OpenAISettings
.
Tutorial: Desktop app that calls web APIs: Acquire a token
// get your access token using any of the MSAL methods
var accessToken = result.AccessToken;
var auth = new OpenAIAuthentication(accessToken);
var settings = new OpenAISettings(resourceName: "your-resource", deploymentId: "deployment-id", apiVersion: "api-version", useActiveDirectoryAuthentication: true);
var api = new OpenAIClient(auth, settings);
Using either the OpenAI-DotNet or com.openai.unity packages directly in your front-end app may expose your API keys and other sensitive information. To mitigate this risk, it is recommended to set up an intermediate API that makes requests to OpenAI on behalf of your front-end app. This library can be utilized for both front-end and intermediary host configurations, ensuring secure communication with the OpenAI API.
In the front end example, you will need to securely authenticate your users using your preferred OAuth provider. Once the user is authenticated, exchange your custom auth token with your API key on the backend.
Follow these steps:
- Setup a new project using either the OpenAI-DotNet or com.openai.unity packages.
- Authenticate users with your OAuth provider.
- After successful authentication, create a new
OpenAIAuthentication
object and pass in the custom token with the prefixsess-
. - Create a new
OpenAISettings
object and specify the domain where your intermediate API is located. - Pass your new
auth
andsettings
objects to theOpenAIClient
constructor when you create the client instance.
Here's an example of how to set up the front end:
var authToken = await LoginAsync();
var auth = new OpenAIAuthentication($"sess-{authToken}");
var settings = new OpenAISettings(domain: "api.your-custom-domain.com");
var api = new OpenAIClient(auth, settings);
This setup allows your front end application to securely communicate with your backend that will be using the OpenAI-DotNet-Proxy, which then forwards requests to the OpenAI API. This ensures that your OpenAI API keys and other sensitive information remain secure throughout the process.
In this example, we demonstrate how to set up and use OpenAIProxyStartup
in a new ASP.NET Core web app. The proxy server will handle authentication and forward requests to the OpenAI API, ensuring that your API keys and other sensitive information remain secure.
- Create a new ASP.NET Core minimal web API project.
- Add the OpenAI-DotNet nuget package to your project.
- Powershell install:
Install-Package OpenAI-DotNet-Proxy
- Manually editing .csproj:
<PackageReference Include="OpenAI-DotNet-Proxy" />
- Powershell install:
- Create a new class that inherits from
AbstractAuthenticationFilter
and override theValidateAuthentication
method. This will implement theIAuthenticationFilter
that you will use to check user session token against your internal server. - In
Program.cs
, create a new proxy web application by callingOpenAIProxyStartup.CreateDefaultHost
method, passing your customAuthenticationFilter
as a type argument. - Create
OpenAIAuthentication
andOpenAIClientSettings
as you would normally with your API keys, org id, or Azure settings.
public partial class Program
{
private class AuthenticationFilter : AbstractAuthenticationFilter
{
public override void ValidateAuthentication(IHeaderDictionary request)
{
// You will need to implement your own class to properly test
// custom issued tokens you've setup for your end users.
if (!request.Authorization.ToString().Contains(userToken))
{
throw new AuthenticationException("User is not authorized");
}
}
}
public static void Main(string[] args)
{
var auth = OpenAIAuthentication.LoadFromEnv();
var settings = new OpenAIClientSettings(/* your custom settings if using Azure OpenAI */);
var openAIClient = new OpenAIClient(auth, settings);
var proxy = OpenAIProxyStartup.CreateDefaultHost<AuthenticationFilter>(args, openAIClient);
proxy.Run();
}
}
Once you have set up your proxy server, your end users can now make authenticated requests to your proxy api instead of directly to the OpenAI API. The proxy server will handle authentication and forward requests to the OpenAI API, ensuring that your API keys and other sensitive information remain secure.
List and describe the various models available in the API. You can refer to the Models documentation to understand what models are available and the differences between them.
Also checkout model endpoint compatibility to understand which models work with which endpoints.
To specify a custom model not pre-defined in this library:
var model = new Model("model-id");
The Models API is accessed via OpenAIClient.ModelsEndpoint
Lists the currently available models, and provides basic information about each one such as the owner and availability.
var api = new OpenAIClient();
var models = await api.ModelsEndpoint.GetModelsAsync();
foreach (var model in models)
{
Debug.Log(model.ToString());
}
Retrieves a model instance, providing basic information about the model such as the owner and permissions.
var api = new OpenAIClient();
var model = await api.ModelsEndpoint.GetModelDetailsAsync("text-davinci-003");
Debug.Log(model.ToString());
Delete a fine-tuned model. You must have the Owner role in your organization.
var api = new OpenAIClient();
var isDeleted = await api.ModelsEndpoint.DeleteFineTuneModelAsync("your-fine-tuned-model");
Assert.IsTrue(isDeleted);
⚠️ Beta Feature
Build assistants that can call models and use tools to perform tasks.
The Assistants API is accessed via OpenAIClient.AssistantsEndpoint
Returns a list of assistants.
var api = new OpenAIClient();
var assistantsList = await api.AssistantsEndpoint.ListAssistantsAsync();
foreach (var assistant in assistantsList.Items)
{
Debug.Log($"{assistant} -> {assistant.CreatedAt}");
}
Create an assistant with a model and instructions.
var api = new OpenAIClient();
var request = new CreateAssistantRequest("gpt-3.5-turbo-1106");
var assistant = await api.AssistantsEndpoint.CreateAssistantAsync(request);
Retrieves an assistant.
var api = new OpenAIClient();
var assistant = await api.AssistantsEndpoint.RetrieveAssistantAsync("assistant-id");
Debug.Log($"{assistant} -> {assistant.CreatedAt}");
Modifies an assistant.
var api = new OpenAIClient();
var createRequest = new CreateAssistantRequest("gpt-3.5-turbo-1106");
var assistant = await api.AssistantsEndpoint.CreateAssistantAsync(createRequest);
var modifyRequest = new CreateAssistantRequest("gpt-4-1106-preview");
var modifiedAssistant = await api.AssistantsEndpoint.ModifyAsync(assistant.Id, modifyRequest);
// OR AssistantExtension for easier use!
var modifiedAssistantEx = await assistant.ModifyAsync(modifyRequest);
Delete an assistant.
var api = new OpenAIClient();
var isDeleted = await api.AssistantsEndpoint.DeleteAssistantAsync("assistant-id");
// OR AssistantExtension for easier use!
var isDeleted = await assistant.DeleteAsync();
Assert.IsTrue(isDeleted);
Returns a list of assistant files.
var api = new OpenAIClient();
var filesList = await api.AssistantsEndpoint.ListFilesAsync("assistant-id");
// OR AssistantExtension for easier use!
var filesList = await assistant.ListFilesAsync();
foreach (var file in filesList.Items)
{
Debug.Log($"{file.AssistantId}'s file -> {file.Id}");
}
Create an assistant file by attaching a File to an assistant.
var api = new OpenAIClient();
var filePath = "assistant_test_2.txt";
await File.WriteAllTextAsync(filePath, "Knowledge is power!");
var fileUploadRequest = new FileUploadRequest(filePath, "assistant");
var file = await api.FilesEndpoint.UploadFileAsync(fileUploadRequest);
var assistantFile = await api.AssistantsEndpoint.AttachFileAsync("assistant-id", file.Id);
// OR use extension method for convenience!
var assistantFIle = await assistant.AttachFileAsync(file);
Uploads and attaches a file to an assistant.
Assistant extension method, for extra convenience!
var api = new OpenAIClient();
var filePath = "assistant_test_2.txt";
await File.WriteAllTextAsync(filePath, "Knowledge is power!");
var assistantFile = await assistant.UploadFileAsync(filePath);
Retrieves an AssistantFile.
var api = new OpenAIClient();
var assistantFile = await api.AssistantsEndpoint.RetrieveFileAsync("assistant-id", "file-id");
// OR AssistantExtension for easier use!
var assistantFile = await assistant.RetrieveFileAsync(fileId);
Debug.Log($"{assistantFile.AssistantId}'s file -> {assistantFile.Id}");
Remove a file from an assistant.
Note: The file will remain in your organization until deleted with FileEndpoint.
var api = new OpenAIClient();
var isRemoved = await api.AssistantsEndpoint.RemoveFileAsync("assistant-id", "file-id");
// OR use extension method for convenience!
var isRemoved = await assistant.RemoveFileAsync("file-id");
Assert.IsTrue(isRemoved);
Removes a file from the assistant and then deletes the file from the organization.
Assistant extension method, for extra convenience!
var api = new OpenAIClient();
var isDeleted = await assistant.DeleteFileAsync("file-id");
Assert.IsTrue(isDeleted);
⚠️ Beta Feature
Create Threads that Assistants can interact with.
The Threads API is accessed via OpenAIClient.ThreadsEndpoint
Create a thread.
var api = new OpenAIClient();
var thread = await api.ThreadsEndpoint.CreateThreadAsync();
Debug.Log($"Create thread {thread.Id} -> {thread.CreatedAt}");
Create a thread and run it in one request.
See also: Thread Runs
var api = new OpenAIClient();
var assistant = await api.AssistantsEndpoint.CreateAssistantAsync(
new CreateAssistantRequest(
name: "Math Tutor",
instructions: "You are a personal math tutor. Answer questions briefly, in a sentence or less.",
model: "gpt-4-1106-preview"));
var messages = new List<Message> { "I need to solve the equation `3x + 11 = 14`. Can you help me?" };
var threadRequest = new CreateThreadRequest(messages);
var run = await assistant.CreateThreadAndRunAsync(threadRequest);
Debug.Log($"Created thread and run: {run.ThreadId} -> {run.Id} -> {run.CreatedAt}");
Retrieves a thread.
var api = new OpenAIClient();
var thread = await api.ThreadsEndpoint.RetrieveThreadAsync("thread-id");
// OR if you simply wish to get the latest state of a thread
thread = await thread.UpdateAsync();
Debug.Log($"Retrieve thread {thread.Id} -> {thread.CreatedAt}");
Modifies a thread.
Note: Only the metadata can be modified.
var api = new OpenAIClient();
var thread = await api.ThreadsEndpoint.CreateThreadAsync();
var metadata = new Dictionary<string, string>
{
{ "key", "custom thread metadata" }
}
thread = await api.ThreadsEndpoint.ModifyThreadAsync(thread.Id, metadata);
// OR use extension method for convenience!
thread = await thread.ModifyAsync(metadata);
Debug.Log($"Modify thread {thread.Id} -> {thread.Metadata["key"]}");
Delete a thread.
var api = new OpenAIClient();
var isDeleted = await api.ThreadsEndpoint.DeleteThreadAsync("thread-id");
// OR use extension method for convenience!
var isDeleted = await thread.DeleteAsync();
Assert.IsTrue(isDeleted);
Create messages within threads.
Returns a list of messages for a given thread.
var api = new OpenAIClient();
var messageList = await api.ThreadsEndpoint.ListMessagesAsync("thread-id");
// OR use extension method for convenience!
var messageList = await thread.ListMessagesAsync();
foreach (var message in messageList.Items)
{
Debug.Log($"{message.Id}: {message.Role}: {message.PrintContent()}");
}
Create a message.
var api = new OpenAIClient();
var thread = await api.ThreadsEndpoint.CreateThreadAsync();
var request = new CreateMessageRequest("Hello world!");
var message = await api.ThreadsEndpoint.CreateMessageAsync(thread.Id, request);
// OR use extension method for convenience!
var message = await thread.CreateMessageAsync("Hello World!");
Debug.Log($"{message.Id}: {message.Role}: {message.PrintContent()}");
Retrieve a message.
var api = new OpenAIClient();
var message = await api.ThreadsEndpoint.RetrieveMessageAsync("thread-id", "message-id");
// OR use extension methods for convenience!
var message = await thread.RetrieveMessageAsync("message-id");
var message = await message.UpdateAsync();
Debug.Log($"{message.Id}: {message.Role}: {message.PrintContent()}");
Modify a message.
Note: Only the message metadata can be modified.
var api = new OpenAIClient();
var metadata = new Dictionary<string, string>
{
{ "key", "custom message metadata" }
};
var message = await api.ThreadsEndpoint.ModifyMessageAsync("thread-id", "message-id", metadata);
// OR use extension method for convenience!
var message = await message.ModifyAsync(metadata);
Debug.Log($"Modify message metadata: {message.Id} -> {message.Metadata["key"]}");
Returns a list of message files.
var api = new OpenAIClient();
var fileList = await api.ThreadsEndpoint.ListFilesAsync("thread-id", "message-Id");
// OR use extension method for convenience!
var fileList = await thread.ListFilesAsync("message-id");
var fileList = await message.ListFilesAsync();
foreach (var file in fileList.Items)
{
Debug.Log(file.Id);
}
Retrieves a message file.
var api = new OpenAIClient();
var file = await api.ThreadsEndpoint.RetrieveFileAsync("thread-id", "message-id", "file-id");
// OR use extension method for convenience!
var file = await message.RetrieveFileAsync();
Debug.Log(file.Id);
Represents an execution run on a thread.
Returns a list of runs belonging to a thread.
var api = new OpenAIClient();
var runList = await api.ThreadsEndpoint.ListRunsAsync("thread-id");
// OR use extension method for convenience!
var runList = await thread.ListRunsAsync();
foreach (var run in runList.Items)
{
Debug.Log($"[{run.Id}] {run.Status} | {run.CreatedAt}");
}
Create a run.
var api = new OpenAIClient();
var assistant = await api.AssistantsEndpoint.CreateAssistantAsync(
new CreateAssistantRequest(
name: "Math Tutor",
instructions: "You are a personal math tutor. Answer questions briefly, in a sentence or less.",
model: "gpt-4-1106-preview"));
var thread = await api.ThreadsEndpoint.CreateThreadAsync();
var message = await thread.CreateMessageAsync("I need to solve the equation `3x + 11 = 14`. Can you help me?");
var run = await thread.CreateRunAsync(assistant);
Debug.Log($"[{run.Id}] {run.Status} | {run.CreatedAt}");
Retrieves a run.
var api = new OpenAIClient();
var run = await api.ThreadsEndpoint.RetrieveRunAsync("thread-id", "run-id");
// OR use extension method for convenience!
var run = await thread.RetrieveRunAsync("run-id");
var run = await run.UpdateAsync();
Debug.Log($"[{run.Id}] {run.Status} | {run.CreatedAt}");
Modifies a run.
Note: Only the metadata can be modified.
var api = new OpenAIClient();
var metadata = new Dictionary<string, string>
{
{ "key", "custom run metadata" }
};
var run = await api.ThreadsEndpoint.ModifyRunAsync("thread-id", "run-id", metadata);
// OR use extension method for convenience!
var run = await run.ModifyAsync(metadata);
Debug.Log($"Modify run {run.Id} -> {run.Metadata["key"]}");
When a run has the status: requires_action
and required_action.type
is submit_tool_outputs
, this endpoint can be used to submit the outputs from the tool calls once they're all completed. All outputs must be submitted in a single request.
var api = new OpenAIClient();
var function = new Function(
nameof(WeatherService.GetCurrentWeather),
"Get the current weather in a given location",
new JObject
{
["type"] = "object",
["properties"] = new JObject
{
["location"] = new JObject
{
["type"] = "string",
["description"] = "The city and state, e.g. San Francisco, CA"
},
["unit"] = new JObject
{
["type"] = "string",
["enum"] = new JArray { "celsius", "fahrenheit" }
}
},
["required"] = new JArray { "location", "unit" }
});
testAssistant = await api.AssistantsEndpoint.CreateAssistantAsync(new CreateAssistantRequest(tools: new Tool[] { function }));
var run = await testAssistant.CreateThreadAndRunAsync("I'm in Kuala-Lumpur, please tell me what's the temperature in celsius now?");
// waiting while run is Queued and InProgress
run = await run.WaitForStatusChangeAsync();
var toolCall = run.RequiredAction.SubmitToolOutputs.ToolCalls[0];
Debug.Log($"tool call arguments: {toolCall.FunctionCall.Arguments}");
var functionArgs = JsonConvert.DeserializeObject<WeatherArgs>(toolCall.FunctionCall.Arguments);
var functionResult = WeatherService.GetCurrentWeather(functionArgs);
var toolOutput = new ToolOutput(toolCall.Id, functionResult);
run = await run.SubmitToolOutputsAsync(toolOutput);
// waiting while run in Queued and InProgress
run = await run.WaitForStatusChangeAsync();
var messages = await run.ListMessagesAsync();
foreach (var message in messages.Items.OrderBy(response => response.CreatedAt))
{
Debug.Log($"{message.Role}: {message.PrintContent()}");
}
Returns a list of run steps belonging to a run.
var api = new OpenAIClient();
var runStepList = await api.ThreadsEndpoint.ListRunStepsAsync("thread-id", "run-id");
// OR use extension method for convenience!
var runStepList = await run.ListRunStepsAsync();
foreach (var runStep in runStepList.Items)
{
Debug.Log($"[{runStep.Id}] {runStep.Status} {runStep.CreatedAt} -> {runStep.ExpiresAt}");
}
Retrieves a run step.
var api = new OpenAIClient();
var runStep = await api.ThreadsEndpoint.RetrieveRunStepAsync("thread-id", "run-id", "step-id");
// OR use extension method for convenience!
var runStep = await run.RetrieveRunStepAsync("step-id");
var runStep = await runStep.UpdateAsync();
Debug.Log($"[{runStep.Id}] {runStep.Status} {runStep.CreatedAt} -> {runStep.ExpiresAt}");
Cancels a run that is in_progress
.
var api = new OpenAIClient();
var isCancelled = await api.ThreadsEndpoint.CancelRunAsync("thread-id", "run-id");
// OR use extension method for convenience!
var isCancelled = await run.CancelAsync();
Assert.IsTrue(isCancelled);
Given a chat conversation, the model will return a chat completion response.
The Chat API is accessed via OpenAIClient.ChatEndpoint
Creates a completion for the chat message
var api = new OpenAIClient();
var messages = new List<Message>
{
new Message(Role.System, "You are a helpful assistant."),
new Message(Role.User, "Who won the world series in 2020?"),
new Message(Role.Assistant, "The Los Angeles Dodgers won the World Series in 2020."),
new Message(Role.User, "Where was it played?"),
};
var chatRequest = new ChatRequest(messages, Model.GPT4);
var response = await api.ChatEndpoint.GetCompletionAsync(chatRequest);
var choice = response.FirstChoice;
Debug.Log($"[{choice.Index}] {choice.Message.Role}: {choice.Message} | Finish Reason: {choice.FinishReason}");
var api = new OpenAIClient();
var messages = new List<Message>
{
new Message(Role.System, "You are a helpful assistant."),
new Message(Role.User, "Who won the world series in 2020?"),
new Message(Role.Assistant, "The Los Angeles Dodgers won the World Series in 2020."),
new Message(Role.User, "Where was it played?"),
};
var chatRequest = new ChatRequest(messages);
var response = await api.ChatEndpoint.StreamCompletionAsync(chatRequest, partialResponse =>
{
Console.Write(partialResponse.FirstChoice.Delta.ToString());
});
var choice = response.FirstChoice;
Debug.Log($"[{choice.Index}] {choice.Message.Role}: {choice.Message} | Finish Reason: {choice.FinishReason}");
Only available with the latest 0613 model series!
var api = new OpenAIClient();
var messages = new List<Message>
{
new Message(Role.System, "You are a helpful weather assistant."),
new Message(Role.User, "What's the weather like today?"),
};
foreach (var message in messages)
{
Debug.Log($"{message.Role}: {message}");
}
// Define the tools that the assistant is able to use:
var tools = new List<Tool>
{
new Function(
nameof(WeatherService.GetCurrentWeather),
"Get the current weather in a given location",
new JObject
{
["type"] = "object",
["properties"] = new JObject
{
["location"] = new JObject
{
["type"] = "string",
["description"] = "The city and state, e.g. San Francisco, CA"
},
["unit"] = new JObject
{
["type"] = "string",
["enum"] = new JArray {"celsius", "fahrenheit"}
}
},
["required"] = new JArray { "location", "unit" }
})
};
var chatRequest = new ChatRequest(messages, tools: tools, toolChoice: "auto");
var response = await api.ChatEndpoint.GetCompletionAsync(chatRequest);
messages.Add(response.FirstChoice.Message);
Debug.Log($"{response.FirstChoice.Message.Role}: {response.FirstChoice} | Finish Reason: {response.FirstChoice.FinishReason}");
var locationMessage = new Message(Role.User, "I'm in Glasgow, Scotland");
messages.Add(locationMessage);
Debug.Log($"{locationMessage.Role}: {locationMessage.Content}");
chatRequest = new ChatRequest(messages, tools: tools, toolChoice: "auto");
response = await api.ChatEndpoint.GetCompletionAsync(chatRequest);
messages.Add(response.FirstChoice.Message);
if (!string.IsNullOrEmpty(response.ToString()))
{
Debug.Log($"{response.FirstChoice.Message.Role}: {response.FirstChoice} | Finish Reason: {response.FirstChoice.FinishReason}");
var unitMessage = new Message(Role.User, "celsius");
messages.Add(unitMessage);
Debug.Log($"{unitMessage.Role}: {unitMessage.Content}");
chatRequest = new ChatRequest(messages, tools: tools, toolChoice: "auto");
response = await api.ChatEndpoint.GetCompletionAsync(chatRequest);
}
var usedTool = response.FirstChoice.Message.ToolCalls[0];
Debug.Log($"{response.FirstChoice.Message.Role}: {usedTool.Function.Name} | Finish Reason: {response.FirstChoice.FinishReason}");
Debug.Log($"{usedTool.Function.Arguments}");
var functionArgs = JsonConvert.DeserializeObject<WeatherArgs>(usedTool.Function.Arguments.ToString());
var functionResult = WeatherService.GetCurrentWeather(functionArgs);
messages.Add(new Message(usedTool, functionResult));
Debug.Log($"{Role.Tool}: {functionResult}");
// System: You are a helpful weather assistant.
// User: What's the weather like today?
// Assistant: Sure, may I know your current location? | Finish Reason: stop
// User: I'm in Glasgow, Scotland
// Assistant: GetCurrentWeather | Finish Reason: tool_calls
// {
// "location": "Glasgow, Scotland",
// "unit": "celsius"
// }
// Tool: The current weather in Glasgow, Scotland is 20 celsius
⚠️ Beta Feature Currently, GPT-4 with vision does not support themessage.name
parameter, functions/tools, nor theresponse_format
parameter.
var api = new OpenAIClient();
var messages = new List<Message>
{
new Message(Role.System, "You are a helpful assistant."),
new Message(Role.User, new List<Content>
{
"What's in this image?",
new ImageUrl("https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg", ImageDetail.Low)
})
};
var chatRequest = new ChatRequest(messages, model: "gpt-4-vision-preview");
var response = await api.ChatEndpoint.GetCompletionAsync(chatRequest);
Debug.Log($"{response.FirstChoice.Message.Role}: {response.FirstChoice.Message.Content} | Finish Reason: {response.FirstChoice.FinishDetails}");
You can even pass in a Texture2D
!
var api = new OpenAIClient();
var messages = new List<Message>
{
new Message(Role.System, "You are a helpful assistant."),
new Message(Role.User, new List<Content>
{
"What's in this image?",
texture
})
};
var chatRequest = new ChatRequest(messages, model: "gpt-4-vision-preview");
var result = await apiChatEndpoint.GetCompletionAsync(chatRequest);
Debug.Log($"{result.FirstChoice.Message.Role}: {result.FirstChoice} | Finish Reason: {result.FirstChoice.FinishDetails}");
⚠️ Beta Feature
Important notes:
- When using JSON mode, always instruct the model to produce JSON via some message in the conversation, for example via your system message. If you don't include an explicit instruction to generate JSON, the model may generate an unending stream of whitespace and the request may run continually until it reaches the token limit. To help ensure you don't forget, the API will throw an error if the string "JSON" does not appear somewhere in the context.
- The JSON in the message the model returns may be partial (i.e. cut off) if
finish_reason
is length, which indicates the generation exceeded max_tokens or the conversation exceeded the token limit. To guard against this, checkfinish_reason
before parsing the response. - JSON mode will not guarantee the output matches any specific schema, only that it is valid and parses without errors.
var messages = new List<Message>
{
new Message(Role.System, "You are a helpful assistant designed to output JSON."),
new Message(Role.User, "Who won the world series in 2020?"),
};
var chatRequest = new ChatRequest(messages, "gpt-4-1106-preview", responseFormat: ChatResponseFormat.Json);
var response = await api.ChatEndpoint.GetCompletionAsync(chatRequest);
foreach (var choice in response.Choices)
{
Debug.Log($"[{choice.Index}] {choice.Message.Role}: {choice} | Finish Reason: {choice.FinishReason}");
}
response.GetUsage();
Converts audio into text.
The Audio API is accessed via OpenAIClient.AudioEndpoint
Generates audio from the input text.
var api = new OpenAIClient();
var request = new SpeechRequest("Hello world!");
var (path, clip) = await api.AudioEndpoint.CreateSpeechAsync(request);
audioSource.PlayOneShot(clip);
Transcribes audio into the input language.
var api = new OpenAIClient();
var request = new AudioTranscriptionRequest(audioClip, language: "en");
var result = await api.AudioEndpoint.CreateTranscriptionAsync(request);
Debug.Log(result);
Translates audio into into English.
var api = new OpenAIClient();
var request = new AudioTranslationRequest(audioClip);
var result = await api.AudioEndpoint.CreateTranslationAsync(request);
Debug.Log(result);
Given a prompt and/or an input image, the model will generate a new image.
The Images API is accessed via OpenAIClient.ImagesEndpoint
Creates an image given a prompt.
var api = new OpenAIClient();
var request = new ImageGenerationRequest("A house riding a velociraptor", Models.Model.DallE_3);
var imageResults = await api.ImagesEndPoint.GenerateImageAsync(request);
foreach (var result in imageResults)
{
Debug.Log(result.ToString());
Assert.IsNotNull(result.Texture);
}
Creates an edited or extended image given an original image and a prompt.
var api = new OpenAIClient();
var request = new ImageEditRequest(Path.GetFullPath(imageAssetPath), Path.GetFullPath(maskAssetPath), "A sunlit indoor lounge area with a pool containing a flamingo", size: ImageSize.Small);
var imageResults = await api.ImagesEndPoint.CreateImageEditAsync(request);
foreach (var result in imageResults)
{
Debug.Log(result.ToString());
Assert.IsNotNull(result.Texture);
}
Creates a variation of a given image.
var api = new OpenAIClient();
var request = new ImageVariationRequest(imageTexture, size: ImageSize.Small);
var imageResults = await api.ImagesEndPoint.CreateImageVariationAsync(request);
foreach (var result in imageResults)
{
Debug.Log(result.ToString());
Assert.IsNotNull(result.Texture);
}
Alternatively, the endpoint can directly take a Texture2D with Read/Write enabled and Compression set to None.
var api = new OpenAIClient();
var request = new ImageVariationRequest(imageTexture, size: ImageSize.Small);
var imageResults = await api.ImagesEndPoint.CreateImageVariationAsync(request);
foreach (var result in imageResults)
{
Debug.Log(result.ToString());
Assert.IsNotNull(result.Texture);
}
Files are used to upload documents that can be used with features like Fine-tuning.
The Files API is accessed via OpenAIClient.FilesEndpoint
Returns a list of files that belong to the user's organization.
var api = new OpenAIClient();
var fileList = await api.FilesEndpoint.ListFilesAsync();
foreach (var file in fileList)
{
Debug.Log($"{file.Id} -> {file.Object}: {file.FileName} | {file.Size} bytes");
}
Upload a file that can be used across various endpoints. The size of all the files uploaded by one organization can be up to 100 GB.
The size of individual files can be a maximum of 512 MB. See the Assistants Tools guide to learn more about the types of files supported. The Fine-tuning API only supports .jsonl files.
var api = new OpenAIClient();
var fileData = await api.FilesEndpoint.UploadFileAsync("path/to/your/file.jsonl", "fine-tune");
Debug.Log(fileData.Id);
Delete a file.
var api = new OpenAIClient();
var isDeleted = await api.FilesEndpoint.DeleteFileAsync(fileId);
Assert.IsTrue(isDeleted);
Returns information about a specific file.
var api = new OpenAIClient();
var file = await GetFileInfoAsync(fileId);
Debug.Log($"{file.Id} -> {file.Object}: {file.FileName} | {file.Size} bytes");
Downloads the file content to the specified directory.
var api = new OpenAIClient();
var downloadedFilePath = await api.FilesEndpoint.DownloadFileAsync(fileId);
Debug.Log(downloadedFilePath);
Assert.IsTrue(File.Exists(downloadedFilePath));
Manage fine-tuning jobs to tailor a model to your specific training data.
Related guide: Fine-tune models
The Files API is accessed via OpenAIClient.FineTuningEndpoint
Creates a job that fine-tunes a specified model from a given dataset.
Response includes details of the enqueued job including job status and the name of the fine-tuned models once complete.
var api = new OpenAIClient();
var fileId = "file-abc123";
var request = new CreateFineTuneRequest(fileId);
var job = await api.FineTuningEndpoint.CreateJobAsync(Model.GPT3_5_Turbo, request);
Debug.Log($"Started {job.Id} | Status: {job.Status}");
List your organization's fine-tuning jobs.
var api = new OpenAIClient();
var jobList = await api.FineTuningEndpoint.ListJobsAsync();
foreach (var job in jobList.Items.OrderByDescending(job => job.CreatedAt)))
{
Debug.Log($"{job.Id} -> {job.CreatedAt} | {job.Status}");
}
Gets info about the fine-tune job.
var api = new OpenAIClient();
var job = await api.FineTuningEndpoint.GetJobInfoAsync(fineTuneJob);
Debug.Log($"{job.Id} -> {job.CreatedAt} | {job.Status}");
Immediately cancel a fine-tune job.
var api = new OpenAIClient();
var isCancelled = await api.FineTuningEndpoint.CancelFineTuneJobAsync(fineTuneJob);
Assert.IsTrue(isCancelled);
Get status updates for a fine-tuning job.
var api = new OpenAIClient();
var eventList = await api.FineTuningEndpoint.ListJobEventsAsync(fineTuneJob);
Debug.Log($"{fineTuneJob.Id} -> status: {fineTuneJob.Status} | event count: {eventList.Events.Count}");
foreach (var @event in eventList.Items.OrderByDescending(@event => @event.CreatedAt))
{
Debug.Log($" {@event.CreatedAt} [{@event.Level}] {@event.Message}");
}
Get a vector representation of a given input that can be easily consumed by machine learning models and algorithms.
Related guide: Embeddings
The Edits API is accessed via OpenAIClient.EmbeddingsEndpoint
Creates an embedding vector representing the input text.
var api = new OpenAIClient();
var response = await api.EmbeddingsEndpoint.CreateEmbeddingAsync("The food was delicious and the waiter...", Models.Embedding_Ada_002);
Debug.Log(response);
Given a input text, outputs if the model classifies it as violating OpenAI's content policy.
Related guide: Moderations
The Moderations API can be accessed via OpenAIClient.ModerationsEndpoint
Classifies if text violates OpenAI's Content Policy.
var api = new OpenAIClient();
var isViolation = await api.ModerationsEndpoint.GetModerationAsync("I want to kill them.");
Assert.IsTrue(isViolation);
Additionally you can also get the scores of a given input.
var api = new OpenAIClient();
var response = await api.ModerationsEndpoint.CreateModerationAsync(new ModerationsRequest("I love you"));
Assert.IsNotNull(response);
Debug.Log(response.Results?[0]?.Scores?.ToString());
⚠️ Deprecated, and soon to be removed.
Given a prompt, the model will return one or more predicted completions, and can also return the probabilities of alternative tokens at each position.
The Completions API is accessed via OpenAIClient.CompletionsEndpoint
var api = new OpenAIClient();
var response = await api.CompletionsEndpoint.CreateCompletionAsync("One Two Three One Two", temperature: 0.1, model: Model.Davinci);
Debug.Log(response);
To get the
CompletionResponse
(which is mostly metadata), use its implicit string operator to get the text if all you want is the completion choice.
⚠️ Deprecated, and soon to be removed.
Streaming allows you to get results are they are generated, which can help your application feel more responsive, especially on slow models like Davinci.
var api = new OpenAIClient();
await api.CompletionsEndpoint.StreamCompletionAsync(response =>
{
foreach (var choice in response.Completions)
{
Debug.Log(choice);
}
}, "My name is Roger and I am a principal software engineer at Salesforce. This is my resume:", maxTokens: 200, temperature: 0.5, presencePenalty: 0.1, frequencyPenalty: 0.1, model: Model.Davinci);
⚠️ Deprecated, and soon to be removed.
Given a prompt and an instruction, the model will return an edited version of the prompt.
The Edits API is accessed via OpenAIClient.EditsEndpoint
Creates a new edit for the provided input, instruction, and parameters using the provided input and instruction.
var api = new OpenAIClient();
var request = new EditRequest("What day of the wek is it?", "Fix the spelling mistakes");
var response = await api.EditsEndpoint.CreateEditAsync(request);
Debug.Log(response);