Dify Support 💙

• Add support for Dify💙 by @uezo in #309
• Add support for Dify TTS and STT by @uezo in #311

Other changes

• Fix bug where mic volume changes are not applied immediately by @uezo in #307
• Enhance camera functionality with manual still capture and sub-camera switching by @uezo in #308

Full Changelog: 0.7.4...0.7.5

👀 Enhanced Vision Capabilities

This update introduces autonomous vision input for Gemini and Claude, and adds vision input support for WebGL. Now, various AIs can offer richer conversational experiences with integrated vision input across different platforms.

• Support autonomous vision input for Gemini✨ #302
• Refactor Vision input and various related improvements #303
• Support autonomous vision input for Claude✹ #304
• Add vision input support for WebGL #305

Full Changelog: 0.7.3...0.7.4

👀 Support dynamic vision input for ChatGPT

By adding a SimpleCamera to the scene and including [vision:camera] in the response message, the system will autonomously capture images when visual input is required for a response.

• Add autonomous image input handling for ChatGPT #298

📦 Easy setup by modularized UI components

Microphone volume sliders and request input forms have been modularized. These can now be used immediately by simply adding the prefabs to the scene without any additional setup.

• Modularize UI components for easy scene addition in #300

🎙️ dB-based microphone volume

• Change volume measurement from amplitude to decibels #296
• Fix incorrect volume measurement bug #299

✨ Other changes

• Switch from function call to tool call for ChatGPT Function Calling #297
• Remove deprecated ChatGPT-related modules in #301

Full Changelog: v0.7.2...0.7.3

🖼️ Support vision

Set image to request message as payload for multimodal conversation.

• Support message with image in GPT-4V and Claude3 #286

🚀 Configuration-free demo

Just start without any configurations. Set API key (and others if you want) at runtime.

• Add configuration-free demo #288

Full Changelog: v0.7.1...v0.7.2

🥰😇 Change avatar on runtime

Support changing avatar on runtime. Use ModelController.SetAvatar to change the avatar to another on runtime.
If you want to try it on editor, set another avatar on the scene to the inspector of ModelController and push Change Avatar button(appears runtime only).

• Add support for changing avatar on runtime #280

🎙️ Better microphone management

Mute microphone when not listening for WakeWord or VoiceRequest.
Also added IsMuted property to DialogController so that you can control mute/unmute manually. Default is false.

• IsMuted == false: Microphone will be on when WakeWordListener or VoiceRequestProvider is listening.
• IsMuted == true: Microphone will be off.

You can change the value manually on inspector or in script.

• Mute microphone when not listening for WakeWord or VoiceRequest #281
• Mute microphone when avatar speaking #282

🍱 Other changes

• Set message windows automatically without configurations #278
• Prevent UnityWebRequest error when calling LLMs #279
• Update demo for v0.7.1 #283

Full Changelog: v0.7.0...v0.7.1

🤖 LLM-based Dialog Processing

✅Multiple LLMs: ChatGPT / Azure OpenAI Service, Anthropic Claude, Google Gemini Pro and others
✅Agents: Function Calling (ChatGPT / Gemini) or your prompt engineering
✅Multimodal: GPT-4V and Gemini-Pro-Vision are suppored
✅Emotions: Autonomous face expression and animation

We've developed a versatile framework that standardizes the processes of Routing, Chatting (including facial expressions and motions), and executing Tools using various Large Language Models (LLMs). This framework allows for easy customization by simply swapping out LLM-specific components.

Additionally, you can support any LLM by creating your own components that implement the ILLMService interface.

To use this new LLM-based dialog, attach LLMRouter, LLMContentSkill and ILLMService component like ChatGPTService in ChatdollKit.LLM package. Also attach function skills that extends LLMFunctionSkillBase if you build an AI agent. See DemoChatGPT that works right out-of-the-box🎁.

• Support multiple LLMs: ChatGPT, Claude and Gemini✨ #271
• Support custom request parameters and headers for LLM APIs #272
• Fix bug that iOS build failed on Xcode #273
• Fix bug that LLMFunctionSkill fails #275

NOTE: ChatdollKit.Dialog.Processor.ChatGPT* components are deprecated.

🐉 Other Changes

• Refactoring for Future tech integration and reduced dependencies #270
• Improve stability of AzureStreamVoiceRequestProvider #274
• Make microphone volume controller works with NonRecordingVRP #276
• Update demo for v0.7.0 #277

Full Changelog: v0.6.6...v0.7.0

🐘 Improve stability

Previously, ChatGPT (OpenAI API) sometimes does not return a response, which prevents the character from continuing the conversation.😔
However, in this update, the system will automatically retry in the absence of a response, allowing the conversation to continue.

• Retry when no response from ChatGPT #263

💃 Autonomous animation

Set animations before starting conversation. (e.g. at Start())

chatGPTContentSkill.RegisterAnimation("waving_arm", new Model.Animation("BaseParam", 10, 3.0f));
chatGPTContentSkill.RegisterAnimation("look_away", new Model.Animation("BaseParam", 6, 3.0f, "AGIA_Layer_look_away_01", "Additive Layer"));

List them in the prompt and provide a guide on how to use them.

* You can express your emotions through the following animations:

- waving_arm
- look_away

* If you want to express emotions with gestures, insert the animation into the response message like [anim:waving_arm].

• Support autonomous animation💃 #264

😴 Multiple idling mode

You can have multiple idling mode. The default is normal. Add animations and face expressions if you want to add another mode like below:

modelController.AddIdleAnimation(new Model.Animation("BaseParam", 101, 5f), mode: "sleep");
modelController.AddIdleFace("sleep", "Blink");

Switch using ChangeIdlingModeAsync.

modelController.ChangeIdlingModeAsync("sleep");

The animation for sleep starts immediately.

• Support switching idling mode #267

🍩 Other updates

• Make it easier to set microphone device name #265
• Small changes for v0.6.6 #266
• Update demo #268

Full Changelog: v0.6.5...v0.6.6

🌊 Azure OpenAI Service

To use Azure OpenAI Service set following info on inspector of ChatGPTService component:

1. Endpoint url with configurations to Chat Completion Url

format: https://{your-resource-name}.openai.azure.com/openai/deployments/{deployment-id}/chat/completions?api-version={api-version}

2. API Key to Api Key

3. Set true to Is Azure

NOTE: Model on inspector is ignored. Engine in url is used.

• Add support for Azure OpenAI Service #259

🗣️ OpenAI Speech Services

Support multiple languages and allow you to switch between them without changing configurations.

NOTE: Text-to-Speech doesn't support WebGL.

• Add support for OpenAI Speech Services #260

🚰 Real-time stream speech recognition

Use AzureStreamVoiceRequestProvider to perform real-time speech recognition.
This component depends on Azure Speech SDK.

NOTE: Microphone volume slider doesn't support controlling this component.

• Add support for stream speech recognition #261

☺️ Other small changes

• Small changes for v0.6.5 #262

Full Changelog: v0.6.4...v0.6.5

🥰 Make Unagirl more kawaii

• Make Unagirl (3D model for demo) more kawaii #256

By applying eye makeup, I enhanced the attractiveness of her face.🪄

• Enhanced the double eyelids.
• Increased the volume of the lower eyelashes.

🐕 Support Retrieval Augmented Generation

• Make it possible to use request, state and user info in ChatGPTFunctionSkills #257
• Update demo and example by @uezo in #258

You can create RAG-based AI agent just by adding ChatGPT function skill. See RetrievalQASkill example that can answer based on OpenAI terms of use.

Full Changelog: v0.6.3...v0.6.4

Feature updates⚡️

Minor updates but they are necessary for creating apps.

• Make it possible to set role to the next message after function calling #247
• Add support for muting microphone #248
• Add character message window #249
• Support favoring specific phrases on Google Text-to-Speech #250

Unagirl, a new demo character debuts🥰

• Unagirl debuts as a new demo character🥰 #255

See Unagirl in dress at vroid hub https://hub.vroid.com/characters/7204218679890800631/models/6961078537805052952

Other updates and fixes🐈

• Improve HTTP error debugging #251
• Use UniTask in HTTP request #252
• Fix VoiceRecorder fails in starting #253
• Fix warning message when no skill router is attached #254

Full Changelog: v0.6.2...v0.6.3