中文 | English
- Added LLM tools to send messages to Work_wechat, DingTalk, and Feishu, as well as external functions to call.
- Added OpenAI’s TTS functionality, which can achieve voice output.
comfyui is an extremely minimalist UI interface, primarily used for AI drawing and other workflows based on the SD model. This project aims to develop a complete set of nodes for LLM workflow construction based on comfyui. It allows users to quickly and conveniently build their own LLM workflows and easily integrate them into their existing SD workflows.The picture shows a workflow of LLM implementing multi-tool calling, for more workflows please refer to the workflow folder.
-
Building a Modular AI with ComfyUI×LLM: A Step-by-Step Tutorial (Super Easy!)
-
Teach you GPT-4o access to comfyui | Make workflow call another workflow | Make LLM a tool
-
If there are any issues with the plugin or you have other questions, feel free to join the QQ group: 931057213.
- You can right-click in the comfyui interface, select
llm
from the context menu, and you will find the nodes for this project. how to use nodes - Supports API integration or local large model integration. Modular implementation for tool invocation.When entering the base_url, please use a URL that ends with
/v1/
.You can use ollama to manage your model. Then, enterhttps://localhost:11434/v1/
for the base_url,ollama
for the api_key, and your model name for the model_name, such as: llama3. If the call fails with a 503 error, you can try turning off the proxy server. - Local knowledge base integration with RAG support.
- Ability to invoke code interpreters.
- Enables online queries, including Google search support.
- Implement conditional statements within ComfyUI to categorize user queries and provide targeted responses.
- Supports looping links for large models, allowing two large models to engage in debates.
- Attach any persona mask, customize prompt templates.
- Supports various tool invocations, including weather lookup, time lookup, knowledge base, code execution, web search, and single-page search.
- Use LLM as a tool node.
- Rapidly develop your own web applications using API + Streamlit.The picture below is an example of a drawing application.
- Added a dangerous omnipotent interpreter node that allows the large model to perform any task.
- It is recommended to use the
show_text
node under thefunction
submenu of the right-click menu as the display output for the LLM node. - Supported the visual features of GPT-4O!
- A new workflow intermediary has been added, which allows your workflow to call other workflows!
- Adapted to all models with an interface similar to OpenAI, such as: Tongyi Qianwen/QWEN, Zhigu Qingyan/GLM, DeepSeek, Kimi/Moonshot. Please fill in the base_url, api_key, and model_name of these models into the LLM node to call them.
- Added an LVM loader, now you can call the LVM model locally, support lava-llama-3-8b-v1_1-gguf model, other LVM models should theoretically run if they are GUFF format.The example workflow can be found here: start_with_LVM.json.
- I wrote a
fastapi.py
file, and if you run it directly, you’ll get an OpenAI interface onhttps://127.0.0.1:8817/v1/
. Any application that can call GPT can now invoke your comfyui workflow! I will create a tutorial to demonstrate the details on how to do this. - I’ve separated the LLM loader and the LLM chain, dividing the model loading and model configuration. This allows for sharing models across different LLM nodes!
- macOS and mps devices are now supported! Thanks to bigcat88 for their contribution!
Install using one of the following methods:
- Search for comfyui_LLM_party in the comfyui manager and install it with one click.
- Restart comfyui.
- Navigate to the
custom_nodes
subfolder under the ComfyUI root folder. - Clone this repository with
git clone https://github.com/heshengtao/comfyui_LLM_party.git
.
- Click
CODE
in the upper right corner. - Click
download zip
. - Unzip the downloaded package into the
custom_nodes
subfolder under the ComfyUI root folder.
- Navigate to the
comfyui_LLM_party
project folder. - Enter
pip install -r requirements.txt
in the terminal to deploy the third-party libraries required by the project into the comfyui environment. Please ensure you are installing within the comfyui environment and pay attention to anypip
errors in the terminal. - If you are using the comfyui launcher, you need to enter
path_in_launcher_configuration\python_embeded\python.exe path_in_launcher_configuration\python_embeded\Scripts\pip.exe install -r requirements.txt
in the terminal to install. Thepython_embeded
folder is usually at the same level as yourComfyUI
folder.
Configure the APIKEY using one of the following methods:
- Open the
config.ini
file in thecomfyui_LLM_party
project folder. - Enter your
openai_api_key
andbase_url
inconfig.ini
. - If you want to use the Google search tool, enter your
google_api_key
andcse_id
inconfig.ini
.
- Open the comfyui interface.
- Create a new Large Language Model (LLM) node and directly enter your
openai_api_key
andbase_url
in the node. - Create a new Google Search Tool (google_tool) node and directly enter your
google_api_key
andcse_id
in the node.
- More model adaptations, at least covering the API interfaces of mainstream large models and local calls of mainstream open-source models, as well as more LVM model adaptations. Currently, I have only adapted the visual function calls of GPT-4;
- More ways to build agents. The work I have completed in this area includes importing an LLM as a tool to another LLM, achieving radial construction of LLM workflows, and importing one workflow as a node into another workflow. I might develop some cooler functions in this area in the future.
- More automation features. In the future, I will introduce more nodes that automatically push images, text, videos, and audio to other applications, as well as listening nodes that implement automatic replies to mainstream social software and forums.
- More knowledge base management functions. The project already supports local file search and web search. In the future, I will introduce knowledge graph search and long-term memory search. This will allow agents to think logically about professional knowledge and always remember certain key information when conversing with users.
- More tools, more persona. This part is the easiest to do but also requires the most accumulation. I hope that in the future, this project can have as many custom nodes as comfyui, with a multitude of tools and persona.
This open-source project and its contents (hereinafter referred to as "Project") are provided for reference purposes only and do not imply any form of warranty, either expressed or implied. The contributors of the Project shall not be held responsible for the completeness, accuracy, reliability, or suitability of the Project. Any reliance you place on the Project is strictly at your own risk. In no event shall the contributors of the Project be liable for any indirect, special, or consequential damages or any damages whatsoever resulting from the use of the Project.