comfyui_LLM_party

C‌‌​‎​‎‏​‍‎​‎​‎‏​‌‎​‎‍​‍‏​‍‌​‌‏omfyui_llm_party aims to develop a complete set of nodes for LLM workflow construction based on comfyui as the front end. It allows users to quickly and conveniently build their own LLM workflows and easily integrate them into their existing image workflows.

Effect display

https://github.com/user-attachments/assets/945493c0-92b3-4244-ba8f-0c4b2ad4eba6

Project Overview

ComfyUI LLM Party, from the most basic LLM multi-tool call, role setting to quickly build your own exclusive AI assistant, to the industry-specific word vector RAG and GraphRAG to localize the management of the industry knowledge base; from a single agent pipeline, to the construction of complex agent-agent radial interaction mode and ring interaction mode; from the access to their own social APP (QQ, Feishu, Discord) required by individual users, to the one-stop LLM + TTS + ComfyUI workflow required by streaming media workers; from the simple start of the first LLM application required by ordinary students, to the various parameter debugging interfaces commonly used by scientific researchers, model adaptation. All of this, you can find the answer in ComfyUI LLM Party.

Quick Start

0. If you have never used ComfyUI and encounter some dependency issues while installing the LLM party in ComfyUI, please click here to download the Windows portable package that includes the LLM party. Please note that this portable package contains only the party and manager plugins, and is exclusively compatible with the Windows operating system.(If you need to install LLM party into an existing comfyui, this step can be skipped.)
1. Drag the following workflows into your comfyui, then use comfyui-Manager to install the missing nodes.

• Use API to call LLM: start_with_LLM_api
• Using aisuite to call LLM: start_with_aisuite
• Manage local LLM with ollama: start_with_Ollama
• Use local LLM in distributed format: start_with_LLM_local
• Use local LLM in GGUF format: start_with_LLM_GGUF
• Use local VLM in distributed format: start_with_VLM_local (Currently, support is extended for Llama-3.2-Vision/Qwen/Qwen2.5-VL/deepseek-ai/Janus-Pro.)
• Use local VLM in GGUF format: start_with_VLM_GGUF
• Utilize API calls to LLM for generating SD prompts and images: start_with_VLM_API_for_SD
• Employ ollama to call minicpm for generating SD prompts and images: start_with_ollama_minicpm_for_SD
• Utilize the local qwen-vl to generate SD prompts and images: start_with_qwen_vl_local_for_SD

2. If you are using API, fill in your base_url (it can be a relay API, make sure it ends with /v1/), for example: https://api.openai.com/v1/ and api_key in the API LLM loader node.
3. If you are using ollama, turn on the is_ollama option in the API LLM loader node, no need to fill in base_url and api_key.
4. If you are using a local model, fill in your model path in the local model loader node, for example: E:\model\Llama-3.2-1B-Instruct. You can also fill in the Huggingface model repo id in the local model loader node, for example: lllyasviel/omost-llama-3-8b-4bits.
5. Due to the high usage threshold of this project, even if you choose the quick start, I hope you can patiently read through the project homepage.

Latest update

1. The LLM API node has now implemented a streaming output mode, which will display the text returned by the API in real-time on the console, allowing you to see the API's output live without waiting for the entire request to complete.
2. The LLM API node has added a reasoning_content output, which can automatically separate the reasoning and response of the R1 model.
3. A new branch named only_api has been added to the repository, containing only the API calling components. This is designed for users who require only API invocation. To use this branch, simply execute the command git clone -b only_api https://github.com/heshengtao/comfyui_LLM_party.git in the custom node folder of comfyui, and then follow the environment deployment instructions provided on the project's main page. Please note! It is essential to ensure that there are no other folders named comfyui_LLM_party within the custom node folder.
4. The VLM local loader node now supports deepseek-ai/Janus-Pro, with an example workflow: Janus-Pro.
5. The VLM local loader node has already supported Qwen/Qwen2.5-VL-3B-Instruct, but you need to update the transformer to the latest version (pip install -U transformers), example workflow: qwen-vl
6. A brand new image hosting node has been added, currently supporting the image hosting services at https://sm.ms (with the regional domain for China being https://smms.app) and https://imgbb.com. More image hosting services will be supported in the future. Sample workflow: Image Hosting
7. The imgbb image hosting service, which is compatible by default with the party, has been updated to the domain imgbb. The previous image hosting service was replaced due to its unfriendliness towards users in mainland China. I sincerely apologize, as it seems that the API service for the image hosting at https://imgbb.io has been discontinued. Therefore, the code has reverted to the original https://imgbb.com. Thank you for your understanding. In the future, I will update a node that supports more image hosting services.
8. The MCP tool has been updated. You can modify the configuration in the 'mcp_config.json' file located in the party project folder to connect to your desired MCP server. You can find various MCP server configuration parameters that you may want to add here: modelcontextprotocol/servers. The default configuration for this project is the Everything server, which serves as a testing MCP server to verify its functionality. Reference workflow: start_with_MCP. Developer note: The MCP tool node can connect to the MCP server you have configured and convert the tools from the server into tools that can be directly used by LLMs. By configuring different local or cloud servers, you can experience all LLM tools available in the world.

User Guide

1. For the instructions for using the node, please refer to: how to use nodes

2. If there are any issues with the plugin or you have other questions, feel free to join the QQ group: 931057213 | discord：discord.

3. More workflows please refer to the workflow folder.

Vedio tutorial

Model support

1. Support all API calls in openai format(Combined with oneapi can call almost all LLM APIs, also supports all transit APIs), base_url selection reference config.ini.example, which has been tested so far:

• openai (Perfectly compatible with all OpenAI models, including the 4o and o1 series!)
• ollama (Recommended! If you are calling locally, it is highly recommended to use the ollama method to host your local model!)
• Azure OpenAI
• llama.cpp (Recommended! If you want to use the local gguf format model, you can use the llama.cpp project's API to access this project!)
• Grok
• Tongyi Qianwen /qwen
• zhipu qingyan/glm
• deepseek
• kimi/moonshot
• doubao
• spark
• Gemini(The original Gemini API LLM loader node has been deprecated in the new version. Please use the LLM API loader node, with the base_url selected as: https://generativelanguage.googleapis.com/v1beta/openai/)

2. Support for all API calls compatible with aisuite:

• anthropic/claude
• aws
• vertex
• huggingface

3. Compatible with most local models in the transformer library (the model type on the local LLM model chain node has been changed to LLM, VLM-GGUF, and LLM-GGUF, corresponding to directly loading LLM models, loading VLM models, and loading GGUF format LLM models). If your VLM or GGUF format LLM model reports an error, please download the latest version of llama-cpp-python from llama-cpp-python. Currently tested models include:

• ClosedCharacter/Peach-9B-8k-Roleplay(Recommended! Role-playing model)
• lllyasviel/omost-llama-3-8b-4bits(Recommended! Rich prompt model)
• meta-llama/llama-2-7b-chat-hf
• Qwen/Qwen2-7B-Instruct
• openbmb/MiniCPM-V-2_6-gguf
• lmstudio-community/Meta-Llama-3.1-8B-Instruct-GGUF
• meta-llama/Llama-3.2-11B-Vision-Instruct
• Qwen/Qwen2.5-VL-3B-Instruct
• deepseek-ai/Janus-Pro

4. Model download

• Quark cloud address
• Baidu cloud address, extraction code: qyhu

Download

• You can configure the language in config.ini, currently only Chinese (zh_CN) and English (en_US), the default is your system language.
• Install using one of the following methods:

Method 1:

1. Search for comfyui_LLM_party in the comfyui manager and install it with one click.
2. Restart comfyui.

Method 2:

1. Navigate to the custom_nodes subfolder under the ComfyUI root folder.
2. Clone this repository with git clone https://github.com/heshengtao/comfyui_LLM_party.git.

Method 3:

1. Click CODE in the upper right corner.
2. Click download zip.
3. Unzip the downloaded package into the custom_nodes subfolder under the ComfyUI root folder.

Environment Deployment

1. Navigate to the comfyui_LLM_party project folder.
2. Enter pip install -r requirements.txt in the terminal to deploy the third-party libraries required by the project into the comfyui environment. Please ensure you are installing within the comfyui environment and pay attention to any pip errors in the terminal.
3. If you are using the comfyui launcher, you need to enter path_in_launcher_configuration\python_embeded\python.exe -m pip install -r requirements.txt in the terminal to install. The python_embeded folder is usually at the same level as your ComfyUI folder.
4. If you have some environment configuration problems, you can try to use the dependencies in requirements_fixed.txt.

Configuration

• The language can be configured in config.ini, currently only Chinese (zh_CN) and English (en_US) are available, with the default set to your system language.
• In config.ini, you can configure whether to enable fast installation. The fast_installed option defaults to False, and if you do not require the usage of the GGUF model, it can be set to True.
• APIKEY can be configured using one of the following methods

Method 1:

1. Open the config.ini file in the project folder of the comfyui_LLM_party.
2. Enter your openai_api_key, base_url in config.ini.
3. If you are using an ollama model, fill in http://127.0.0.1:11434/v1/ in base_url, ollama in openai_api_key, and your model name in model_name, for example: llama3.
4. If you want to use Google search or Bing search tools, enter your google_api_key, cse_id or bing_api_key in config.ini.
5. If you want to use image input LLM, it is recommended to use image bed imgbb and enter your imgbb_api in config.ini.
6. Each model can be configured separately in the config.ini file, which can be filled in by referring to the config.ini.example file. After you configure it, just enter model_name on the node.

Method 2:

1. Open the comfyui interface.
2. Create a Large Language Model (LLM) node and enter your openai_api_key and base_url directly in the node.
3. If you use the ollama model, use LLM_api node, fill in http://127.0.0.1:11434/v1/ in base_url node, fill in ollama in api_key, and fill in your model name in model_name, for example: llama3.
4. If you want to use image input LLM, it is recommended to use graph bed imgbb and enter your imgbb_api_key on the node.

Changelog

Click here

Next Steps Plan:

1. More model adaptations;
2. More ways to build agents;
3. More automation features;
4. More knowledge base management features;
5. More tools, more personas.

Another useful open-source project of mine:

super-agent-party is a 3D AI desktop companion with limitless possibilities! With RAG, web search, long-term memory, Code Interpreter, MCP, A2A, Comfyui, QQ bot, and more features!

Disclaimer:

This open-source project and its contents (hereinafter referred to as "Project") are provided for reference purposes only and do not imply any form of warranty, either expressed or implied. The contributors of the Project shall not be held responsible for the completeness, accuracy, reliability, or suitability of the Project. Any reliance you place on the Project is strictly at your own risk. In no event shall the contributors of the Project be liable for any indirect, special, or consequential damages or any damages whatsoever resulting from the use of the Project.

Special thanks:

loan list

Some of the nodes in this project have borrowed from the following projects. Thank you for your contributions to the open-source community!

1. pythongosssss/ComfyUI-Custom-Scripts
2. lllyasviel/Omost

Support:

Follow us

1. If you want to continue to pay attention to the latest features of this project, please follow the Bilibili account: 派酱
2. youtube@comfyui-LLM-party

Star History