llm-server-docs

varunvasudeva1MIT更新放缓

在 Debian 上从零搭建完全本地私有的 LLM 服务器，集成聊天、联网搜索、RAG、模型管理、MCP、图像生成和语音合成。

End-to-end documentation to set up your own local & fully private LLM server on Debian. Equipped with chat, web search, RAG, model management, MCP servers, image generation, and TTS.

暂无版本849star62fork同步于 2026-11-07 07:28 UTC

本地部署私有LLM RAG 模型管理语音合成

对比 GitHub 官网 comfyui debian docker huggingface kokoro-fastapi

怎么上手

由 README 提炼，执行前请确认

原文

sudo apt update
sudo apt upgrade

硬件要求按 README 说明

README 说明任意现代 CPU 与 GPU 组合均可，AMD GPU 已被 Ollama 原生支持；参考配置为 2x Nvidia RTX 3090（24GB）、96GB DDR4、1TB NVMe。纯 CPU 可跳过 GPU 驱动与功耗限制步骤。

纯 CPU 可运行需要 CUDA有 Docker 镜像

下载：源码 zip·GitHub Releases（安装包）本站不镜像安装包

项目说明

由 README 自动翻译提炼

没人工点评？写下你的经验

在 Debian 上从零搭建完全本地的私有 LLM 服务器，含聊天、搜索、RAG、图像与语音

在 Debian 上从零搭建完全本地私有的 LLM 服务器，集成聊天、联网搜索、RAG、模型管理、MCP、图像生成和语音合成。

核心能力

用 Ollama/llama.cpp/vLLM 三选一作为推理引擎
用 llama-swap 加 systemd 管理模型服务
用 Open WebUI 提供类云厂商的聊天界面
用 SearXNG 接入联网搜索并集成到聊天
用 Tailscale 与 Caddy 做安全远程访问

容易踩的坑

需要先装好 Debian 系统并具备基础 Linux 终端操作能力；Nvidia 用户需按官方指引装 CUDA Toolkit 与驱动并重启验证 nvidia-smi；AMD GPU 需跳过 nvidia-smi 与功耗限制相关步骤。

内容由模型根据 README 生成，可能不完整或不准确，以官方文档为准。看原文 README

用户点评

登录用户的真实使用记录，可按版本和硬件筛选

共 0 条

还没有人点评。第一个说出真实体验的人最有价值。

README 原文

未翻译

在 GitHub 打开

Local LLaMA Server Setup Documentation

TL;DR: End-to-end documentation to set up your own local & fully private LLM server on Debian. Equipped with chat, web search, RAG, model management, MCP servers, image generation, and TTS, along with steps for configuring SSH, firewall, and secure remote access via Tailscale.

Software Stack:

Inference Engine (Ollama, llama.cpp, vLLM)
Search Engine (SearXNG)
Model Server (llama-swap, systemd service)
Chat Platform (Open WebUI)
MCP Proxy Server (mcp-proxy, MCPJungle)
Text-to-Speech Server (Kokoro FastAPI)
Image Generation Server (ComfyUI)
VPN/Remote Access (Tailscale)
Reverse Proxy (Caddy)
DNS Provider (Cloudflare)

Software Stack Architectural Diagram

Local LLaMA Server Setup Documentation

About

This repository outlines the steps to run a server for running local language models. It uses Debian specifically, but most Linux distros should follow a very similar process. It aims to be a guide for Linux beginners like me who are setting up a server for the first time.

The process involves installing the requisite drivers, setting the GPU power limit, setting up auto-login, and scheduling the init.bash script to run at boot. All these settings are based on my ideal setup for a language model server that runs most of the day but a lot can be customized to suit your needs.

[!IMPORTANT] No part of this guide was written using AI - any hallucinations are the good old human kind. AI is used to format Markdown elements and vet for any technical inaccuracies. While I've done my absolute best to ensure correctness in every step/command, check everything you execute in a terminal. Enjoy!

Priorities

Simplicity: It should be relatively straightforward to set up the components of the solution.
Stability: The components should be stable and capable of running for weeks at a time without any intervention necessary.
Security: The components should be able to be tightly secured and limited in their capability to damage the system in case of a known vulnerability affecting any of the components.
Maintainability: The components and their interactions should be uncomplicated enough that you know enough to maintain them as they evolve (because they will evolve).
Aesthetics: The result should be as close to a cloud provider's chat platform as possible. A homelab solution doesn't necessarily need to feel like it was cobbled together haphazardly.
Modularity: Components in the setup should be able to be swapped out for newer/more performant/better maintained alternatives easily. Standard protocols (OpenAI-compatibility, MCPs, etc.) help with this a lot and, in this guide, they are always preferred over bundled solutions.
Open source: The code should be able to be verified by a community of engineers. Chat platforms and LLMs involve large amounts of personal data conveyed in natural language and it's important to know that data isn't going outside your machine.

Prerequisites

Any modern CPU and GPU combination should work for this guide. Previously, compatibility with AMD GPUs was an issue but the latest releases of Ollama have worked through this and AMD GPUs are now supported natively.

For reference, this guide was built around the following system:

CPU: Intel Core i5-12600KF
Memory: 96GB 3200MHz DDR4 RAM
Storage: 1TB M.2 NVMe SSD
GPU: 2x Nvidia RTX 3090 (24GB)

[!NOTE] AMD GPUs: Power limiting is skipped for AMD GPUs as AMD has recently made it difficult to set power limits on their GPUs. Naturally, skip any steps involving nvidia-smi or nvidia-persistenced and the power limit in the init.bash script.

CPU-only: You can skip the GPU driver installation and power limiting steps. The rest of the guide should work as expected.

[!NOTE] This guide uses ~/ (or /home/<username>) as the base directory. If you're working in different directory, please modify all your commands accordingly.

To begin the process of setting up your server, you will need the following:

Fresh install of Debian
Internet connection
Basic understanding of the Linux terminal
Peripherals like a monitor, keyboard, and mouse

To install Debian on your newly built server hardware:

Download the Debian ISO from the official website.
Create a bootable USB using a tool like Rufus for Windows or Balena Etcher for MacOS.
Boot into the USB and install Debian.

For a more detailed guide on installing Debian, refer to the official documentation. For those who aren't yet experienced with Linux, I recommend using the graphical installer - you will be given an option between the text-based installer and graphical installer.

I also recommend installing a lightweight desktop environment like XFCE for ease of use. Other options like GNOME or KDE are also available - GNOME may be a better option for those using their server as a primary workstation as it is more feature-rich (and, as such, heavier) than XFCE.

General

Allow `sudo` Permissions

To do a bunch of things in this guide, we need to be root. But we don't have the ability to act as root until root allows our user to. First, we'll switch to root and grant our user permission to run commands with sudo.

[!TIP] sudo stands for "superuser doer" - in Linux, it signals to the OS that you want the command you're running to be running on behalf of the root user. This should be used sparingly on a highly secure system (we want user-specific permissions for most processes) and carefully since it can affect the system with the same power as the root user. Don't overthink its use or worry when using it - just know that it can be dangerous if used incorrectly.

Switch to root:
```
su root
```
Run the following command to add your user to the sudo group (which has the permissions we're looking for):
```
sudo usermod -a -G sudo <username>
```
Replace <username> with your username.

Save and exit (Ctrl+X).
Close your existing terminal and open a new session. This is required to see the changes.
(Optional) Test your new permissions by running ls (command to view the contents of a directory) with sudo:
```
sudo ls
```

Update System Packages

Update the system by running the following commands:
```
sudo apt update
sudo apt upgrade
```

Now, we'll install the required GPU drivers that allow programs to utilize their compute capabilities.

Nvidia GPUs

Follow Nvidia's guide on downloading CUDA Toolkit. The instructions are specific to your machine and the website will lead you to them interactively.

Run the following commands:

sudo apt install linux-headers-amd64
sudo apt install nvidia-driver firmware-misc-nonfree

Reboot the server.
Run the following command to verify the installation:
```
nvidia-smi
```

AMD GPUs

Run the following commands:

deb http://deb.debian.org/debian bookworm main contrib non-free-firmware
apt install firmware-amd-graphics libgl1-mesa-dri libglx-mesa0 mesa-vulkan-drivers xserver-xorg-video-all

Reboot the server.

We'll also install some packages that are not installed on Debian by default but may be required later:

sudo apt install libcurl cmake build-essential

Schedule Startup Script

In this step, we'll create a script called init.bash. This script will be run at boot to set the GPU power limit and start the server using Ollama. We set the GPU power limit lower because it has been seen in testing and inference that there is only a 5-15% performance decrease for a 30% reduction in power consumption. This is especially important for servers that are running 24/7.

Run the following commands:
```
touch init.bash
nano init.bash
```
Add the following lines to the script:
```
#!/bin/bash
sudo nvidia-smi -pm 1
sudo nvidia-smi -pl <power_limit>
```
Replace <power_limit> with the desired power limit in watts. For example, sudo nvidia-smi -pl 250.

For multiple GPUs, modify the script to set the power limit for each GPU:
```
sudo nvidia-smi -i 0 -pl <power_limit>
sudo nvidia-smi -i 1 -pl <power_limit>
```
Save and exit the script.
Make the script executable:
```
chmod +x init.bash
```

Adding the init.bash script to the crontab will schedule it to run at boot.

Run the following command:
```
crontab -e
```
Add the following line to the file:
```
@reboot /path/to/init.bash
```
Replace /path/to/init.bash with the path to the init.bash script.
(Optional) Add the following line to shutdown the server at 12am:
```
0 0 * * * /sbin/shutdown -h now
```
Save and exit the file.

Configure Script Permissions

We want init.bash to run the nvidia-smi commands without having to enter a password. This is done by giving nvidia-persistenced and nvidia-smi passwordless sudo permissions, and can be achieved by editing the sudoers file.

AMD users can skip this step as power limiting is not supported on AMD GPUs.

Run the following command to edit the sudoers file:
```
sudo visudo
```

Add the following lines to the file:

<username> ALL=(ALL) NOPASSWD: /usr/bin/nvidia-persistenced
<username> ALL=(ALL) NOPASSWD: /usr/bin/nvidia-smi

Replace <username> with your username.

Save and exit the file.

[!IMPORTANT] Ensure that you add these lines AFTER %sudo ALL=(ALL:ALL) ALL. The order of the lines in the file matters - the last matching line will be used so if you add these lines before %sudo ALL=(ALL:ALL) ALL, they will be ignored.

Configure Auto-Login (optional)

When the server boots up, we may want it to automatically log in to a user account and run the init.bash script. This is done by configuring the lightdm display manager.

Run the following command:
```
sudo nano /etc/lightdm/lightdm.conf
```
Find the following commented line. It should be in the [Seat:*] section.
```
# autologin-user=
```
Uncomment the line and add your username:
```
autologin-user=<username>
```
Replace <username> with your username.
Save and exit the file.

Docker

📖 Documentation

Docker is a containerization platform that allows you to run applications in isolated environments. This subsection follows Docker's guide to install Docker Engine on Debian. The commands are listed below, but visiting the guide is recommended in case instructions have changed.

If you already have a Docker installation on your system, it's a good idea to re-install so there are no broken/out-of-date dependencies. The command below will iterate through your system's installed packages and remove the ones associated with Docker.
```
for pkg in docker.io docker-doc docker-compose podman-docker containerd runc; do sudo apt purge $pkg; done
```

Run the following commands:

# Add Docker's official GPG key:
sudo apt update
sudo apt install ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

# Add the repository to Apt sources:
sudo tee /etc/apt/sources.list.d/docker.sources <<EOF
Types: deb
URIs: https://download.docker.com/linux/debian
Suites: $(. /etc/os-release && echo "$VERSION_CODENAME")
Components: stable
Signed-By: /etc/apt/keyrings/docker.asc
EOF

sudo apt update

Install the Docker packages:

sudo apt install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

Verify the installation (optional):
```
sudo systemctl status docker
```
If it hasn't started, try manually starting the daemon:
```
sudo systemctl start docker
```

Add User to Docker Group

So that we can forgo the use of sudo for Docker commands and run them directly under our user, we will add our user to the docker group. This isn't strictly necessary but convenient. This sub-section uses Docker's own post-install documentation as a reference - have a look just in case things have changed since the time of writing.

[!CAUTION] Being in the docker group grants significant system access - effectively root-equivalent privileges to the Docker daemon. Only add trusted users to this group.

Add your user to the Docker group:
```
sudo usermod -aG docker $USER
```
Apply the changes:
```
newgrp docker
```
Verify your new permissions by checking running containers:
```
docker ps -a
```

If you encounter "permission denied" and Docker commands still require sudo, log out and log back in.

[!NOTE] If you decide not to do this, you will need to append sudo in front of any docker commands.

Nvidia Container Toolkit

You will most likely want to use GPUs via Docker - this will require Nvidia Container Toolkit, which allows Docker to allocate/de-allocate memory on Nvidia GPUs. The steps for installing this are listed below, but it is recommended to reference Nvidia's documentation for the most up-to-date commands.

Configure the repository

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
&& curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
    sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
    sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

Update packages:
```
sudo apt update
```

Install Nvidia Container Toolkit packages:

sudo apt install -y nvidia-container-toolkit

Restart the Docker daemon:
```
sudo systemctl restart docker
```

Create a Network

We'll be running most services via Docker containers. To allow multiple containers to communicate with each other, we can open up ports via UFW (which we'll configure later) but this is less optimal than creating a Docker network. This way, all containers on the network can securely talk to each other without needing to open ports for the many services we could run via UFW, inherently creating a more secure setup.

We'll call this network app-net: you can call it anything you like, just be sure to update the commands that use it later.

Run the following:

docker network create app-net

That's it! Now, when we create containers, we can reference it as follows:

Docker Run

docker run <container> --network app-net

Docker Compose

services:
  <container>:
    # add this
    networks:
      - app-net

# add this
networks:
  app-net:
    external: true

Replace <container> with the actual service - don't forget to add other parameters too.

With this configured, we can now call containers by name and port. Let's pretend we're calling the /health endpoint in llama-swap from open-webui (two actual containers we'll be creating later on) to ensure that the containers can see and speak to each other. Run (CTRL+C to quit):

docker exec -i open-webui curl http://llama-swap:8080/health

You could also run it the other way to be extra sure:

docker exec -it llama-swap curl http://open-webui:8080

[!IMPORTANT] The port is always the internal port the container is running on. If a container runs on 1111:8080, for example, 1111 is the port on the host (where you might access it, like http://localhost:1111 or http://<server_ip>:1111) and 8080 is the internal port the container is running on. Thus, trying to access the container on 1111 via app-net will not work. Remembering this when specifying URLs in services will save you a lot of unnecessary "why isn't this working?" pains.

Harden Docker Containers

Docker security is is critical for public-facing services. It involves reducing attack surfaces, eliminating terminal abilities for containers, and limiting resource usage. Even if you only intend to access your services via a private network (preferred if possible), shoring up security is never a bad idea because it limits how much damage can be done to your server and its services if it ever was to become compromised somehow.

📖 See the full guide at Docker Security.

[!TIP] The combined example at the end provides a ready-to-paste YAML block with all hardening directives. You can copy it directly into your existing Compose files and customize as needed.

Helpful Commands

In the process of setting up this server (or anywhere down the rabbit hole of setting up services), you're likely to use Docker often. For the uninitiated, here are helpful commands that will make navigating and troubleshooting containers easier:

See available/running containers: docker ps -a
Restart a container: docker restart <container_name>
View a container's logs (live): docker logs -f <container_name> (CTRL+C to quit)
Rename a container: docker rename <container_name> <new_container_name>
Sometimes, a single service will spin up multiple containers, e.g. xyz-server and xyz-db. To restart both simultaneously, run the following command from inside the directory containing the Compose file: docker compose restart
Recreate a service: docker compose down && docker compose up -d
Test a container's configuration: docker compose config --quiet
View a container's resolved configuration: docker compose config
List Docker networks: docker network ls

[!TIP] There are no set rules when it comes to how you set up your Docker containers/services. However, here are my two cents:
It's cleanest to use Docker Compose (docker compose up -d with a docker-compose.yaml file as opposed to docker run -d <image_name>). Unless you take copious notes on your homelab and its setup, this method is almost self-documenting and keeps a neat trail of the services you run via their compose files. One compose file per directory is standard.

[!TIP] Occasionally, restarting a container is not enough to persist changes to configurations. If ever you find that changes you made to a service are not showing up, recreate the service altogether first before continuing to other troubleshooting.

HuggingFace CLI

📖 Documentation

HuggingFace is the leading open-source ML/AI platform - it hosts models (including LLMs), datasets, and demo apps that can be used to test models. For the purpose of this guide, we'll be using HuggingFace to download popular open-source LLMs.

[!NOTE] Only needed for llama.cpp/vLLM.

Create a new virtual environment:

python3 -m venv hf-env
source hf-env/bin/activate

Download the huggingface_hub package using pip:
```
pip install -U "huggingface_hub[cli]"
```
Create an authentication token on https://huggingface.com
Log in to HF Hub:
```
hf auth login
```
Enter your token when prompted.
Run the following to verify your login:
```
hf auth whoami
```
The result should be your username.

Manage Models

Models can be downloaded either to the default location (.cache/huggingface/hub) or to any local directory you specify. Where the model is stored can be defined using the --local-dir command line flag. Not specifying this will result in the model being stored in the default location. Storing the model in the folder where the packages for the inference engine are stored is good practice - this way, everything you need to run inference on a model is stored in the same place. However, if you use the same models with multiple backends frequently (e.g. using Qwen_QwQ-32B-Q4_K_M.gguf with both llama.cpp and vLLM), either set a common model directory or use the default HF option without specifying this flag.

First, activate the virtual environment that contains huggingface_hub:

source hf-env/bin/activate

Download Models

Models are downloaded using their HuggingFace tag. Here, we'll use bartowski/Qwen_QwQ-32B-GGUF as an example. To download a model, run:

hf download bartowski/Qwen_QwQ-32B-GGUF Qwen_QwQ-32B-Q4_K_M.gguf --local-dir models

Ensure that you are in the correct directory when you run this.

Delete Models

To delete a model in the specified location, run:

rm <model_name>

To delete a model in the default location, run:

hf delete-cache

This will start an interactive session where you can remove models from the HuggingFace directory. In case you've been saving models in a different location than .cache/huggingface, deleting models from there will free up space but the metadata will remain in the HF cache until it is deleted properly. This can be done via the above command but you can also simply delete the model directory from .cache/huggingface/hub.

Search Engine

[!NOTE] This step is optional but highly recommended for grounding LLMs with relevant search results from reputable sources. Targeted web searches via MCP tool calls make reports generated by LLMs much less prone to random hallucinations.

SearXNG

🌟 GitHub
📖 Documentation
📖 Docker Installation Reference

To power our search-based workflows, we don't want to rely on a search provider that can monitor searches. While using any search engine has this problem, metasearch engines like SearXNG mitigate it to a decent degree. SearXNG aggregates results from over 245 search services and does not track/profile users. You can use a hosted instance on the Internet but, considering the priorities of this guide and how trivial it is to set one up, we'll be spinning up our own instance on its default port (8080).

Create the project structure and fetch the compose file:

mkdir -p searxng/core-config && cd searxng
curl -fsSL -O https://raw.githubusercontent.com/searxng/searxng/master/container/docker-compose.yml

Append app-net network to the core service in docker-compose.yml:

services:
  core:
    # ...existing config...
    networks:
      default:
      app-net:
        aliases:
          # this allows reference to the "searxng-core" container via "searxng"
          - searxng

networks:
  default:
  app-net:
    external: true

Create core-config/settings.yml:

use_default_settings: true

search:
  formats:
    - html
    - json

server:
  secret_key: "CHANGE-ME-TO-RANDOM-STRING"
  image_proxy: true
  limiter: false

Start the services:
```
docker compose up -d
```

[!TIP] Run openssl rand -base64 32 to generate a secure random key.

SearXNG is now accessible at http://localhost:8080 externally and http://searxng:8080 from containers on app-net.

Open WebUI Integration

In case you want a simple web search workflow and want to skip MCP servers/agentic setups, Open WebUI supports web search functionality natively. Navigate to Admin Panel > Settings > Web Search and set the following values:

Enable Web Search
Web Search Engine: searxng
Searxng Query URL: http://searxng:8080/search?q=<query>
API Key: anything-you-like

Inference Engine

The inference engine is the software layer that loads model weights and manages text generation. This guide supports three popular engines: Ollama, llama.cpp, and vLLM. Each offers different trade-offs between ease of use, control, and performance.

If you're unsure where to begin and want some more context, see a detailed comparison in Comparing Inference Engines. It goes into what inference is, how the engines tackle it differently, and what the best fit for you might be.

Ollama

🌟 GitHub
📖 Documentation
🔧 Engine Arguments

Ollama will be installed as a service, so it runs automatically at boot.

Download Ollama from the official repository:

curl -fsSL https://ollama.com/install.sh | sh

We want our API endpoint to be reachable by the rest of the LAN. For Ollama, this means setting OLLAMA_HOST=0.0.0.0 in the ollama.service.

Run the following command to edit the service:
```
systemctl edit ollama.service
```
Find the [Service] section and add Environment="OLLAMA_HOST=0.0.0.0" under it. It should look like this:
```
[Service]
Environment="OLLAMA_HOST=0.0.0.0"
```
Save and exit.

Reload the environment.

systemctl daemon-reload
systemctl restart ollama

[!TIP] If you installed Ollama manually or don't use it as a service, remember to run ollama serve to properly start the server. Refer to Ollama's troubleshooting steps if you encounter an error.

llama.cpp

🌟 GitHub
📖 Documentation
🔧 Engine Arguments

[!TIP] This is my recommended inference engine. My biggest reasons are

the --fit flag and its ability to automatically load any model by calculating offload requirements based on available system resources, and

insanely quick model load times, especially compared to vLLM.

Clone the llama.cpp GitHub repository:

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp

Build the binary:

CPU
```
cmake -B build
cmake --build build --config Release -j <threads>
```
CUDA
```
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j <threads>
```
Replace <threads> with [number of threads your CPU has] - 2. This offers a massive boost to the build process as it leverages your system resources better.

For other systems looking to use Metal, Vulkan and other low-level graphics APIs, view the complete llama.cpp build documentation to leverage accelerated inference.

vLLM

🌟 GitHub
📖 Documentation
🔧 Engine Arguments

vLLM comes with its own OpenAI-compatible API that we can use just like Ollama. Where Ollama runs GGUF model files, vLLM can run AWQ, GPTQ, GGUF, BitsAndBytes, and safetensors (the default release type) natively.

Manual Installation (Recommended)

Create a directory and virtual environment for vLLM:

mkdir vllm
cd vllm
python3 -m venv .venv
source .venv/bin/activate

Install vLLM using pip:
```
pip install vllm
```
Serve with your desired flags. It uses port 8000 by default, but I'm using port 8556 here so it doesn't conflict with any other services:
```
vllm serve <model> --port 8556
```
To use as a service, add the following block to init.bash to serve vLLM on startup:
```
source .venv/bin/activate
vllm serve <model> --port 8556
```
Replace <model> with your desired model tag, copied from HuggingFace.

Docker Installation

Run:

docker run --gpus all \
-v ~/.cache/huggingface:/root/.cache/huggingface \
--env "HUGGING_FACE_HUB_TOKEN=<your_hf_hub_token>" \
-p 8556:8000 \
--ipc=host \
vllm/vllm-openai:latest \
--model <model>

Replace <your_hf_hub_token> with your HuggingFace Hub token and <model> with your desired model tag, copied from HuggingFace.

To serve a different model:

First stop the existing container:

docker ps -a
docker stop <vllm_container_ID>

If you want to run the exact same setup again in the future, skip this step. Otherwise, run the following to delete the container and not clutter your Docker container environment:
```
docker rm <vllm_container_ID>
```

Rerun the Docker command from the installation with the desired model.

docker run --gpus all \
-v ~/.cache/huggingface:/root/.cache/huggingface \
--env "HUGGING_FACE_HUB_TOKEN=<your_hf_hub_token>" \
-p 8556:8000 \
--ipc=host \
vllm/vllm-openai:latest \
--model <model>

Model Server

[!NOTE] Only needed for manual installations of llama.cpp/vLLM. Ollama manages model management via its CLI.

While the above steps will help you get up and running with an OpenAI-compatible LLM server, they will not help with this server persisting after you close your terminal window or restart your physical server. They also won't allow a chat platform to reliably reference and swap between the various models you have available - a likely use-case in a landscape where different models specialize in different tasks. Running the inference engine via Docker can achieve this persistence with the -d (for "detach") flag but (i) services like llama.cpp and vLLM are usually configured without Docker and (ii) it can't swap models on-demand. This necessitates a server that can manage loading/unloading, swapping, and listing available models.

llama-swap

🌟 GitHub
📖 Documentation
📖 Configuration

[!TIP] This is my recommended way to run llama.cpp/vLLM models.

llama-swap is a lightweight proxy server for LLMs that solves our pain points from above. It's an extremely configurable tool that allows a single point of entry for models from various backends. Models can be set up in groups, listed/unlisted easily, configured with customized hyperparameters, and monitored using streamed logs in the llama-swap web UI.

In the installation below, we'll use Qwen3-4B-Instruct-2507-UD-Q4_K_XL.gguf for llama.cpp and Qwen/Qwen3-4B-Instruct-2507 for vLLM. We'll also use port 7000 to serve the models on.

Create a new directory with the config.yaml file:

sudo mkdir llama-swap
cd llama-swap
sudo nano config.yaml

Enter the following and save:

llama.cpp

models:
  "qwen3-4b":
    proxy: "http://127.0.0.1:7000"
    cmd: |
      /app/llama-server
      -m /models/Qwen3-4B-Instruct-2507-UD-Q4_K_XL.gguf
      --port 7000 --host 0.0.0.0

vLLM (Docker)

models:
  "qwen3-4b":
    proxy: "http://127.0.0.1:7000"
    cmd: |
      docker run --name qwen-vllm
      --init --rm -p 7000:8080 --ipc=host
      vllm/vllm-openai:latest
      -m /models/Qwen/Qwen3-4B-Instruct-2507
    cmdStop: docker stop qwen-vllm

vLLM (local):

models:
  "qwen3-4b":
    proxy: "http://127.0.0.1:7000"
    cmd: |
      source /app/vllm/.venv/bin/activate &&
      /app/vllm/.venv/bin/vllm serve
      -m /models/Qwen/Qwen3-4B-Instruct-2507
      --port 7000 --host 0.0.0.0
    cmdStop: pkill -f "vllm serve"

Install the container:

We use the cuda tag here, but llama-swap offers cpu, intel, vulkan, and musa tags as well. Releases can be found here.

llama.cpp

docker run -d --gpus all --restart unless-stopped --network app-net --pull=always --name llama-swap -p 9292:8080 \
-v /path/to/models:/models \
-v /home/<username>/llama-swap/config.yaml:/app/config.yaml \
-v /home/<username>/llama.cpp/build/bin/llama-server:/app/llama-server \
ghcr.io/mostlygeek/llama-swap:cuda

vLLM (Docker/local)

docker run -d --gpus all --restart unless-stopped --network app-net --pull=always --name llama-swap -p 9292:8080 \
-v /path/to/models:/models \
-v /home/<username>/vllm:/app/vllm \
-v /home/<username>/llama-swap/config.yaml:/app/config.yaml \
ghcr.io/mostlygeek/llama-swap:cuda

Replace <username> with your actual username and /path/to/models with the path to your model files.

[!NOTE] llama-swap prefers Docker-based vLLM due to cleanliness of environments and adherence to SIGTERM signals sent by the server. I've written out both options here.

This should result in a functioning llama-swap instance running at http://localhost:9292, which can be confirmed by running curl http://localhost:9292/health. It is highly recommended that you read the configuration documentation. llama-swap is thoroughly documented and highly configurable - utilizing its capabilities will result in a tailored setup ready to deploy as you need it.

`systemd` Service

📖 llama.cpp Server Documentation
📖 llama.cpp Preset Documentation
📖 vLLM Online Serving Documentation

The other way to persist a model across system reboots is to start the inference engine in a .service file that will run alongside the Linux operating system when booting, ensuring that it is available whenever the server is on.

llama.cpp can now natively allow for model-switching via its --models-preset flag. Compared to llama-swap, llama.cpp's in-built switcher naturally has less overhead because it's bundled with the binary but it also doesn't come with a UI where one can switch models, view activity & logs, monitor system performance during inference, etc. llama.cpp and llama-swap are functionally equivalent with respect to model-switching. For vLLM, if you're willing to live with the relative compromise of not being able to swap models/backends and are satisfied with running one model, systemd is the lowest overhead solution and works great.

Let's call the service we're about to build llm-server.service. We'll assume all models are in the models child directory - you can change this as you need to.

Create the systemd service file:

sudo nano /etc/systemd/system/llm-server.service

llama.cpp

Create the models.ini file (in your llama.cpp directory):

version = 1

[*]
jinja = true
no-mmap = true
no-warmup = true
flash-attn = true

[qwen3.6-27b]
model = models/Qwen3.6-27B/Qwen_Qwen3.6-27B-Q8_0.gguf
mmproj = models/Qwen3.6-27B/mmproj-BF16.gguf
ctx-size = 180000
parallel = 1
context-shift = true
fit = off
tensor-split = 12,11
spec-type = draft-mtp
spec-draft-n-max = 2
temp = 0.6
min-p = 0.0
top-p = 0.95
top-k = 20
presence-penalty = 0.0
repeat-penalty = 1.0
chat-template-kwargs = {"preserve_thinking": true}

[gemma-4-31b]
model = models/Gemma4-31B/gemma-4-31B-it-qat-UD-Q4_K_XL.gguf
mmproj = models/Gemma4-31B/mmproj-BF16.gguf
model-draft = models/Gemma4-31B/gemma-4-31b-it-qat-q4_0-assistant.gguf
ctx-size = 130000
main-gpu = 0
n-gpu-layers = 99
parallel = 1
fit = on
spec-type = draft-mtp
spec-draft-n-max = 2
chat-template-kwargs = {"preserve_thinking": true}
temp = 1.0
min-p = 0.0
top-p = 0.95
top-k = 64
cache-ram = 2048
ctx-checkpoints = 2
image-min-tokens = 300
image-max-tokens = 512

Configure the service file:

[Unit]
Description=llama.cpp Server
After=network.target

[Service]
Type=simple
User=<username>
Group=<username>
WorkingDirectory=/home/<username>/llama.cpp

ExecStart=/home/<username>/llama.cpp/build/bin/llama-server \
    --models-preset models.ini \
    --port 9292 \
    --host 0.0.0.0

# Restart policy
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

vLLM

Configure the service file:

[Unit]
Description=LLM Server Service
After=network.target

[Service]
User=<user>
Group=<user>
WorkingDirectory=/home/<user>/vllm/
ExecStart=/bin/bash -c 'source .venv/bin/activate && vllm serve --port <port> --host 0.0.0.0 -m /home/<username>/vllm/models/<model>'
Restart=always
RestartSec=10s

[Install]
WantedBy=multi-user.target

Replace <username>, <port>, and <model> with your Linux username, desired port for serving, and desired model respectively.

Reload the systemd daemon:

sudo systemctl daemon-reload

Run the service:

If llm-server.service doesn't exist:

sudo systemctl enable llm-server.service
sudo systemctl start llm-server

If llm-server.service does exist:
```
sudo systemctl restart llm-server
```

(Optional) Check the service's status:

sudo systemctl status llm-server

Open WebUI Integration

llama-swap

Navigate to Admin Panel > Settings > Connections and set the following values:

Enable OpenAI API
API Base URL: http://llama-swap:8080/v1
API Key: anything-you-like

`systemd` Service

Follow the same steps as above.

Enable OpenAI API
API Base URL: http://localhost:<port>/v1
API Key: anything-you-like

Replace <port> with your desired port.

[!WARNING] If you use the systemd service, ensure the port is open to the local network via ufw. Refer to Firewall.

Chat Platform

Open WebUI

🌟 GitHub
📖 Documentation

Open WebUI is a web-based interface for managing models and chats, and provides a beautiful, performant UI for communicating with your models. You will want to do this if you want to access your models from a web interface. If you're fine with using the command line or want to consume models through a plugin/extension, you can skip this step.

To install without Nvidia GPU support, run the following command:

docker run -d -p 3000:8080 --network app-net --add-host=host.docker.internal:host-gateway -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:main

For Nvidia GPUs, run the following command:

docker run -d -p 3000:8080 --network app-net --gpus all --add-host=host.docker.internal:host-gateway -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:cuda

You can access it by navigating to http://localhost:3000 in your browser or http://<server_ip>:3000 from another device on the same network. There's no need to add this to the init.bash script as Open WebUI will start automatically at boot via Docker Engine.

MCP Proxy Server

Model Context Protocol (MCP) is a protocol that tools (functions/scripts written in code) to LLMs in a standardized way. Generally, models are being trained more and more with the ability to natively call tools in order to power agentic tasks - think along the lines of having a model use sequential thinking to formulate multiple thoughts, execute multiple targeted web searches, and provide a response leveraging real-time information. MCP also, probably more importantly for most people, enables models to call third-party tools like those for GitHub, Azure, etc. A complete list, curated and maintained by Anthropic, can be found here.

Most guides on the Internet concerning MCP will have you spin up an MCP server via a client like VS Code, Cline, etc. since most agentic uses are for coding or Claude Desktop, which is a proprietary app by Anthropic and not at all what we're aiming to achieve in this guide with respect to privacy. There are other chat clients that support MCP server management from the UI itself (LobeChat, Cherry Studio, etc.), but we want to be able to manage MCP servers in a central and modular way. That way, (i) they aren't tied to a specific client and are available to every client you may use them with and (ii) if you switch chat platforms in the future, your MCP servers require zero changes because they run as a decoupled service - little bit more maintenance for a lot more flexibility down the line. We can do this by setting up an MCP proxy server.

This proxy server will take the MCP servers running via stdio (standard IO) protocol (that can only be accessed by an application running on that device) and make them compatible with streamable HTTP. Any MCP-enabled client can use streamable HTTP so they'll also be able to use all the servers we install on our physical server. This centralizes the management of your MCP servers: create/edit/delete servers in one place, use any of them from your various clients (Open WebUI, VS Code, etc.).

We'll use the fetch, sequential-thinking, and searxng MCP servers to get started. The process for adding more servers will be identical.

mcp-proxy

🌟 GitHub

mcp-proxy is a server proxy that allows switching between transports (stdio to streamable HTTP and vice versa). I'll be using port 3131 to avoid conflicts - feel free to change this as you need to. I will also be extending mcp-proxy to include uv: most MCP servers use either npx or uv and setting mcp-proxy up without uv is going to hamper your ability to run the MCP servers you'd like. If you don't require uv, (i) don't add the build section in the compose file and (ii) skip step 4.

Create a compose file

mkdir mcp-proxy
cd mcp-proxy
sudo nano docker-compose.yaml

Enter the following:

services:
  mcp-proxy:
    container_name: mcp-proxy
    build:
      context: .
      dockerfile: Dockerfile
    networks:
      - app-net
    volumes:
      - .:/config
      - /:/<server_hostname>:ro
    restart: unless-stopped
    ports:
      - 3131:3131
    command: "--pass-environment --port=3131 --host 0.0.0.0 --transport streamablehttp --named-server-config /config/servers.json"

networks:
  app-net:
    external: true

Replace <server_hostname> with your actual server's hostname (or whatever else). This is primarily useful when adding filesystem or similar MCP servers that read from and write files to the file system. Feel free to skip if that isn't your goal.

Create a servers.json file:

{
    "mcpServers": {
        "fetch": {
            "disabled": false,
            "timeout": 60,
            "command": "uvx",
            "args": [
                "mcp-server-fetch"
            ],
            "transportType": "stdio"
        },
        "sequential-thinking": {
            "command": "npx",
            "args": [
                "-y",
                "@modelcontextprotocol/server-sequential-thinking"
            ]
        },
        "searxng": {
            "command": "npx",
            "args": ["-y", "mcp-searxng"],
            "env": {
                "SEARXNG_URL": "http://searxng:8080/search?q=<query>"
            }
        }
    }
}

Create a Dockerfile:

sudo nano Dockerfile

Enter the following:

FROM ghcr.io/sparfenyuk/mcp-proxy:latest

# Install dependencies for nvm and Node.js
RUN apk add --update npm

# Install the 'uv' package
RUN python3 -m ensurepip && pip install --no-cache-dir uv

ENV PATH="/usr/local/bin:/usr/bin:$PATH" \
    UV_PYTHON_PREFERENCE=only-system

ENTRYPOINT ["catatonit", "--", "mcp-proxy"]

Start the container with docker compose up -d

Your mcp-proxy container should be up and running! Adding servers is simple: add the relevant server to servers.json (you can use the same configuration that the MCP server's developer provides for VS Code, it's identical) and then restart the container with docker restart mcp-proxy.

MCPJungle

🌟 GitHub

MCPJungle is another MCP proxy server with a different focus. It focuses on providing more of a "production-grade" experience, a lot of which is disabled by default in the development mode of the application. We'll use the standard development version of the container here on port 4141.

Create a compose file:

mkdir mcpjungle
cd mcpjungle
sudo nano docker-compose.yaml

Enter the following and save:

# MCPJungle Docker Compose configuration for individual users.
# Use this compose file if you want to run MCPJungle locally for your personal MCP management & Gateway.
# The mcpjungle server runs in development mode.
services:
  db:
    image: postgres:latest
    container_name: mcpjungle-db
    environment:
      POSTGRES_USER: mcpjungle
      POSTGRES_PASSWORD: mcpjungle
      POSTGRES_DB: mcpjungle
    ports:
      - "5432:5432"
    networks:
      - app-net
    volumes:
      - db_data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "PGPASSWORD=mcpjungle pg_isready -U mcpjungle"]
      interval: 10s
      timeout: 5s
      retries: 5
    restart: unless-stopped

  mcpjungle:
    image: mcpjungle/mcpjungle:${MCPJUNGLE_IMAGE_TAG:-latest-stdio}
    container_name: mcpjungle-server
    environment:
      DATABASE_URL: postgres://mcpjungle:mcpjungle@db:5432/mcpjungle
      SERVER_MODE: ${SERVER_MODE:-development}
      OTEL_ENABLED: ${OTEL_ENABLED:-false}
    ports:
      - "4141:8080"
    networks:
      - app-net
    volumes:
      # Mount host filesystem current directory to enable filesystem MCP server access
      - .:/host/project:ro
      - /home/<username>:/host:ro
      # Other options:
      # - ${HOME}:/host/home:ro
      # - /tmp:/host/tmp:rw
    depends_on:
      db:
        condition: service_healthy
    restart: always

volumes:
  db_data:

networks:
  app-net:
    external: true

Start the container with docker compose up -d

Create a tool file:

sudo nano fetch.json

Enter the following and save:

{
    "name": "fetch",
    "transport": "stdio",
    "command": "npx",
    "args": ["mcp-server-fetch"]
}

docker exec -i mcpjungle-server /mcpjungle register -c /host/project/fetch.json

Repeat steps 3 and 4 for every tool mentioned. Commands for sequential-thinking and searxng can be found below.

sequential-thinking

{
    "name": "sequential-thinking",
    "transport": "stdio",
    "command": "npx",
    "args": ["-y", "@modelcontextprotocol/server-sequential-thinking"]
}

searxng

{
    "name": "searxng",
    "transport": "stdio",
    "command": "npx",
    "args": ["-y", "mcp-searxng"],
    "env": {
        "SEARXNG_URL": "http://searxng:8080/search?q=<query>"
    }
}

Comparison

The choice between the two services is yours entirely: I use mcp-proxy because I find the workflow slightly less cumbersome than MCPJungle. Here's a comparison with the stengths of each service.

mcp-proxy > MCPJungle

Servers can just be added to servers.json and will be registered automatically on container restart - MCPJungle requires manual registration of tools via the CLI
Uses the standard MCP syntax that most clients accept for configuration
Lighter footprint in that it doesn't need to spin up a separate database container
Uses stateful connections - MCPJungle spins up a new connection per tool call, which can lead to some performance overhead

MCPJungle > mcp-proxy

Combines all tools under one endpoint, making it very easy to integrate into a chat frontend
Capable of creating a very configurable setup with tool groups, access control, selective tool enabling/disabling
Supports enterprise features like telemetry

Open WebUI Integration

Open WebUI recently added support for streamable HTTP - where once you may have had to use mcpo, Open WebUI's way of automatically generating an OpenAPI-compatible HTTP server, you can use the MCP servers you've set up as-is with no changes.

mcp-proxy

Navigate to Admin Panel > Settings > External Tools. Click the + button to add a new tool and enter the following information:

URL: http://mcp-proxy:<port>/servers/<tool_name>/mcp
API Key: anything-you-like
ID: <tool_name>
Name: <tool_name>

Replace <port> with the port of the MCP service and <tool_name> with the specific tool you're adding.

MCPJungle

Follow the same steps as above. By design, MCPJungle exposes all tools via one endpoint, so you should only have to add once:

URL: http://mcpjungle-server:8080/mcp
API Key: anything-you-like
ID: <tool_name>
Name: <tool_name>

[!IMPORTANT] When configuring models in Open WebUI (via Admin Panel > Settings > Models > my-cool-model > Advanced Params), change the Function Calling parameter from Default to Native. This step will allow the model to use multiple tool calls to formulate a single response instead of just one.

VS Code/Claude Desktop Integration

The steps for integrating your MCP proxy server in another client such as VS Code (Claude Desktop, Zed, etc.) will be similar, if not exactly the same.

Add the following key and value to your mcp.json file:

"your-mcp-proxy-name": {
    "timeout": 60,
    "type": "stdio",
    "command": "npx",
    "args": [
    "mcp-remote",
    "http://<your-server-url>/mcp",
    "--allow-http"
    ]
}

Text-to-Speech Server

Kokoro FastAPI

🌟 GitHub

Kokoro FastAPI is a text-to-speech server that wraps around and provides OpenAI-compatible API inference for Kokoro-82M, a state-of-the-art TTS model. The documentation for this project is fantastic and covers most, if not all, of the use cases for the project itself.

To install Kokoro-FastAPI, run

git clone https://github.com/remsky/Kokoro-FastAPI.git
cd Kokoro-FastAPI
docker compose up --build

The server can be used in two ways: an API and a UI. By default, the API is served on port 8880 and the UI is served on port 7860.

Open WebUI Integration

Navigate to Admin Panel > Settings > Audio and set the following values:

Text-to-Speech Engine: OpenAI
API Base URL: http://host.docker.internal:8880/v1
API Key: anything-you-like
Set Model: kokoro
Response Splitting: None (this is crucial - Kokoro uses a novel audio splitting system)

The server can be used in two ways: an API and a UI. By default, the API is served on port 8880 and the UI is served on port 7860.

Image Generation Server

ComfyUI

🌟 GitHub
📖 Documentation

ComfyUI is a popular open-source graph-based tool for generating images using image generation models such as Stable Diffusion XL, Stable Diffusion 3, and the Flux family of models.

Clone and navigate to the repository:

git clone https://github.com/comfyanonymous/ComfyUI
cd ComfyUI

Set up a new virtual environment:

python3 -m venv comfyui-env
source comfyui-env/bin/activate

Download the platform-specific dependencies:

Nvidia GPUs

pip install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu121

AMD GPUs

pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/rocm6.0

Intel GPUs

Read the installation instructions from ComfyUI's GitHub.

Download the general dependencies:
```
pip install -r requirements.txt
```

Now, we have to download and load a model. Here, we'll use FLUX.1 [dev], a new, state-of-the-art medium-tier model by Black Forest Labs that fits well on an RTX 3090 24GB. Since we want this to be set up as easily as possible, we'll use a complete checkpoint that can be loaded directly into ComfyUI. For a completely customized workflow, CLIPs, VAEs, and models can be downloaded separately. Follow this guide by ComfyUI's creator to install the FLUX.1 models in a fully customizable way.

[!NOTE] FLUX.1 [schnell] HuggingFace (smaller, ideal for <24GB VRAM)

FLUX.1 [dev] HuggingFace (larger, ideal for 24GB VRAM)

Download your desired model into /models/checkpoints.
If you want ComfyUI to be served at boot and effectively run as a service, add the following lines to init.bash:
```
cd /path/to/comfyui
source comfyui/bin/activate
python main.py --listen
```
Replace /path/to/comfyui with the correct relative path to init.bash.

Otherwise, to run it just once, simply execute the above lines in a terminal window.

Open WebUI Integration

Navigate to Admin Panel > Settings > Images and set the following values:

Image Generation Engine: ComfyUI
API Base URL: http://localhost:8188

[!TIP] You'll either need more than 24GB of VRAM or to use a small language model mostly on CPU to use Open WebUI with FLUX.1 [dev]. FLUX.1 [schnell] and a small language model, however, should fit cleanly in 24GB of VRAM, making for a faster experience if you intend to regularly use both text and image generation together.

SSH

Enabling SSH allows you to connect to the server remotely. After configuring SSH, you can connect to the server from another device on the same n

怎么上手

项目说明

核心能力

容易踩的坑

最近发版

用户点评

README 原文

Local LLaMA Server Setup Documentation

Table of Contents

About

Priorities

Prerequisites

General

Allow sudo Permissions

Update System Packages

Schedule Startup Script

Configure Script Permissions

Configure Auto-Login (optional)

Docker

Add User to Docker Group

Nvidia Container Toolkit

Create a Network

Harden Docker Containers

Helpful Commands

HuggingFace CLI

Manage Models

Download Models

Delete Models

Search Engine

SearXNG

Open WebUI Integration

Inference Engine

Ollama

llama.cpp

vLLM

Model Server

llama-swap

systemd Service

Open WebUI Integration

llama-swap

systemd Service

Chat Platform

Open WebUI

MCP Proxy Server

mcp-proxy

MCPJungle

Comparison

Open WebUI Integration

mcp-proxy

MCPJungle

VS Code/Claude Desktop Integration

Text-to-Speech Server

Kokoro FastAPI

Open WebUI Integration

Image Generation Server

ComfyUI

Open WebUI Integration

SSH

相关项目

Allow `sudo` Permissions

`systemd` Service

`systemd` Service