chatllm.cpp

ChatLLM.cpp

中文版

Inference of a bunch of models from less than 1B to more than 300B, for real-time multimodal chat with RAG on your computer (CPU & GPU), pure C++ implementation based on @ggerganov's ggml.

Deliver accurate or better results than other implementations o-c-c-a-s-i-onally.

| Supported Models | Download Quantized Models |

graph TD;
ggml --> chatllm.cpp
chatllm.cpp --> AlphaGeometryRE
chatllm.cpp --> WritingTools
chatllm.cpp --> LittleAcademia
chatllm.cpp --> LLMirror
subgraph coding[ ]
    AlphaGeometryRE
    WritingTools
    LittleAcademia
    LLMirror
end
ggml[<a href="https://github.com/ggml-org/ggml"                     style="text-decoration:none;">ggml</a>            <br><span style="font-size:10px;">Machine learning library</span>];
chatllm.cpp[<a href="https://github.com/foldl/chatllm.cpp"          style="text-decoration:none;">chatllm.cpp</a>     <br><span style="font-size:10px;">LLM inference</span>];
AlphaGeometryRE[<a href="https://github.com/foldl/alphageometryre"  style="text-decoration:none;">AlphaGeometryRE</a> <br><span style="font-size:10px;">AlphaGeometry Re-engineered</span>];
WritingTools[<a href="https://github.com/foldl/WritingTools"        style="text-decoration:none;">Writing Tools</a>   <br><span style="font-size:10px;">AI aided writing</span>];
LittleAcademia[<a href="https://github.com/foldl/little-academia"   style="text-decoration:none;">Little Academia</a> <br><span style="font-size:10px;">Learn programming</span>];
LLMirror[<a href="https://github.com/foldl/llmirror"   style="text-decoration:none;">LL·Mirror</a> <br><span style="font-size:10px;">LLM introspection</span>];

What's New:

• 2026-10-02: Intern-Decision
• 2026-09-27: Spark-2.5
• 2026-09-25: TeleOCR
• 2026-09-08: MiniCPM5-2B
• 2026-08-22: Instella-MoE
• 2026-08-04: Nanbeige4.2
• 2026-07-24: G9v3
• 2026-07-16: Introspection with lens
• 2026-07-15: OvisOCR2
• 2026-06-27: PaddleOCR-VL
• 2026-06-22: Mellum-2
• 2026-06-14: HY-MT2
• 2026-05-28: MiniCPM5
• 2026-05-05: Gemma-4
• 2026-03-28: InternVL3.5
• 2026-03-27: Qianfan-OCR
• 2026-03-22: Penguin-VL
• 2026-03-06: Qwen3.5
• 2026-03-03: GLM-OCR
• 2025-10-10: I can draw: Janus-Pro
• 2025-06-21: I can hear: Qwen2-Audio
• 2025-05-23: I can see: Fuyu
• 2025-05-21: Re-quantization when loading (e.g. --re-quantize q4_k)
• 2025-05-17: I can speak: Orpheus-TTS
• 2025-03-24: GGMM file format
• 2025-02-21: Distributed inference
• 2025-02-10: GPU acceleration 🔥
• 2024-12-09: Reversed role
• 2024-11-21: Continued generation
• 2024-11-01: generation steering
• 2024-06-15: Tool calling
• 2024-05-29: ggml is forked instead of submodule
• 2024-05-14: OpenAI API, CodeGemma Base & Instruct supported
• 2024-05-08: Layer shuffling

Features

• Accelerated memory-efficient CPU/GPU inference with int4/int8 quantization, optimized KV cache and parallel computing;

• Use OOP to address the similarities between different Transformer based models;

• Streaming generation with typewriter effect;

• Continuous chatting (content length is virtually unlimited)

  Two methods are available: Restart and Shift. See --extending options.

• Retrieval Augmented Generation (RAG) 🔥

• LoRA;

• Python/JavaScript/C/Nim Bindings, web demo, and more possibilities.

Quick Start

As simple as main_nim -i -m :model_id. Check it out.

Usage

Preparation

Clone the ChatLLM.cpp repository into your local machine:

git clone --recursive https://github.com/foldl/chatllm.cpp.git && cd chatllm.cpp

If you forgot the --recursive flag when cloning the repository, run the following command in the chatllm.cpp folder:

git submodule update --init --recursive

Quantize Model

Some quantized models can be downloaded on demand.

Install dependencies of convert.py:

pip install -r requirements.txt

Use convert.py to transform models into quantized GGML format. For example, to convert the fp16 base model to q8_0 (quantized int8) GGML model, run:

# For models such as ChatLLM2-6B, InternLM, LlaMA, LlaMA-2, Baichuan-2, etc
python convert.py -i path/to/model -t q8_0 -o quantized.bin --name ModelName

# For some models such as CodeLlaMA, model type should be provided by `-a`
# Find `-a ...` option for each model in `docs/models.md`.
python convert.py -i path/to/model -t q8_0 -o quantized.bin -a CodeLlaMA --name ModelName

Use --name to specify model's name in English. Optionally, use --native_name to specify model's name in another language. Use -l to specify the path of the LoRA model to be merged, such as:

python convert.py -i path/to/model -l path/to/lora/model -o quantized.bin --name ModelName

Note: Appropriately, only HF format is supported (with a few exceptions); Format of the generated .bin files is different from the one (GGUF) used by llama.cpp.

Customized Quantization

It's possible to specify quantization for each tensor using -tt pattern type, where pattern is a regex pattern to match against tensor names. Multiple -tt ... ... can be used. In this example, all tensors are using q4_k quantization except embeddings are using q8_0.

... -t q4_k -tt lm_head q8_0 -tt model.embed_tokens.weight q8_0

Supported Types

Type	GGML_TYPE
f32	GGML_TYPE_F32
f16	GGML_TYPE_F16
q8_0	GGML_TYPE_Q8_0
q4_0	GGML_TYPE_Q4_0
q4_1	GGML_TYPE_Q4_1
q4_k	GGML_TYPE_Q4_K

Build

In order to build this project you have several different options.

• Using CMake:

  cmake -B build
  cmake --build build -j --config Release
  

  The executable is ./build/bin/main.

  There are lots of GGML_... options to play with. Example: Vulkan acceleration together with RPC and backend dynamic loading:

  cmake -B build -DGGML_VULKAN=1 -DGGML_RPC=1 -DGGML_CPU_ALL_VARIANTS=1 -DGGML_BACKEND_DL=1
  

Run

Now you may chat with a quantized model by running:

./build/bin/main -m llama2.bin  --seed 100                      # Llama-2-Chat-7B
# Hello! I'm here to help you with any questions or concerns ....

To run the model in interactive mode, add the -i flag. For example:

# On Windows
.\build\bin\Release\main -m model.bin -i

# On Linux (or WSL)
rlwrap ./build/bin/main -m model.bin -i

In interactive mode, your chat history will serve as the context for the next-round conversation.

Run ./build/bin/main -h to explore more options!

Plan Nim

All Python scripts are going to be rewritten in Nim, with following exceptions:

• when pickle is used

Acknowledgements

• This project is started as refactoring of ChatGLM.cpp, without which, this project could not be possible.

• Thank those who have released their the model sources and checkpoints.

• chat_ui.html adapted from Ollama-Chat.

Note

• This project is a hobby project to learn DL & GGML, and under active development.

• This project is never supposed to be used seriously and security vulnerabilities won't be treated seriously, either.

• PRs of features won't be accepted, while PRs for bug fixes are warmly welcome.

  AI Usage Policy: Slop codes generated by AI won't be accepted.