Lvllm

Base: lvllm-v2.5.0 · Contents: full post-v2.5.0 tree up to release tip 2f47920d7f (Qwen3.8 NVFP4 validation + SM8x tiled prefill attention + KV-startup line + scheduler fixes) · Tested: 4-GPU mixed TP4 (2×3090+2×5060Ti) + pure-sm86 soak, config semantics re-pass

Install (Python 3.12, Linux x86_64)

pip install https://github.com/guqiong96/Lvllm/releases/download/lvllm-v2.5.1/lvllm-2.5.1-cp312-cp312-manylinux_2_34_x86_64.whl

Dependencies (incl. the bundled FlashInfer wheel with the SM120 sparse-MLA fix) resolve automatically; first serve JIT-compiles the sparse-MLA kernels with nvcc (a few minutes), then caches under ~/.cache/flashinfer/. On a mixed-arch host export FLASHINFER_CUDA_ARCH_LIST="<all ranks>" (e.g. "8.6 12.0f").

Source: tag lvllm-v2.5.1 (branch main).

Wheel artifacts

Build environment: tip 2f47920d7f · Python 3.12.11 · torch 2.13.0+cu130 · nvcc CUDA 13.2 · gcc-toolset-14 (GCC 14.2.1) · TORCH_CUDA_ARCH_LIST="7.5 8.0 8.6 8.9 10.0 12.0" · VLLM_VERSION_OVERRIDE=2.5.1.

Note: these built wheels' METADATA pins flashinfer to the lvllm-v2.5.0 release URL (byte-identical asset, up forever); requirements/cuda.txt on this tag already points at this release, so source/editable installs are self-contained too.

Verify after install:

python -c "import vllm; print(vllm.__version__)"   # -> 2.5.1
cuobjdump --list-elf $(python -c "import vllm,os;print(os.path.dirname(vllm.__file__))")/_C_stable_libtorch.abi3.so | grep -oE 'sm_[0-9]+' | sort -u
# expected: sm_75 sm_80 sm_86 sm_89 sm_90 sm_100 sm_120

(_moe_C_stable_libtorch = sm_75/80/86/89/100/120, _flashkda_C = sm_100/120; sm_90 is not in the MoE arch list by design of the build recipe.)

lvllm-v2.5.1 (2026-09-20)

Base: lvllm-v2.5.0 · Release Type: patch (crash fixes + prefill speed + scheduling)

Behavior changes (read before upgrading)

• VLLM_DSV4_UNPACK_INDEXER is now ON by default. The indexer-pool carve-out below is active unless you set VLLM_DSV4_UNPACK_INDEXER=0 (escape hatch keeps the old packed layout). Cost: ~5 % more bytes per pool block (≈ −11 % block count); this is the fix for the mixed-arch Xid31 crash, so opting out is only meaningful on single-architecture hosts.
• YAML config key: false now actually takes effect. --config files expanded booleans to --no-* flags only when the flag was visible on the top-level parser; serve options live on the serve subparser, so every *: false line (e.g. async_scheduling: false) was silently dropped until now. Anything that relied on a dropped false line keeps its default only by accident — check your YAML.

Fixes

• DeepSeek-V4.x mixed-arch TP prefix-reuse crash (Xid31 FAULT_PDE) — under block-outermost interleaved KV packing, indexer pages packed with a ≈1 MB stride0 faulted on Blackwell page tables on SM120 ranks once prefix reuse reached the same block depth. The unpacked indexer region (now default; VLLM_DSV4_UNPACK_INDEXER=0 opts out) carves indexer layers out of the interleaved pool into a dense tail region (natural per-page stride0); MLA/compressor groups keep interleaving. Validated on the exact crash seed under both guarded (blocking) and production (FULL-graph) timings, plus GSM8K 100/100 on SM120 TP2. (PR #111)

• Hybrid-model scheduling livelocks on tight KV pools — three scheduler-side fixes: the full-sequence admission gate now splits recycling-aware

Release Notes — lvllm-v2.5.0

Base Version: vllm commit 71888f507a (upstream/main, 2026-09-14) + lk_moe Release Type: Feature release (Lvllm)

Models and supported architectures

• native — done by upstream vLLM.
• new — support added by this Lvllm release.
• fixed here — upstream path corrected by this Lvllm release.

Installation

pip install lvllm-2.5.0-*.whl

All dependencies resolve automatically, including the FlashInfer wheel attached to this release (flashinfer_python-0.7.0-py3-none-any.whl, contains the SM120 sparse-MLA fix #5075; not on PyPI). First serve JIT-compiles the sparse-MLA kernels (nvcc required, a few minutes), then results are cached under ~/.cache/flashinfer/. On machines with mixed architectures add FLASHINFER_CUDA_ARCH_LIST="12.0f".

Benchmark reference (2× RTX 3090, TP2)

DeepSeek-V4.1-Flash

Launch: commands/dsv41_serve_tp2_3090_dspark.sh

Measured (single request, greedy):

DeepSeek-V4-Flash (0731)

Same hardware as V4.1. Launch: commands/dsv4_0731_serve_tp2_5060ti.sh (dspark variant …_dspark.sh; SM86/TP4 variants also under commands/).

Measured (single request, greedy):

Qwen3.8-Flash-Next

Launch: commands/qwen38_serve_tp4.sh (plain) / commands/qwen38_serve_tp4_mtp.sh (MTP).

Measured (single request, greedy):

GLM-5.3-Flash

Launch: commands/glm53_serve_tp2_5060ti_plain.sh (SM120 plain) / commands/glm53_serve_tp4.sh (mixed TP4; `--kv-cache-dtype

Release Notes — v0.2.4.0

Base Version: vLLM [3116c5d] (upstream) + lk_moe v2.4.2 Release Type: Feature integration release

Run command Qwen3.8-Flash-Next ( SM120+) [epyc 7642 x 2, 16 x ddr4 3200, 5060Ti x 2]

normal decode: 35t/s :

CUDA_DEVICE_ORDER="PCI_BUS_ID" \
CUDA_VISIBLE_DEVICES="1,2" \
VLLM_PLE_CPU_OFFLOAD="1" \
LVLLM_MOE_NUMA_ENABLED="1" \
LK_THREADS="48" \
OMP_NUM_THREADS="1" \
LK_THREAD_BINDING="CPU_CORE" \
LVLLM_GPU_PREFETCH_WINDOW="1" \
LVLLM_GPU_PREFILL_MIN_BATCH_SIZE="1024" \
LVLLM_ENABLE_NUMA_INTERLEAVE="1" \
LK_POWER_SAVING="1" \
VLLM_USE_V2_MODEL_RUNNER="1" \
FLASHINFER_DISABLE_VERSION_CHECK="1" \
vllm serve \
  --model "/home/guqiong/Models/Qwen3.8-Flash-Next-NVFP4" \
  --host "0.0.0.0" \
  --port 8070 \
  --tensor-parallel-size 2 \
  --max-model-len 66000 \
  --gpu-memory-utilization 0.95 \
  --trust-remote-code \
  --tokenizer-mode "auto" \
  --served-model-name "Qwen3.8-Flash-Next-NVFP4" \
  --compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY","mode":"VLLM_COMPILE"}' \
  --enable-prefix-caching \
  --enable-chunked-prefill \
  --max-num-batched-tokens 4096 \
  --dtype "bfloat16" \
  --max-num-seqs 2 \
  --enable-auto-tool-choice \
  --reasoning-parser "qwen3" \
  --tool-call-parser "qwen3_xml" \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
  --default-chat-template-kwargs '{"enable_thinking": false}' \
  --disable-custom-all-reduce
 

Note: do not add the VLLM_PLE_CPU_OFFLOAD parameter when running other models.

refactor: optimize LK_THREADS calculation and use cfg.use_gpu_prefill in lk-moe v2.3.3 fix https://github.com/guqiong96/Lsglang/issues/16#issuecomment-5297172594: add NUMA interleave override when LVLLM_MOE_NUMA_ENABLED=1 and LVLLM_ENABLE_NUMA_INTERLEAVE=1

⚠️ Important Note Set OMP_NUM_THREADS=1, otherwise model file shard loading will be 10x slower.

⚠️ Important Note Concurrent tilelang JIT compilation corrupts shared cache, crashing vllm. Workaround: rm ~/.cache/vllm/ -rf If vLLM crashes with OOM due to FlashInfer Autotuner during startup, add --no-enable-flashinfer-autotune to the launch command to disable it.

⚠️ Important Note

pip install https://github.com/guqiong96/Lvllm/releases/download/lvllm-v2.3.11/lvllm-2.3.11-cp312-cp312-manylinux_2_34_x86_64.whl

# check the latest version at: https://github.com/guqiong96/Lvllm/releases

pip install https://github.com/guqiong96/Lvllm/releases/download/lvllm-v2.3.10/flashinfer_cubin-0.6.16.post3-py3-none-any.whl

Command for 2× RTX 5060 Ti

LVLLM_MOE_NUMA_ENABLED=1 \
LK_THREADS=48 \
OMP_NUM_THREADS=1 \
LK_THREAD_BINDING=CPU_CORE \
LVLLM_GPU_PREFETCH_WINDOW=1 \
LVLLM_GPU_PREFILL_MIN_BATCH_SIZE=1024 \
LK_POWER_SAVING=1 \
FLASHINFER_DISABLE_VERSION_CHECK=1 \
vllm serve /home/guqiong/Downloads/DeepSeek-V4-Flash-0731 \
  --host 0.0.0.0 \
  --port 8070 \
  --tensor-parallel-size 2 \
  --max-model-len 32000 \
  --gpu-memory-utilization 0.95 \
  --trust-remote-code \
  --served-model-name DeepSeek-V4-Flash-0731 \
  --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY", "mode": "VLLM_COMPILE"}' \
  --enable-prefix-caching \
  --enable-chunked-prefill \
  --max-num-batched-tokens 4096 \
  --dtype bfloat16 \
  --max-num-seqs 2 \
  --enable-auto-tool-choice \
  --kv-cache-dtype fp8_ds_mla \
  --tokenizer-mode deepseek_v4 \
  --tool-call-parser deepseek_v4 \
  --reasoning-parser deepseek_v4 \
  --speculative-config '{"method":"dspark","num_speculative_tokens":5,"draft_sample_method":"probabilistic"}' \
  --disable-custom-all-reduce

RTX Pro 6000 Optimization

For detailed optimization parameters, see: https://github.com/guqiong96/Lvllm#optimization

Docker： https://hub.docker.com/u/garenleeasa by @garen-lee-asa Note: Add --privileged to the container run command, which grants full access including NUMA topology visibility and SYS_NICE capability in one shot.