LoongForge

LoongForge is an open-source training framework developed by the Baidu AI Cloud Baige team, built to deliver faster training for mainstream LLMs, VLMs, diffusion, and embodied models.

• High Performance — Built on Megatron-LM and torch-native backends, with deep optimizations for training speed that keep training loss aligned with the baseline.
• Easy to Use — Ready-to-run configs and launch scripts for 40+ open-source models, plus checkpoint conversion and data preprocessing tools. Supports pre-training, mid-training, SFT, and LoRA.
• Production-Proven — Open-sourced from AIAK-Training-LLM, which powers enterprise customers' proprietary models and several open-source projects. Production runs reach 5,000+ XPUs.

---

Example: training the embodied model DreamZero on LoongForge reaches 4.38× the open-source baseline's throughput, with loss curves staying aligned

🏗️ Architecture

Since optimal training strategies differ across model families and scales, LoongForge adopts a multi-backend architecture.

• Megatron Backend — For LLM, VLM, and diffusion models, extended with MoE parallelism, per-component heterogeneous parallelism, long-sequence optimizations, etc.
• torch-native Backend — For embodied models (VLA and WAM). Built on native PyTorch with DDP / ZeRO-1 / FSDP / HSDP, with deep optimizations for representative models across I/O, communication strategy, kernel efficiency, etc.

🔥 Latest News

• [2026/10] 🤖 Added VLA training support for LingBot-VLA-v2, built on the new tensor_wise distributed optimizer with Muon. --optimizer muon also runs under plain DDP for any model.
• [2026/10] 🤖 Introducing LoongForge-VLDA, a Vision-Language-Delta-Action model trained with LoongForge: 98.5% on LIBERO, 84.5% zero-shot on LIBERO-Plus, and +17–18 points over π0.5 on real robots. [project page] [paper]
• [2026/10] 🦾 Open-sourced ego2robot — a pipeline that converts first-person videos of human manipulation into dual-arm robot training data, outputting LeRobot v3.0 format across 16 robot morphologies.
• [2026/09] ✨ Added training support for GLM-5.3-flash.
• [2026/09] ✨ Added Kimi-K3 BF16 training support for both LLMs and VLMs.
• [2026/09] ⚡ Added an optimized DreamZero Wan2.2-5B FSDP recipe with cache-aware data loading, compiled attention blocks, frozen-module handling, and FSDP2 Delta-FP8 Param AllGather.
• [2026/08] 🤖 Added VLA training support for Wall-OSS-0.5, with custom fused operators for higher training throughput.
• [2026/08] 📄 Released the TAOT paper — topology-aware dynamic expert replica placement that tackles expert-parallel (EP) load imbalance in MoE training, cutting overhead by up to 74% over industry solutions, with 1.43× speedup measured on a real training case. [blog]
• [2026/08] ✨ Added training support for GLM-5.2, along with a GLM-5.2 + MoonViT custom-composition example for extending GLM with multimodal capabilities.
• [2026/08] ✨ Added training support for MiniCPM-V-4.6 and Qwen3.8-27B.
• [2026/08] 🧪 Introduced a unified evaluation module for the embodied stack, currently covering Pi0.5 / xVLA / GR00T, with more models on the way.

• [2026/07] 🐳 Unified the prebuilt Docker images — all model families (LLM / VLM / Embodied / Diffusion) now share a single image.
• [2026/07] 🤖 Released LoongForge-Embodied, a torch-native DDP/FSDP training subsystem for embodied models (Pi0.5, GR00T-N1.6/N1.7, xVLA, LingBot-VA, FastWAM, DreamZero, and Cosmos3), with up to 4.38× speedup.
• [2026/07] ✨ Added training support for DeepSeek-V4-Flash / DeepSeek-V4-Pro.
• [2026/07] ✨ Added training support for Qwen-Image-Edit-2511.
• [2026/06] 🤖 Expanded VLA coverage with GR00T N1.6; 2.3× speedup on GR00T training. [blog]
• [2026/05] ⚡ Accelerated Wan 2.2 training by 116%, and added CP and data packing support.
• [2026/05] ✨ Added training support for Kimi K2.5 / K2.6, and introduced INT4 / NVFP4 PTQ.
• [2026/05] 🎉 v0.1.0 — first official tagged release of LoongForge.
• [2026/05] 🌟 Powered the training and public release of LLaVA-OneVision-2.0.
• [2026/04] 🧩 Added training support for MiniMax-M2.7 on both NVIDIA GPU and Kunlun XPU.
• [2026/04] 🚀 LoongForge source code publicly available on GitHub. [blog]
• [2025/10] 🌟 Powered the training and public release of LLaVA-OneVision-1.5 under AIAK-Training-LLM, the predecessor of LoongForge. [blog]

🤖 Embodied Models

Data processing, training, and evaluation for Vision-Language-Action (VLA) and World-Action (WAM) models, with 1.79×–4.38× training throughput over open-source baselines across benchmarked models. [Docs] (Paper coming soon)

Data

• Unified Data Loading — A unified data-loading interface supporting mainstream formats like LeRobot (v2.0 / v2.1 / v3.0); each model's data pipeline matches its original open-source implementation.
• Feature Cache Precomputation (optional) — Precompute frozen-encoder outputs for DreamZero (VAE latents, text embeddings) and FastWAM (text embeddings) ahead of training, read directly during training to avoid re-encoding every step. [DreamZero] [FastWAM]
• Ego2Robot — Converts first-person videos of human manipulation into LeRobot v3.0 training data across 16 robot morphologies, each adapted to its own joint layout and gripper. [README]

Training

• Per-Model Deep Optimization — Custom optimizations targeting each model's bottlenecks (I/O, communication strategy, torch.compile, CUDA Graph, custom fused operators, etc.), reaching 1.79×–4.38× measured throughput over open-source baselines (see Performance).
• Precision Alignment — Training loss aligns with open-source baselines; comparison curves are already available for several models and will keep expanding alongside new model support. [Precision Comparisons]
• Multiple Distributed Strategies — Supports DDP / DDP + ZeRO-1 / FSDP / HSDP, selectable by model scale and cluster topology. [Tutorial]
• FP8 Communication Optimization (optional) — On supported NVIDIA GPUs, compresses cross-rank traffic to FP8: blockwise FP8 delta compression for FSDP2 parameter AllGather, and FP8 compression for DDP gradient all-reduce. [Usage]

Evaluation

• Unified Evaluation — Closed-loop evaluation of trained policies in the LIBERO / CALVIN / SimplerEnv / RoboTwin / ManiSkill simulators; coverage is expanding continuously. [README]

• Evaluation Results — Success rates on par with the corresponding open-source baselines, measured per model across LIBERO / RoboTwin / SimplerEnv:

🚀 LLM · VLM · Diffusion

Built on the Megatron backend, supporting large-scale pre-training and SFT for LLMs and VLMs, as well as diffusion model training. Key optimizations for each scenario:

LLM

• MoE EP Communication Optimization — Overlapped All2All / activation offload / compute, with further memory reduction beyond upstream Megatron-LM. [Usage]
• MoE Expert Load Balancing — Topology-aware dynamic replication of hot experts to balance EP workloads, with up to 74% lower overhead than industry solutions. [TAOT Paper]
• Adaptive FP8 Training — End-to-end FP8 for LLMs and VLMs with standard blockwise FP8; an optional adaptive mode picks per-operator precision by GEMM shape and efficiency. [Usage]
• Custom Fused Operators — Fused kernels like FusedDSA for DSA-style models — TileLang version open-sourced, high-performance CUDA version available on Baidu Baige platform.
• Long-Sequence Training — Scales LLM training to long sequences via Context Parallel (CP) and chunked-pipeline scheduling.

VLM

• Flexible Composition — Assemble VLMs from interchangeable ViT and LLM components (e.g. GLM-5.2 + MoonViT) straight from config — no custom model code. [Usage]
• Heterogeneous Parallelism — Independent TP / DP / recompute / freeze per model component (e.g. ViT vs. LLM) for optimal throughput and memory. [blog] [Usage]
• Decoupled Encoder-Decoder Training — Eliminates encoder-induced pipeline bubbles by separating ViT and LLM into independent tasks. [Usage]
• DP Load Balancing — Improves multi-node scaling efficiency with load-aware data redistribution that mitigates sequence-packing imbalance. [blog] [Usage]
• Flexible Data Pipeline — Energon WebDataset input for multimodal data, with both online and offline sequence packing. [Usage]

Diffusion

• Context Parallelism — Supports Ring, Ulysses, and hybrid context parallelism (CP) to split long video sequences across GPUs.
• Variable-Length Sample Packing — Packs multiple variable-length video samples into a single sequence to reduce padding waste, compatible with context parallelism. [Usage]

Compatibility

• Mcore Bridge — Supports both offline bidirectional Megatron ↔ HuggingFace conversion and online native HF load/save. [Usage]
• Heterogeneous Hardware — Native support for NVIDIA GPUs and Kunlun XPUs via a minimally-intrusive plugin design.

📖 Deep-dive: LLM · VLM

📊 Performance

Training throughput speedups over mainstream open-source baselines — each model and its baseline were benchmarked on the same machine type with the same training hyperparameters:

DeepSeek-V3.2 Lite reflects DSA operator-level optimizations and was validated on a reduced-layer configuration due to test-bed scale limits. Numbers were measured at a point in time and may evolve as implementations change on both sides.

⚡ Quick Start

1. Install

Using NVIDIA GPU as an example, pull the prebuilt image and start a container (requires the NVIDIA Container Toolkit):

docker pull loongforge/loongforge:latest
mkdir -p workspace
docker run --gpus all --ipc=host -it --rm \
  -v "$(pwd)/workspace:/workspace/data" \
  -w /workspace/LoongForge \
  loongforge/loongforge:latest bash

The example above mounts ./workspace to /workspace/data inside the container, for storing weights, datasets, and training outputs — these persist after the container exits.

For more installation options including source builds, see: Full GPU installation guide · Kunlun XPU installation guide

2. Pick a tutorial — by hardware and modality

• NVIDIA GPU: LLM · VLM · VLA & WAM (Embodied) · Diffusion
• Kunlun XPU: Kunlun XPU Tutorials

3. Launch example

Launch scripts for every supported model are already included in this repo: NVIDIA GPU scripts live under examples/, Kunlun XPU scripts under examples_xpu/ — see Supported Models for the full list.

The example below walks through DDP fine-tuning of Pi0.5 on LIBERO (single node, 8 GPUs); run all commands inside the container above.

① Prepare weights, tokenizer, and dataset (hf is the Hugging Face CLI):

hf download lerobot/pi05_base --local-dir /workspace/data/pi05_base
hf download google/paligemma-3b-pt-224 --local-dir /workspace/data/paligemma-3b-pt-224
hf download lerobot/libero_10 --repo-type dataset --local-dir /workspace/data/libero_10

② Launch training (paths are set via environment variables; defaults to 20 steps to validate the workflow):

export CHECKPOINT_PATH=/workspace/data/pi05_base
export TOKENIZER_PATH=/workspace/data/paligemma-3b-pt-224
export DATA_PATH=/workspace/data/libero_10
export OUTPUT_DIR=/workspace/data/outputs/pi05

bash examples/embodied/pi05/run_pi05_ddp_finetune.sh

Logs and TensorBoard data are saved under OUTPUT_DIR. For FSDP / ZeRO-1 launch options and LIBERO evaluation, see the Pi0.5 tutorial.

🏛️ Supported Models

Click any model for its training examples; see configs/models/ for the matching parameter configs. See the model support matrix for all variants.

[!TIP] For VLMs, also see the custom composition example, which assembles a new model from a ViT and an LLM purely through configuration.

🌟 Awesome Projects Built with LoongForge

See more on the Projects page, and if you've built something with LoongForge (or its predecessor, AIAK-Training-LLM), let us know — we'd love to feature it.

From the Community

• LLaVA-OneVision-2.0 — Next-generation multimodal model, with new VideoCaption and Spatial datasets
• Innovator-VL — Scientific multimodal LLM for advanced reasoning
• LLaVA-OneVision-1.5 — Fully open framework for democratized multimodal training
• Qianfan-VL — Domain-enhanced vision-language models for enterprise, 3B–70B parameters

From the LoongForge Team

• LoongForge-VLDA — Vision-Language-Delta-Action model for robotic manipulation: 98.5% on LIBERO, 84.5% zero-shot on LIBERO-Plus, and 17–18 points ahead of π₀.₅ on real robots.

https://github.com/user-attachments/assets/8563afca-f79a-4a45-bb4f-36e06c0ccbcb

📝 Community Blogs

• 百度&哈工大提出机器人操作新范式VLDA，从预测未来转向学习变化！
• VLA 训练如何一步步提速：以 Pi0.5 为例的 3 倍加速实践
• 百度百舸 LoongForge 团队提出 TAOT：拓扑感知传输破解大规模 MoE 训练端到端加速 1.43 倍
• 加速 GR00T N1.6 具身 VLA 模型训练！LoongForge 开源框架实现三层全链路优化，吞吐提升 2.3 倍、训练周期缩短 56.6%
• 多模态模型终于不用「抽帧看世界」？LLaVA-OneVision-2.0 全开源全帧率技术解读
• LoongForge 多模态异构并行训练加速，从问题到方案的全面解析
• 百度百舸开源全模态训练框架 LoongForge：一套代码跑通 GPU 与昆仑芯，多模态训练提速 45%
• 刷新记录！百度百舸助力 LLaVA-OneVision-1.5 高效训练

📂 Repository Layout

LoongForge/
├── loongforge/                   # Core training framework
│   ├── train/                    # Training entry points and trainers
│   │   ├── pretrain/             #   Pretraining
│   │   ├── sft/                  #   Supervised fine-tuning
│   │   └── diffusion/            #   Diffusion training
│   ├── models/                   # Unified model abstractions
│   │   ├── foundation/           #   LLM backbones
│   │   ├── encoder/              #   Vision encoders
│   │   ├── omni_models/          #   Multimodal composition (encoder + projector + decoder)
│   │   ├── diffusion/            #   Diffusion models
│   │   └── common/               #   Shared base configs, layers and utilities
│   ├── embodied/                 # Embodied (VLA + world-action) training: models, data, trainers, distributed, eval
│   ├── data/                     # Data loading and processing pipelines
│   ├── tokenizer/                # Tokenizers
│   └── utils/                    # Config registry, constants and shared helpers
├── third_party/                  # Third-party dependencies and patches
│   ├── Loong-Megatron/            # Patched Megatron-LM (git submodule)
│   └── patches/                  # TransformerEngine patches
├── configs/                      # Hydra YAML configs (models, data)
├── examples/                     # Launch scripts for NVIDIA GPUs
├── examples_xpu/                 # Launch scripts for Kunlun XPUs
├── cookbook/                     # End-to-end project recipes built on LoongForge
├── tools/                        # Standalone dev tools: checkpoint conversion, data preprocessing, etc. (see tools/README.md)
├── ops/                          # Custom fused operators (incl. open-sourced TileLang)
├── docker/                       # Dockerfiles (GPU & XPU)
├── tests/                        # Test suites (E2E, regression, unit)
└── docs/                         # Documentation

📝 Citation

If you find LoongForge helpful, please consider citing this project:

@software{LoongForge2026,
  title  = {LoongForge: A high-performance framework for training LLMs, VLMs, diffusion, and embodied models},
  author = {{The LoongForge Authors}},
  year   = {2026},
  url    = {https://github.com/baidu-baige/LoongForge}
}

If you find TAOT helpful, please consider citing our paper:

@article{zhang2026taot,
  title   = {{TAOT}: Topology-Aware Optimal Transport for Dynamic Expert Replica Placement in {MoE} Training},
  author  = {Zhang, Lingyun and Zhang, Henghua and Gu, Shilei and Mo, Kai and Han, Shuai and Li, Shiyong and Wang, Yanpeng and Shen, Dou},
  journal = {arXiv preprint arXiv:2608.03676},
  year    = {2026},
  url     = {https://arxiv.org/abs/2608.03676}
}

If you find LoongForge-VLDA helpful, please consider citing our paper:

@article{zhang2026vlda,
  title   = {{Vision-Language-Delta-Action}: Learning Compact Deltas as Actionable Transition Interfaces for Robotic Manipulation},
  author  = {Zhang, Lingsen and Li, Wei and He, Jie and Zhang, Henghua and Mo, Kai and Li, Shiyong and Wang, Yanpeng and Shao, Rui},
  journal = {arXiv preprint arXiv:2610.25874},
  year    = {2026},
  url     = {https://arxiv.org/abs/2610.25874}
}

🤝 Contributing

We warmly welcome community contributions — bug reports, feature proposals, and PRs alike. Please read our Contributing Guidelines before submitting.

Thanks to all our contributors:

💬 Contact Us

Channel	What it's for
GitHub Issues	Bug reports, usage questions, and feature requests
Developer Communities	WeChat group, Xiaohongshu, and more
Email	Enterprise adoption, large-scale deployment, partnership, or any other topic

🙏 Acknowledgments

LoongForge stands on the shoulders of the open-source community. Its Megatron backend builds on NVIDIA's Megatron-LM, and the project also draws on HuggingFace Transformers, LLaMA-Factory, Megatron-Bridge, LeRobot, and the official implementations of the models we support (e.g. OpenPI, NVIDIA Isaac GR00T).

We also thank the LINUX DO community for providing a welcoming space for technical discussion, and AtomGit for hosting LoongForge's official mirror.

📄 License

LoongForge is released under the Apache License 2.0; some files derive from third-party projects — see their file headers.