登录

Automodel

NVIDIA-NeMoApache-2.0活跃维护

基于 PyTorch 原生分布式训练的 LLM/VLM 训练库，开箱即用兼容 Hugging Face，方便微调 Llama、Gemma、GLM 等主流大模型。

🚀 Pytorch Distributed native training library for LLMs/VLMs with OOTB Hugging Face support

v0.7.0发布于 2026-10-21 23:48 UTC1,080star +1 / 7 天377fork同步于 2026-11-10 05:56 UTC
分布式训练大模型微调PyTorchHugging Face多模态
对比 GitHub 官网agentdeepseek-v4deepseek-v4-flashfinetuninggemma3

怎么上手

由 README 提炼，执行前请确认
原文 
uv venv
uv sync --frozen
uv run python -c "import nemo_automodel; print('NeMo AutoModel ready')"
硬件要求按 README 说明

README 未给出具体显存要求；示例通过 docker run --gpus all 在 GPU 上运行，并支持多节点 SLURM 集群，未说明 CPU 运行方式。

支持 Apple Silicon需要 CUDA多卡 / 张量并行
下载：源码 zip·GitHub Releases（安装包）本站不镜像安装包

项目说明

由 README 自动翻译提炼
没人工点评？写下你的经验

NVIDIA 出品的 PyTorch 原生分布式训练库，开箱即用微调 Hugging Face 上的 LLM 与 VLM

基于 PyTorch 原生分布式训练的 LLM/VLM 训练库，开箱即用兼容 Hugging Face，方便微调 Llama、Gemma、GLM 等主流大模型。

核心能力

  • 用 YAML 配方一键启动 SFT、PEFT、预训练任务
  • 原生支持 FSDP2、专家并行、流水线并行与上下文并行
  • 直接加载 Hugging Face 模型与公开数据集，无需 token
  • 提供 LoRA 等参数高效微调配方，降低显存占用
  • 支持 EAGLE-4 投机解码草稿模型训练与导出

容易踩的坑

VLM 配方需额外安装 vlm 与 vlm-media extra，否则报 qwen_vl_utils 未安装；CUDA 相关依赖（Transformer Engine、Mamba、bitsandbytes）需单独选装 cuda 或 cuda_source extra；多节点训练需自行复制并修改 slurm.sub 中的集群配置。

内容由模型根据 README 生成，可能不完整或不准确，以官方文档为准。看原文 README

最近发版

5 个版本v0.7.0
全部 release
  1. v0.7.0NVIDIA NeMo-Automodel 0.7.0最新正式版2026-10-21 · 2026-10-21 23:48 UTC

    NeMo AutoModel 26.10 Release Notes

    Highlights

    • Expanded model coverage. New training paths for DeepSeek-V4.1-Flash, GLM-5.3-Flash, Qwen3.8-Flash-Next, MiMo-V2.5-Pro and MiMo-V2.6, and additional vision-language models.
    • More MoE training options. Mixture-of-Kittens (MoK) BF16 and MXFP8 experts, compact HybridEP routing, and MXFP4-resident DeepSeek-V4-Flash experts for LoRA.
    • Speculative and discrete-diffusion training. DFlash 2, additional Kimi K3 and DeepSeek-V4.1 draft-training paths, streaming EAGLE-3 feature components, I-DLM training, and SCDD training and sampling.
    • Text-to-image fine-tuning. HunyuanImage-3.0 with expert parallelism and Qwen-Image-2.1 with full-parameter and LoRA recipes.
    • Training and integration improvements. Native FlashAttention-4, explicit packing contracts, weighted domain mixtures, lower-memory checkpoint loading, and public registration of custom model architectures.

    New and Expanded Model Support

    LLM and MoE

    • DeepSeek-V4.1-Flash: Native model and checkpoint conversion, attention and expert LoRA, packed SFT and context parallelism, H100-80GB recipes, and MedPix vision SFT. The release includes the CP8 gradient-synchronization fix. (model guide, #3855, #3872, #3879, #3962, #4221)
    • GLM-5.3-Flash: Native hybrid KDA/DSA model, vision processing, distributed checkpoint loading, and packed context-parallel training. The MedPix recipe uses EP72/CP2 on 72 H100 GPUs; sparse attention combines FlashMLA forward with cuDNN backward. TP, PP, video training, and full-model single-GPU loading are outside this model's supported path. GLM-5.2 and GLM-5.3 also gain cuDNN DSA training backends. (model guide, #3699, #3127, #3747)
    • Qwen3.8: Adds 27B fine-tuning recipes and native Flash-Next language-only training with a sharded Engram embedding table, sparse QSA attention, packed sequences, and contiguous context parallelism. Flash-Next's QSA indexers remain frozen; its checkpoint MTP head is not loaded. TP and PP are disabled in the documented Flash-Next configurations. (model guide, #3560, #3690)
    • MiMo: Adds MiMo-V2.5-Pro, MiMo-V2.6, and MiMo-V2.6-Pro support. The MiMo-V2.6-Pro-RL recipes cover packed 64K Tulu3 text, MedPix full fine-tuning, and MedPix LoRA on GB200. Keep expert-parallel groups within an NVLink domain. (recipe guide, #2514, #3992, #4054)
    • Additional architectures: Dense Nemotron-H Nano 4B, a Laguna XS 2.1 training recipe, and encoder-decoder fine-tuning through NeMoAutoModelForSeq2SeqLM, including a T5-small SQuAD example. The seq2seq path does not wire up CP, TP, PP, or sequen
  2. v0.6.0NVIDIA NeMo-Automodel 0.6.02026-08-26 · 2026-08-26 17:54 UTC
    查看更新说明
    Highlights
    • Frontier-scale MoE and multimodal models. Train Moonshot AI's Kimi K3 (2.8T/104B active), Thinking Machines Lab's Inkling (975B/41B active), MiniMax AI's MiniMax-M3 VL (428B/22B active), Poolside's Laguna, and Zhipu AI's GLM-5.2.
    • Long-context training. Shard packed multi-document sequences with block-diagonal variable-length context parallelism, and shard vision towers by frame to remove the redundant vision compute of a CP fold; recipes reach 128K tokens at EP8/CP32.
    • New attention and kernel backends. Select FlashAttention 3 and 4, MagiAttention, FFPA for headdim=512 layers, QuACK linear/RMSNorm/RoPE kernels, and scoped partial CUDA graphs for MoE.
    • Resilient checkpointing. Checkpoint on preemption signals, write to msc:// cloud directories, consolidate async saves in the background, bound checkpoint retention, and resume past an interrupted save.
    • Diffusion. Fine-tune and generate with LTX-2.3 joint video+audio and Qwen-Image-Edit-2511, and run context parallelism through the diffusers ContextParallelConfig API.
    New Hardware and Precision Support
    • FP8 draft training. Extends the SFT recipes' top-level fp8: block to every speculative recipe, swapping the draft's nn.Linear layers to torchao Float8Linear on SM89 and newer, with emulate: true for older GPUs. (#2963)
    • Quantized checkpoint ingestion. Dequantizes MXFP4 dequantization for Kimi K3 and MXFP8 dequantization for MiniMax-M3 for BF16 training, keeps Kimi-K2.5-VL LoRA keys out of INT4 quantization on save, and fixes a TileLang boolx8 backward codegen failure on the DeepSeek-V4 path. (#3259, #3295, #3467)
    New and Expanded Model Support
    LLM and MoE
  3. v0.5.0NVIDIA NeMo-Automodel 0.5.02026-07-02 · 2026-07-02 19:58 UTC
    查看更新说明

    Release Notes

    • Highlights

      • Blackwell and FP8 support. Run DeepSeek-V4 with TileLang kernels, train MoE models with MXFP8 on GB200, and use context parallelism for Qwen3.5/3.6 MoE VLM. More details in our performance summary
      • Agent SFT. Train Qwen2.5-3B on multi-turn, function-calling ChatML or ShareGPT data with full-SFT and LoRA recipes. The data adapter preserves tool calls, tool responses, and optional reasoning_content.
      • Speculative decoding. Train EAGLE 1/2/3/3.1, P-EAGLE, and DFlash drafts for Llama, Phi, Qwen3, Qwen3-MoE, GPT-OSS, and Nemotron Nano targets.
      • dLLM. Train DFlash drafts and fine-tune LLaDA2 or Nemotron-Labs-Diffusion with the new discrete-diffusion workflows.
      • Diffusion. Fine-tune and generate with FLUX.2-dev, Qwen-Image, and Wan 2.2 T2V-A14B; update FLUX, Hunyuan, and Wan 2.1 recipes for LoRA and throughput.

    • New Hardware and Precision Support

      • GB200 MXFP8 MoE. Adds Transformer Engine MXFP8 grouped-expert training for Qwen3-MoE-30B, GPT-OSS-120B, and Qwen3-MoE-235B, and standardizes the bf16 expert backend on Transformer Engine grouped linear layers.

    • New and Expanded Model Support

    • LLM and MoE

      • DeepSeek-V4 Flash & Pro. Adds a native model, checkpoint adapter, HellaSwag full-SFT and LoRA recipes, a packed-sequence recipe, and large-scale TileLang PP/EP recipes. The implementation covers hash routing, hyper-connections, sparse/compressed attention, FP4/FP8 checkpoint loading, and multi-token prediction.
      • ERNIE 4.5. Adds HellaSwag recipes for the 0.3B dense and 21B-A3B MoE checkpoints.
      • MiMo-V2-Flash. Adds a HellaSwag fine-tuning recipe and a native model implementation with checkpoint conversion.
      • Ling 2.0. Adds Mini, Flash, and 1T model paths with SFT, LoRA, HellaSwag, SQuAD, and pipeline-parallel recipes.
      • Hy3-preview. Adds a 295B MoE DeepEP fine-tuning and LoRA recipe, with model-specific expert-bias handling.
      • Hy-MT2-30B-A3B. Adds a supervised fine-tuning recipe and native model support.
      • MiniMax-M2.7. Adds pipeline-parallel full-SFT and LoRA recipes.
      • Falcon-H1. Adds 0.5B, 1.5B, and 7B full-SFT recipes, plus 7B and 34B PEFT recipes.
      • MiniCPM5-1B. Adds full-SFT and PEFT recipes.
      • Nemotron-3-Ultra-550B-A55B. Adds a 128-GPU throughput benchmark and a 128-GPU full-SFT recipe using a real router, THD sequence packing, and a repeated MTP head.
      • Qwen3.5/3.6 dense and MoE. Adds 27B dense and 35B-A3B VLM SFT/LoRA recipes, a 35B EP8/CP2 long-context recipe, MTP support, and a native dense Qwen3.5 backbone that keeps its SSM-gate parameters in fp32.

    • VLM

      • Qwen3.6 VLM. Adds 27B dense and 35B-A3B SFT/LoRA recipes, including a 4K MedPix EP8/CP2 recipe for the MoE model.
      • Mistral Medium 3.5. Adds a 128B MedPix full-finetuning recipe and a matching LoRA recipe.
      • LLaVA-OneVision 1.5. Adds a 4B full-finetuning recipe and an 8B LoRA recipe.
      • VLM QLoRA. Enables the existing quantization: config block for VLM finetuning, including BitsAndBytes 4-bit NF4 base weights with LoRA adapters.
      • VLM knowledge distillation. Adds a Qwen3.5-VL 9B-to-4B KD recipe with a chunked KD loss, frozen vision/audio towers, and separate CE/KD metrics.
      • Step-3.7-Flash. Adds support for Stepfun AI’s MoE VLM model Step-3.7-Flash with SFT and LoRa recipes using EP + PP across 16 nodes.

    • Omni and Multimodal

      • Bagel. Adds the Bagel model, dataset, pretraining recipe, finetuning recipe, EMA support, checkpoint adapter, and distributed initialization. The one-node examples use token budgets validated for parity runs.
      • Qwen2.5-Omni ASR. Adds AMI and multi-language SFT recipes for the 3B and 7B checkpoints.
      • Qwen3-Omni ASR. Adds AMI and multi-language audio SFT recipes.
      • Nemotron-Omni vision. Adds a dynamic-resolution image path aligned with
  4. v0.4.0NVIDIA NeMo-Automodel 0.4.02026-04-28 · 2026-04-28 20:04 UTC
    查看更新说明

    Release Notes

    • Highlights

      • Expanded VLM line-up: Gemma 4, Mistral 4, Qwen3.5 VL
      • Diffusion and discrete-diffusion LLM (new tracks)
      • NeMo Retriever – bi-encoder + cross-encoder / reranker
      • Knowledge Distillation scaled to TP > 1 and PP (Sepehr Sameni)
      • MoE infrastructure deepening – UCCL-EP, HybridEP, grouped_mm
      • SkyPilot launcher backend (Aditya Saxena, community)
      • End-to-end checkpoint + convergence robustness framework

    • Model Support – newly supported families in r0.4.0

      • LLM
        • GLM 5 (#1372)
        • GLM 5.1 (#1720)
        • MiniMax M2.5 (#1280)
      • VLM / OMNI
        • Gemma 4 family – 2B, 4B, 31B, 26B-A4B MoE (#1658, #1660, #1731)
        • Mistral Small 4 (#1556)
        • Qwen3.5 VL dense – 4B, 9B (#1427)
        • Qwen3.5 VL MoE – 35B (#1373)
      • Diffusion
        • Flux T2I, Hunyuan T2V, Wan 2.1 T2V (see "Diffusion" section)
      • Discrete diffusion LLM
        • LLaDA (see "Discrete Diffusion LLM" section)

    • Diffusion – new track in r0.4.0

      • HuggingFace Diffuser integration
      • r0.4.0 ships full pretrain / finetune / generate pipelines with LoRA support for diffusion models (T2V, T2I)
      • Wan integrated with multi-resolution DataLoader (#1475)
      • Inference utility for diffusion (#1491)
      • LoRA for diffusion (#1653, Linnan Wang)
      • Diffusion processor registry (#1379)
      • Models / recipes shipped
        • Flux T2I – pretrain, SFT, LoRA, generate
        • Hunyuan T2V – SFT, LoRA, generate
        • Wan 2.1 T2V – pretrain, SFT, LoRA, generate
      • Documentation guides for dataset preprocessing and finetuning.

    • Discrete Diffusion LLM (dLLM) – new track in r0.4.0

      • Discrete diffusion LLM SFT support added (#1665)
      • LLaDA SFT recipe (#1672)
      • dLLM generation pipeline (#1692)

    • NeMo Retriever (bi-encoder + cross-encoder)

      • Refactored cross-encoder / reranker training loop (new in r0.4.0) — (#1449).
      • Bi-encoder datasets can be loaded directly from the HuggingFace Hub (#1380)
      • Bi-encoder masking + consistent attn_implementation default (#1349)
      • Resolve retrieval dataset corpus paths relative to training file (#1367)
      • Docs: docs/guides/retrieval/finetune.md

    • Knowledge Distillation — Sepehr Sameni

      • Enable TP > 1 in KD (#1297)
      • TP-aware KDLoss with distributed softmax + T² scaling (#1499)
      • Pipeline-parallelism support for KD (#1500)

    • Parallelism / Performance / Train-loop

      • FSDP2
        • FSDP2 weight prefetching + async TP optimization (#1711)
      • Context Parallel
        • Qwen3.5 dense & MoE CP (#1710, #1560 — alexchiu / Zhaopeng Qiu)
        • Mamba CP for hybrid Nemotron v3 (#1441)
        • 3D mRoPE position_ids sharding under CP (#1482)
        • CP attention-mask hooks for dense / non-TE (#1470)
      • Pipeline Parallel
        • PP shape-inference optimization + pp_seq_len field in PipelineConfig (#1195, #1390)
        • Variable length for PP (#1689 – Zhiqi Li & Hemil Desai)
      • Activation checkpointing
        • Gradient_checkpointing overhead reduction i[n transformers 5.3 (#1621 — Yuki Huang)
      • MoE infrastructure
        • UCCL-EP alternative dispatcher (#1635 – Zhiqi Li & Hemil Desai)
        • HybridEP (#1333, #1666)
        • DeepEP-on-H100 RDMA fallback detection (#1275 — Piotr Żelasko)
        • torch._grouped_mm expert backend (#1228)
        • TE FusedAdam QuantizedTensor compatibility patch (#1417)
        • MoE LoRA rank scaling + torch_mm path (#1300, #1392)
        • Expert / diversity metrics (#1232, #1506), top-k utilization (#1418)
        • Packed sequences for MoE with EP+PP (#1685)
      • FlashOptim integration (#1492)
      • Scheduler-driven python GC (#1391)
      • fp32 RMSNorm backend + cast_model_to_dtype for improved stability (#1493)
      • Native Comet ML experiment tracking (#1411, Logan Vegna, community)
      • Added .generate() with KV-cache for Nemotron v3 (#1332, Piotr Żelasko)
      • Added output_hidden_states for NemotronHForCausalLM (#1386, Desh Raj)

    • Launcher & CLI

      • SkyP
  5. v0.3.0NVIDIA NeMo-Automodel 0.3.02026-03-02 · 2026-03-02 18:57 UTC
    查看更新说明

    Release Notes

    • Hugging Face Transformers v5
      • Upgraded to Transformers v5 with a new device-mesh-only model initialization API
      • Drop-in API compatibility: NeMoAutoModelForCausalLM, NeMoAutoModelForImageTextToText, NeMoAutoModelForSequenceClassification, NeMoAutoTokenizer mirror the standard Transformers Auto* APIs
    • Model Support
      • LLM
        • DeepSeek V3.2
        • Step 3.5 Flash
        • MiniMax M2
        • Nemotron-3-Nano v3 (30B-A3B)
        • Nemotron Flash 1B
        • GLM 4.7, GLM 4.7 Flash
        • Devstral-Small-2-24B
        • FunctionGemma (tool-calling)
        • Ministral3 (3B, 8B, 14B)
      • VLM & OMNI
        • Kimi-VL-A3B
        • Kimi K2.5 VL
        • Nemotron Parse v1.1
        • Qwen3 VL MoE (30B, 235B)
        • Ministral3 VLM (3B, 8B, 14B)
      • Embedding & Retrieval
        • NeMo Biencoder training pipeline with Llama-Embed-Nemotron-8B support
        • Hard negative mining for retrieval training
    • PEFT
      • DoRA (Weight-Decomposed Low-Rank Adaptation)
      • LoRA for MoE models (DeepSeek MoE, Qwen MoE)
      • LoRA support for Biencoder
    • Parallelism
      • Pipeline parallelism for VLMs
      • GroupedExpertsTE backend (prerequisite for MoE FP8)
      • TE RoPE fusion for custom MoE models
      • Norm fusion and RoPE cache for dense models
    • Dataset support for
      • VLM multi-turn chat
      • Inline text dataset format for retrieval
      • Databricks DeltaLake streaming dataset
      • Parquet file support for Megatron dataset preprocessing
      • xLAM tool-calling dataset
      • Answer-only masking in ColumnMappedDataset
    • Checkpointing & logging
      • Streaming safetensors writer for reduced peak memory during checkpoint saving
      • Explicit restore_from for checkpoint loading (replaces auto-loading behavior)
      • Checkpoint custom model code files alongside weights
      • Configurable remote logging frequency via step_scheduler
    • Optimizers
      • Dion optimizer (Muon/orthogonal family)
    • Performance
      • Faster FP8 dequant kernels for DeepSeek V3
      • Meta device initialization enabled by default for reduced peak memory during model setup
      • Combine projection refactor for dense models
      • SDPA as default attention backend when FlashAttention is unavailable
    • Misc Infrastructure
      • Databricks integration (DeltaLake datasets, Unity Catalog checkpointing, DBFS consolidation)
      • Nsys profiling support with model layer name scoping
      • Environment variable dereferencing in YAML configs
      • Improved import time
    • Resolved from 0.2.0
      • MoE perf regression with DeepSeek V3 (resolved via faster FP8 dequant kernels and GroupedExpertsTE backend)
      • PEFT (LoRA) support for MoE models (now available)
      • Validation for packed sequences with TE attention (fixed)
      • Validation support for pipeline parallelism (added)
    • Known Issues
      • Qwen3-next unsupported on blackwell due to FLA lacking support.
      • TransformerEngine’s Fused Adam is not working with DTensor, resolved in the upcoming version.
      • LoRA with TE backend is not supported

    Community Contributions

    • We gratefully acknowledge the following contributions from the OSS community:
      • @onel (Andrei Onel) – Founder of @askmanu, Dublin
        • docs: Add documentation for the new ChatDataset class (#990)
        • docs: Added MLflow guide (#1045)
        • docs: Created guide for quantization aware training (#1088)
        • docs: Documentation update for release 0.2.0 (#1041)
        • docs: Update docs/guides/dataset-overview.md (#1145)
      • @ooooo-create – Community contributor (PaddlePaddle ecosystem)
        • fix: Add DeepEP fallback logic and tests (#1000)
        • fix: respect trust_remote_code when building AutoConfig (#1007)
      • @Sparlitu – Community contributor
        • fix: leave num_epochs unset if max_steps is specified (#1107)
      • @yuhezhang-ai (Yuhe Zhang) – Engineer at Polarr
        • feat: Support LoRA for custom MoEs (#1010)
        • fix: sequence classification metric and training bugs #780 (#841)
      • @therealnaveenkam

用户点评

登录用户的真实使用记录，可按版本和硬件筛选
共 0 条

    还没有人点评。第一个说出真实体验的人最有价值。

    README 原文

    未翻译
    在 GitHub 打开

    🚀 NeMo AutoModel

    codecov CICD NeMo Python 3.11+ Contributions Welcome GitHub Stars

    📖 Documentation • 🔥 Ready-to-Use Recipes • 💡 Examples • Model Coverage • Performance • 🤝 Contributing

    📣 News and Discussions

    Overview

    Nemo AutoModel is a Pytorch DTensor‑native SPMD open-source training library under NVIDIA NeMo Framework, designed to streamline and scale training and finetuning for LLMs, VLMs, diffusion models, and retrieval models. Designed for flexibility, reproducibility, and scale, NeMo AutoModel enables both small-scale experiments and massive multi-GPU, multi-node deployments for fast experimentation in research and production environments.

    What you can expect:

    • Hackable with a modular design that allows easy integration, customization, and quick research prototypes.
    • Minimal ceremony: YAML-driven recipes; override any field using CLI.
    • High performance and flexibility with custom kernels and DTensor support.
    • Seamless integration with Hugging Face for day-0 model support, ease of use, and wide range of supported models.
    • Efficient resource management using Kubernetes and Slurm, enabling scalable and flexible deployment across configurations.
    • Documentation with step-by-step guides and runnable examples.

    Why PyTorch Distributed and SPMD

    • One program, any scale: The same training script runs on 1 GPU or 1000+ by changing the mesh.
    • PyTorch Distributed native: Partition model/optimizer states with DeviceMesh + placements (Shard, Replicate).
    • SPMD first: Parallelism is configuration. No model rewrites when scaling up or changing strategy.
    • Decoupled concerns: Model code stays pure PyTorch; parallel strategy lives in config.
    • Composability: Mix tensor, sequence, and data parallel by editing placements.
    • Portability: Fewer bespoke abstractions; easier to reason about failure modes and restarts.

    Table of Contents

    TL;DR: SPMD turns “how to parallelize” into a runtime layout choice, not a code fork.

    Feature Roadmap

    ✅ Implemented on main | 🔜 Planned

    For released features, see the release notes. Docker examples target the 26.10 container.

    High-throughput scalable training

    • ✅ PyTorch DTensor-native SPMD training Same training script can scale from 1 GPU to large multi-node jobs by changing the device mesh/config.
    • ✅ Composable Parallelism - PyTorch native FSDP2, HSDP, TP, CP, SP and PP for distributed training.
    • ✅ Optimized kernels - Uses NVIDIA-oriented kernel paths such as Transformer Engine, DeepEP, FlexAttn, TorchSDPA, fused attention, rotary embeddings, Triton, and optional kernel patches.
    • ✅ MoE acceleration - Includes MoE routing and DeepEP integration, plus expert-parallel configurations used in DeepSeek, Qwen MoE, GPT-OSS, and Nemotron MoE benchmarks.
    • ✅ FP8 and mixed precision - FP8 support with torchao and Transformer Engine.
    • ✅ MXFP8 MoE training - Transformer Engine and torchao MXFP8 grouped-expert training on GB200.
    • ✅ Activation checkpointing - Trades recomputation for lower activation memory, especially useful with FSDP and memory-efficient losses.
    • ✅ Memory-efficient loss - Linear-Cut / fused linear cross entropy avoids materializing full logits for the loss, reducing output-layer memory pressure.
    • ✅ Sequence packing - Packs variable-length examples together to reduce padding compute and improve GPU utilization.
    • ✅ FlashAttention packed-sequence support - Packed masks can feed variable-length FlashAttention paths using per-document cu_seqlens.
    • ✅ DCP - Supports PyTorch DCP and SafeTensors, sharded and consolidated layouts, merge/reshard utilities, and Hugging Face-compatible outputs.
    • ✅ Async checkpointing - Can write checkpoints in the background to reduce training stalls caused by I/O.
    • ✅ Dion, Muon, and Muown optimizers - Distributed optimizer integrations with typed recipe configuration.
    • ✅ Environment Support - SLURM, interactive, SkyPilot, and Kubernetes (via SkyPilot) launchers.

    SOTA algorithms

    • ✅ Pre-training - Support for model pre-training, including DeepSeekV3.
    • ✅ Learning Algorithms - SFT (Supervised Fine-Tuning), PEFT (LoRA, QLoRA), and QAT (Quantization-Aware Training).
    • ✅ Knowledge distillation - Support for knowledge distillation with LLMs.
    • ✅ VLM knowledge distillation - Chunked KD loss and a Qwen3.5-VL teacher-student recipe.
    • ✅ Speculative decoding training - Train target-aligned EAGLE-1/2/3/4, P-EAGLE, DFlash, DFlash 2, and DSpark drafters for faster verified generation.

    Model Coverage and 🤗 Ecosystem compatibility

    • ✅ Transformers v5 🤗 - Built on latest transformers with device-mesh driven parallelism.
    • ✅ 🤗 HuggingFace Integration - Works with dense models (e.g., Qwen, Llama3, etc) and large MoEs (e.g., DSv3, DSv4).
    • ✅ VLM - Finetuning for VLMs (Qwen2.5/3/3.5/3.6 VL, Gemma-3/3n/4 VL, Mistral 3.5/4, LLaVA-OneVision-1.5, Kimi-VL, etc.).
    • ✅ Omnimodal - Finetuning for omnimodal MoE models (Nemotron-3-Nano-Omni, Qwen3-Omni).
    • ✅ Diffusion - Pretraining and LoRA finetuning for image/video diffusion models (Qwen-Image, Qwen-Image-2.1, FLUX, FLUX 3 Image Klein Base, Wan2.1, Wan2.2-T2V-A14B, Hunyuan).
    • ✅ dLLM - Discrete diffusion LM finetuning (LLaDA, LLaDA2, Nemotron-Labs-Diffusion, DiffusionGemma).
    • ✅ Retrieval - Bi-encoder and cross-encoder training with in-batch negative sampling.
    • ✅ Extended MoE support - GPT-OSS, Kimi K3, Qwen3 / Qwen3.5 / Qwen3.6 MoE, Qwen-next, MiniMax-M2.x, GLM-4.7 / GLM-5 / GLM-5.1 / GLM-5.2 / GLM-5.3 / GLM-5.3-Flash, DeepSeek V3.2 / V4 / V4-Flash, ERNIE 4.5, MiMo-V2-Flash / MiMo-V2.5-Pro, Ling 2.0, Hy3-preview.

    Agentic Development and UX

    • ✅ Agent-friendly skills - Curated skills/ for common dev tasks (recipe runs, model onboarding, CI).

    Planned

    • 🔜 Unified Engine API and recipes - Introduce a common engine and consolidate the LLM and VLM recipe paths.
    • 🔜 Composable component configuration - Complete the typed config and .build() refactor across data and remaining components.
    • 🔜 Packed long-context training with CP - Combine THD sequence packing with context parallelism, including DeepSeek V4 coverage.
    • 🔜 Kernel and runtime upgrades - Add partial CUDA graphs, evaluate FlashAttention 3/4, and upgrade to DeepEP v2.
    • 🔜 Multimodal retrieval - Expand vision-language retrieval datasets and models and improve retriever performance.
    • 🔜 Convergence and regression testing - Add automated loss, evaluation, and performance gates to NeMo CI.
    • 🔜 Transformers 5.12 alignment - Upgrade the Transformers integration and restore affected model parity coverage.
    • 🔜 Recipe consistency - Align diffusion recipes with the shared AutoModel recipe and configuration patterns.

    Getting Started

    Run a 10-step Qwen3-0.6B fine-tuning example on two GPUs with the container:

    mkdir -p checkpoints
docker run --gpus all -it --rm --shm-size=8g \
  -v "$PWD/checkpoints:/opt/Automodel/checkpoints" \
  nvcr.io/nvidia/nemo-automodel:26.10.00 \
  automodel /opt/Automodel/examples/llm_finetune/qwen/qwen3_0p6b_hellaswag.yaml \
  --nproc-per-node 2 \
  --step_scheduler.max_steps 10

    The model and HellaSwag dataset are public; no Hugging Face token is required. Checkpoints are saved to the mounted checkpoints directory.

    For source installation, use uv for reproducible Python environments.

    # Setup environment before running any recipes
uv venv

# Choose ONE:
uv sync --frozen  # LLM recipes (default)
# uv sync --frozen --extra vlm --extra vlm-media  # VLM recipes (Qwen/Mistral/Omni need vlm-media for video/vision; fixes: ImportError: qwen_vl_utils is not installed)
# uv sync --frozen --extra cuda  # Optional CUDA deps (e.g., Transformer Engine, Mamba SSM)
# uv sync --frozen --extra cuda_source  # Optional bitsandbytes dependency
# uv sync --frozen --extra all  # Most optional deps (includes `vlm` and `cuda`; NOTE: excludes media — add --extra media for video/image decode)
# uv sync --frozen --all-extras  # Everything (includes `fa`, `moe`, `media`, etc.)

# One-off runs (examples):
# uv run --extra vlm <command>
# uv run --extra cuda <command>

uv run python -c "import nemo_automodel; print('NeMo AutoModel ready')"

    Run a Recipe

    All recipes are launched via the automodel CLI (or its short alias am). Each YAML config specifies the recipe class and all training parameters:

    # LLM example: multi-GPU fine-tuning of Qwen3-0.6B with FSDP2 (no HF token required)
automodel examples/llm_finetune/qwen/qwen3_0p6b_hellaswag.yaml --nproc-per-node 8

# VLM example: single-GPU fine-tuning (Gemma-3-VL) with LoRA
automodel examples/vlm_finetune/gemma3/gemma3_vl_4b_cord_v2_peft.yaml

# Both commands also work with uv run:
uv run automodel examples/llm_finetune/qwen/qwen3_0p6b_hellaswag.yaml --nproc-per-node 8

    [!TIP] NeMo-Run submission: The cli extra adds NeMo Run to the base package: uv pip install "nemo-automodel[cli]". It is additive; the base package still installs its core training dependencies, including PyTorch.

    LLM Pre-training

    LLM Pre-training Single Node

    We provide an example SFT experiment using the FineWeb dataset with a nano-GPT model, ideal for quick experimentation on a single node.

    automodel examples/llm_pretrain/nanogpt_pretrain.yaml --nproc-per-node 8

    LLM Supervised Fine-Tuning (SFT)

    We provide an example SFT experiment using the HellaSwag dataset.

    LLM SFT Single Node

    The default SFT configuration is set to run on a single GPU. To start the experiment:

    automodel examples/llm_finetune/qwen/qwen3_0p6b_hellaswag.yaml

    This fine-tunes Qwen3-0.6B on HellaSwag using a single GPU. The model and dataset are public; no Hugging Face token is required.

    To use multiple GPUs on a single node, add the --nproc-per-node argument:

    automodel examples/llm_finetune/qwen/qwen3_0p6b_hellaswag.yaml --nproc-per-node 8

    LLM SFT Multi Node

    To launch on a SLURM cluster, copy the reference sbatch script and adapt it to your cluster:

    cp slurm.sub my_cluster.sub
# Edit my_cluster.sub — change CONFIG, #SBATCH directives, container, mounts, etc.
sbatch my_cluster.sub

    All cluster-specific settings (nodes, GPUs, partition, container, mounts) live in your sbatch script. NeMo-Run (nemo_run:) sections are also supported -- see our cluster guide for details.

    LLM Parameter-Efficient Fine-Tuning (PEFT)

    We provide a PEFT example using the HellaSwag dataset.

    LLM PEFT Single Node

    # Memory-efficient SFT with LoRA
automodel examples/llm_finetune/qwen/qwen3_0p6b_hellaswag_peft.yaml

# Override any YAML parameter via the command line:
automodel examples/llm_finetune/qwen/qwen3_0p6b_hellaswag_peft.yaml \
  --step_scheduler.local_batch_size 16

    [!NOTE] Launching a multi-node PEFT example uses the same sbatch slurm.sub workflow as the SFT case above.

    VLM Supervised Fine-Tuning (SFT)

    We provide a VLM SFT example using Qwen2.5-VL for end-to-end fine-tuning on image-text data.

    VLM SFT Single Node

    # Qwen2.5-VL on 8 GPUs
automodel examples/vlm_finetune/qwen2_5/qwen2_5_vl_3b_rdr.yaml --nproc-per-node 8

    VLM Parameter-Efficient Fine-Tuning (PEFT)

    We provide a VLM PEFT (LoRA) example for memory-efficient adaptation with Gemma3 VLM.

    VLM PEFT Single Node

    # Gemma-3-VL PEFT on 8 GPUs
automodel examples/vlm_finetune/gemma3/gemma3_vl_4b_medpix_peft.yaml --nproc-per-node 8

    Supported Models

    NeMo AutoModel provides native support for a wide range of models available on the Hugging Face Hub, enabling efficient fine-tuning for various domains. Below is a small sample of ready-to-use families (train as-is or swap any compatible 🤗 causal LM), you can specify nearly any LLM/VLM model available on 🤗 hub:

    DomainModel FamilyModel IDRecipes
    LLMGPT-OSSGPT-OSS-20BSFT
    GPT-OSS-120BSFT
    LLMDeepSeekDeepSeek-V3Pretrain
    LLMKimi K3moonshotai/Kimi-K3SFT
    LLMGLM-5.3zai-org/GLM-5.3SFT
    LLMMoonlightMoonlight-16B-TEPretrain, SFT
    LLMNemotron-Hnvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16LoRA
    LLMLing 2.0inclusionAI/Ling-mini-2.0LoRA SFT, SFT
    inclusionAI/Ling-flash-2.0LoRA SFT, SFT
    inclusionAI/Ling-1TLoRA SFT, SFT
    LLMERNIE 4.5baidu/ERNIE-4.5-0.3B-PTSFT
    baidu/ERNIE-4.5-21B-A3B-PTSFT
    LLMMiMo V2 FlashXiaomiMiMo/MiMo-V2-FlashSFT
    LLMMiMo-V2.5-ProXiaomiMiMo/MiMo-V2.5-ProSFT
    LLMLLaMAmeta-llama/Llama-3.2-1BSFT, PEFT
    meta-llama/Llama-3.2-3B-InstructSFT, PEFT
    meta-llama/Llama-3.1-8BFP8
    meta-llama/Llama-3.3-70B-InstructSFT, PEFT
    LLMMistralmistralai/Mistral-7B-v0.1SFT, PEFT, FP8
    mistralai/Mistral-Nemo-Base-2407SFT, PEFT, FP8
    mistralai/Mixtral-8x7B-Instruct-v0.1SFT, PEFT
    LLMQwenQwen/Qwen2.5-7BSFT, PEFT, FP8
    Qwen/Qwen3-0.6BSFT, PEFT
    Qwen/Qwen3.8-2.4T-A95BSFT
    Qwen/Qwen3.8-Flash-NextSFT
    Qwen/QwQ-32BSFT, PEFT
    LLMGemmagoogle/gemma-3-270mSFT, PEFT
    google/gemma-2-9b-itSFT, PEFT, FP8
    google/gemma-7bSFT, PEFT
    LLMPhimicrosoft/phi-2SFT, PEFT
    microsoft/Phi-3-mini-4k-instructSFT, PEFT
    microsoft/phi-4SFT, PEFT, FP8
    LLMSeedByteDance-Seed/Seed-Coder-8B-InstructSFT, PEFT, [FP8](https://github.com/NVIDIA-NeMo/Automodel/blob/main/examples/llm_finetune/seed/seed_co

    相关项目

    同分类里 star 最高的项目
    对比全部 →
    LLMs-from-scratch
    rasbt·Jupyter Notebook

    从零用 PyTorch 一步步实现类 ChatGPT 大语言模型，覆盖注意力机制、预训练、指令微调全流程，适合边跑代码边理解 LLM 内部原理。

    大语言模型+4零代码微调平台
    108152+18417k2026-11-09 14:59 UTC
    llama-cookbook
    meta-llama·Jupyter Notebook

    Llama 官方示例手册，手把手教你在推理、微调、RAG 等场景中落地 Llama 系列模型，并覆盖多家云服务商的端到端方案。

    Llama+4零代码微调平台
    18589+52.8kv0.0.52026-05-19 18:42 UTC
    OpenLLM
    bentoml·Python

    把 DeepSeek、Llama 等开源大模型一键部署成兼容 OpenAI 的 API 服务，省去自己搭推理后端的麻烦。

    大模型部署+4零代码微调平台
    12587+1847v0.6.302026-11-09 17:03 UTC
    ludwig
    ludwig-ai·Python

    低代码训练框架，用声明式配置就能构建和微调自定义 LLM、神经网络及其他 AI 模型，省去手写大量训练代码。

    低代码训练+4零代码微调平台
    117911.2kv0.17.102026-11-09 21:43 UTC