finetrainers

huggingfaceApache-2.0活跃维护

面向扩散模型的可扩展、显存优化训练框架，基于 PyTorch 和 Diffusers，让大模型微调在有限显存下也能跑起来。

Scalable and memory-optimized training of diffusion models

v0.2.0发布于 2025-04-25 13:23 UTC1,364star145fork同步于 2026-11-07 07:31 UTC

扩散模型显存优化模型微调分布式训练图像生成

对比 GitHub 官网 ai art artificial-intelligence diffusers diffusion

怎么上手

由 README 提炼，执行前请确认

原文

pip install -r requirements.txt
pip install git+https://github.com/huggingface/diffusers

硬件要求按 README 说明

README 给出参考显存（49x512x768、rank 128、FP8 加梯度检查点）：LTX-Video LoRA 约 5 GB、全参约 21 GB；CogVideoX-5b LoRA 约 18 GB、全参约 53 GB；HunyuanVideo LoRA 约 32 GB，全参 OOM。Wan、CogView4、Flux 显存未给出。README 未说明 CPU 训练与 GPU 厂商型号。

下载：源码 zip·GitHub Releases（安装包）本站不镜像安装包

项目说明

由 README 自动翻译提炼

没人工点评？写下你的经验

Hugging Face 出品的扩散模型训练库，支持视频与图像模型的可扩展、省显存微调

面向扩散模型的可扩展、显存优化训练框架，基于 PyTorch 和 Diffusers，让大模型微调在有限显存下也能跑起来。

核心能力

支持 LoRA 与全参数微调，含条件控制训练
支持 DDP、FSDP-2、HSDP、CP 多卡分布式训练
可选 flash、flex、sage、xformers 多种注意力后端
自动识别常见数据集格式，支持图文视频混合与多分辨率分桶
支持大规模数据集预计算与 FP8 权重训练

容易踩的坑

建议用 PyTorch 2.5.1 及以上，旧版本可能导致视频全黑或 OOM；diffusers 需从源码安装，main 分支为开发分支，稳定版应切到 release tag；预计算条件与 latent 可能额外占用显存，需预留余量。

内容由模型根据 README 生成，可能不完整或不准确，以官方文档为准。看原文 README

Finetrainers v0.2.0 🧪

New trainers

Channel concatenated control conditioning for Wan2.1 and CogView4

The training involves adding extra input channels to the patch embedding layer (referred to as the "control injection" layer in finetrainers), to mix conditioning features into the latent stream. This architecture choice is very common and has been seen before in many models - CogVideoX-I2V, HunyuanVideo-I2V, Alibaba's Fun Control models, etc. Due to the popularity and simplicity in the architecture choice, it is a good choice to support standalone as a trainer.

import torch
from diffusers import CogView4Pipeline
from diffusers.utils import load_image
from finetrainers.models.utils import _expand_linear_with_zeroed_weights
from finetrainers.patches import load_lora_weights
from finetrainers.patches.dependencies.diffusers.control import control_channel_concat

dtype = torch.bfloat16
device = torch.device("cuda")
generator = torch.Generator().manual_seed(0)

pipe = CogView4Pipeline.from_pretrained("THUDM/CogView4-6B", torch_dtype=dtype)

in_channels = pipe.transformer.config.in_channels
patch_channels = pipe.transformer.patch_embed.proj.in_features
pipe.transformer.patch_embed.proj = _expand_linear_with_zeroed_weights(pipe.transformer.patch_embed.proj, new_in_features=2 * patch_channels)

load_lora_weights(pipe, "finetrainers/CogView4-6B-Edit-LoRA-v0", "cogview4-lora")
pipe.set_adapters("cogview4-lora", 0.9)
pipe.to(device)

prompt = "Make the image look like it's from an ancient Egyptian mural."
control_image = load_image("examples/training/control/cogview4/omni_edit/validation_dataset/0.png")
height, width = 1024, 1024

with torch.no_grad():
    latents = pipe.prepare_latents(1, in_channels, height, width, dtype, device, generator)
    control_image = pipe.image_processor.preprocess(control_image, height=height, width=width)
    control_image = control_image.to(device=device, dtype=dtype)
    control_latents = pipe.vae.encode(control_image).latent_dist.sample(generator=generator)
    control_latents = (control_latents - pipe.vae.config.shift_factor) * pipe.vae.config.scaling_factor

with control_channel_concat(pipe.transformer, ["hidden_states"], [control_latents], dims=[1]):
    image = pipe(prompt, latents=latents, num_inference_steps=30, generator=generator).images[0]

image.save("output.png")

New models supported

FLUX.1-dev
Wan2.1 I2V

Find example training configs here.

Attention

Support for multiple different attention providers for training and inference - Pytorch native, flash-attn, sageattention, xformers, flex. See docs for more details.

Other major changes

Better regional compilation support

What's Changed

Update project showcase by @a-r-r-o-w in https://github.com/a-r-r-o-w/finetrainers/pull/355
Flux ModelSpec by @a-r-r-o-w in https://github.com/a-r-r-o-w/finetrainers/pull/358
Pytorch regional compilation by @a-r-r-o-w in https://github.com/a-r-r-o-w/finetrainers/pull/361
[Doc] Fix a typo of flux.md by @DarkSharpness in https://github.com/a-r-r-o-w/finetr

v0.1.02025-03-21 · 2025-03-21 11:26 UTC
查看更新说明
Finetrainers v0.1.0 🧪

https://github.com/user-attachments/assets/37f27188-727a-4e9b-a1b9-baa621ddbb88

New models supported
- CogView4
- Wan 2.1
Checkpoints released
Other major changes
- Fully support for Accelerate as a parallelization backend again
- Opt-in precomputation. Helpful for smaller datasets
- Better remote dataset loading support
- Support for datasets>=3.4.0
- Bug fix: Layerwise Casting now works with Wan
- Bug fix: LTX Video training now works with batch_size > 1
What's Changed
- calling save_model_card correctly by @vivkul in https://github.com/a-r-r-o-w/finetrainers/pull/275
- 3D Parallel + Model Spec API by @a-r-r-o-w in https://github.com/a-r-r-o-w/finetrainers/pull/245
- CogVideoX ModelSpec by @a-r-r-o-w in https://github.com/a-r-r-o-w/finetrainers/pull/280
- Remove unused CogVideoX code by @a-r-r-o-w in https://github.com/a-r-r-o-w/finetrainers/pull/282
- Wan T2V ModelSpec by @a-r-r-o-w in https://github.com/a-r-r-o-w/finetrainers/pull/281
- update docs by @a-r-r-o-w in https://github.com/a-r-r-o-w/finetrainers/pull/284
- Improve local dataset loading by @a-r-r-o-w in https://github.com/a-r-r-o-w/finetrainers/pull/289
- Add 3DGS dataset example for Wan by @a-r-r-o-w in https://github.com/a-r-r-o-w/finetrainers/pull/290
- HunyuanVideo ModelSpec by @a-r-r-o-w in https://github.com/a-r-r-o-w/finetrainers/pull/287
- Fix full rank config by @a-r-r-o-w in https://github.com/a-r-r-o-w/finetrainers/pull/292
- update docs by @a-r-r-o-w in https://github.com/a-r-r-o-w/finetrainers/pull/294
- CogView4 ModelSpec by @a-r-r-o-w in https://github.com/a-r-r-o-w/finetrainers/pull/297
- PyPI package by @a-r-r-o-w in https://github.com/a-r-r-o-w/finetrainers/pull/298
- Minor cleanup and doc updates by @a-r-r-o-w in https://github.com/a-r-r-o-w/finetrainers/pull/304
- Remove forceful precomputation behaviour; Improvements to data loading by @a-r-r-o-w in https://github.com/a-r-r-o-w/finetrainers/pull/303
- Webdataset improvements; CogView4 example with The Simpsons webdataset by @a-r-r-o-w in https://github.com/a-r-r-o-w/finetrainers/pull/305
- Fix for #306 by @a-r-r-o-w in https://github.com/a-r-r-o-w/finetrainers/pull/307
- Fix Wan scaling due to upstream changes by @a-r-r-o-w in https://github.com/a-r-r-o-w/finetrainers/pull/308
- Fix CogVideoX config error for invert_scale_latents by @a-r-r-o-w in https://github.com/a-r-r-o-w/finetrainers/pull/309
- add valid names to dataset docs by @neph1 in https://github.com/a-r-r-o-w/finetrainers/pull/318
- Fix Layerwise Casting by @a-r-r-o-w in https://github.com/a-r-r-o-w/finetrainers/pull/316
- Fix enable_model_cpu_offload problems by @a-r-r-o-w in https://github.com/a-r-r-o-w/finetrainers/pull/320
- Patch WanTimeTextImageEmbedding forward only with fp8 by @a-r-r-o-w in https://github.com/a-r-r-o-w/finetrainers/pull/327
- Lower memory requirements on single GPU by @a-r-r-o-w in https://github.com/a-r-r-o-w/finetrainers/pull/321
- Respect report_to if set to none by @a-r-r-o-w in https://github.com/a-r-r-o-w/finetrainers/pull/331
- Cleanup model load methods by @a-r-r-o-w in https://github.com/a-r-r-o-w/finetrainers/pull/333
- add init.py to sft_trainer/ to fix #335 by @jbilcke-hf in https://github.com/a-r-r-o-w/finetrainers/pull/336
- Add version guard for datasets 3.4.0; Fix argument documentation by @a-r-r-o-w in https://github.com/a-r-r-o-w/finetrainers/pull/332
- Update dataset.py by @a-r-r-o-w in https://github.com/a-r-r-o-w/finetrainers/pull/337
- Remove hard-coded batch size configuration in LTXModel by @Linyou in https://github.com/a-r-r-o-w/finetrainers/pull/340
- Fix missing comma in loading hunyuan video to
v0.0.12025-02-24 · 2025-02-24 10:15 UTC
查看更新说明
FineTrainers v0.0.1 🧪

FineTrainers is a work-in-progress library to support (accessible) training of diffusion models. The following models are currently supported (based on Diffusers):
- CogVideoX T2V (versions 1.0 and 1.5)
- LTX Video
- Hunyuan Video
The legacy/deprecated scripts also support CogVideoX I2V and Mochi.

Currently, LoRA and Full-rank finetuning is supported. With time, more models and training techniques will be supported. We thank our many contributors for their amazing work at improve finetrainers. They are mentioned below in the "New Contributors" section.

In a short timespan, finetrainers has found its way into multiple research works, which has been a very motivating factor for us. They are mentioned in the "Featured Projects" section of the README. We hope you find them interesting and continue to build & work on interesting ideas, while sharing your research artifacts openly!

Some artifacts that we've released ourselves is available here: https://huggingface.co/finetrainers

We plan to focus on core algorithms/models that users prefer to have support for quickly, primarily based on the feedback we've received (thanks to everyone who's spoken with me regarding this. Your time is invaluable!) The majors asks are:
- more models and faster support for newer models (we will open this up for contributions after a major open but pending PR, and add many ourselves!)
- compatibility with UIs that do not support standardized implementations from diffusers (we will write two-way conversion scripts for new models that are added to diffusers, so that it is easy to obtain original-format weights from diffusers-format weights)
- more algorithms (Control LoRA, ControlNets for video models and VideoJAM are some of the highly asked techniques -- we will prioritize this!)
- Dataset QoL changes (this is a WIP in an open but pending PR)
Let us know what you'd like to see next & stay tuned for interesting updates!

What's Changed
- CogVideoX LoRA and full finetuning by @a-r-r-o-w in https://github.com/a-r-r-o-w/finetrainers/pull/1
- Low-bit memory optimizers, CpuOffloadOptimizer, Memory Reports by @a-r-r-o-w in https://github.com/a-r-r-o-w/finetrainers/pull/3
- Pin memory support in dataloader by @a-r-r-o-w in https://github.com/a-r-r-o-w/finetrainers/pull/5
- DeepSpeed fixes by @a-r-r-o-w in https://github.com/a-r-r-o-w/finetrainers/pull/7
- refactor readme i. by @sayakpaul in https://github.com/a-r-r-o-w/finetrainers/pull/8
- DeepSpeed and DDP Configs by @a-r-r-o-w in https://github.com/a-r-r-o-w/finetrainers/pull/10
- Full finetuning memory requirements by @a-r-r-o-w in https://github.com/a-r-r-o-w/finetrainers/pull/9
- Multi-GPU parallel encoding support for training videos. by @zRzRzRzRzRzRzR in https://github.com/a-r-r-o-w/finetrainers/pull/6
- CogVideoX I2V; CPU offloading; Model README descriptions by @a-r-r-o-w in https://github.com/a-r-r-o-w/finetrainers/pull/11
- add VideoDatasetWithResizeAndRectangleCrop dataset resize crop by @glide-the in https://github.com/a-r-r-o-w/finetrainers/pull/13
- add "max_sequence_length": model_config.max_text_seq_length, by @glide-the in https://github.com/a-r-r-o-w/finetrainers/pull/15
- readme updates + refactor by @a-r-r-o-w in https://github.com/a-r-r-o-w/finetrainers/pull/14
- Update README.md by @a-r-r-o-w in https://github.com/a-r-r-o-w/finetrainers/pull/17
- merge by @zRzRzRzRzRzRzR in https://github.com/a-r-r-o-w/finetrainers/pull/18
- Darft of Chinese README by @zRzRzRzRzRzRzR in https://github.com/a-r-r-o-w/finetrainers/pull/19
- docs: update README.md by @eltociear in https://github.com/a-r-r-o-w/finetrainers/pull/21
- Update requirements.txt (fixed typo) by @Nojahhh in https://github.com/a-r-r-o-w/finetrainers/pull

用户点评

登录用户的真实使用记录，可按版本和硬件筛选

共 0 条

还没有人点评。第一个说出真实体验的人最有价值。

README 原文

未翻译

在 GitHub 打开

finetrainers 🧪

Finetrainers is a work-in-progress library to support (accessible) training of diffusion models and various commonly used training algorithms.

Quickstart
Features
News
Support Matrix
Featured Projects
Acknowledgements

Quickstart

Clone the repository and make sure the requirements are installed: pip install -r requirements.txt and install diffusers from source by pip install git+https://github.com/huggingface/diffusers. The requirements specify diffusers>=0.32.1, but it is always recommended to use the main branch of Diffusers for the latest features and bugfixes. Note that the main branch for finetrainers is also the development branch, and stable support should be expected from the release tags.

Checkout to the latest stable release tag:

git fetch --all --tags
git checkout tags/v0.2.0

Follow the instructions mentioned in the README for the latest stable release.

Using the main branch

To get started quickly with example training scripts on the main development branch, refer to the following:

The following are some simple datasets/HF orgs with good datasets to test training with quickly:

Please checkout docs/models and examples/training to learn more about supported models for training & example reproducible training launch scripts. For a full list of arguments that can be set for training, refer to docs/args.

[!IMPORTANT] It is recommended to use Pytorch 2.5.1 or above for training. Previous versions can lead to completely black videos, OOM errors, or other issues and are not tested. For fully reproducible training, please use the same environment as mentioned in environment.md.

Features

DDP, FSDP-2 & HSDP, CP support
LoRA and full-rank finetuning; Conditional Control training
Memory-efficient single-GPU training
Multiple attention backends supported - flash, flex, sage, xformers (see attention docs)
Auto-detection of commonly used dataset formats
Combined image/video datasets, multiple chainable local/remote datasets, multi-resolution bucketing & more
Memory-efficient precomputation support with/without on-the-fly precomputation for large scale datasets
Standardized model specification format for training arbitrary models
Fake FP8 training (QAT upcoming!)

News

🔥 2025-04-25: Support for different attention providers added!
🔥 2025-04-21: Wan I2V supported added!
🔥 2025-04-12: Channel-concatenated control conditioning support added for CogView4 and Wan!
🔥 2025-04-08: torch.compile support added!
🔥 2025-04-06: Flux support added!
🔥 2025-03-07: CogView4 support added!
🔥 2025-03-03: Wan T2V support added!
🔥 2025-03-03: We have shipped a complete refactor to support multi-backend distributed training, better precomputation handling for big datasets, model specification format (externally usable for training custom models), FSDP & more.
🔥 2025-02-12: We have shipped a set of tooling to curate small and high-quality video datasets for fine-tuning. See video-dataset-scripts documentation page for details!
🔥 2025-02-12: Check out eisneim/ltx_lora_training_i2v_t2v! It builds off of finetrainers to support image to video training for LTX-Video and STG guidance for inference.
🔥 2025-01-15: Support for naive FP8 weight-casting training added! This allows training HunyuanVideo in under 24 GB upto specific resolutions.
🔥 2025-01-13: Support for T2V full-finetuning added! Thanks to @ArEnSc for taking up the initiative!
🔥 2025-01-03: Support for T2V LoRA finetuning of CogVideoX added!
🔥 2024-12-20: Support for T2V LoRA finetuning of Hunyuan Video added! We would like to thank @SHYuanBest for his work on a training script here.
🔥 2024-12-18: Support for T2V LoRA finetuning of LTX Video added!

Support Matrix

The following trainers are currently supported:

[!NOTE] The following numbers were obtained from the release branch. The main branch is unstable at the moment and may use higher memory.

Model Name	Tasks	Min. LoRA VRAM*	Min. Full Finetuning VRAM^
LTX-Video	Text-to-Video	5 GB	21 GB
HunyuanVideo	Text-to-Video	32 GB	OOM
CogVideoX-5b	Text-to-Video	18 GB	53 GB
Wan	Text-to-Video	TODO	TODO
CogView4	Text-to-Image	TODO	TODO
Flux	Text-to-Image	TODO	TODO

*Noted for training-only, no validation, at resolution 49x512x768, rank 128, with pre-computation, using FP8 weights & gradient checkpointing. Pre-computation of conditions and latents may require higher limits (but typically under 16 GB). ^Noted for training-only, no validation, at resolution 49x512x768, with pre-computation, using BF16 weights & gradient checkpointing.

If you would like to use a custom dataset, refer to the dataset preparation guide here.

Featured Projects 🔥

Checkout some amazing projects citing finetrainers:

Checkout the following UIs built for finetrainers:

Acknowledgements

finetrainers builds on top of & takes inspiration from great open-source libraries - transformers, accelerate, torchtune, torchtitan, peft, diffusers, bitsandbytes, torchao and deepspeed - to name a few.
Some of the design choices of finetrainers were inspired by SimpleTuner. `

finetrainers

怎么上手

项目说明

核心能力

容易踩的坑

最近发版

Finetrainers v0.2.0 🧪

New trainers

New models supported

Attention

Other major changes

What's Changed

Finetrainers v0.1.0 🧪

New models supported

Checkpoints released

Other major changes

What's Changed

FineTrainers v0.0.1 🧪

What's Changed

用户点评

README 原文

finetrainers 🧪

Table of Contents

Quickstart

Using the main branch

Features

News

Support Matrix

Featured Projects 🔥

Acknowledgements

怎么上手

项目说明

核心能力

容易踩的坑

最近发版

Finetrainers v0.2.0 🧪

New trainers

New models supported

Attention

Other major changes

What's Changed

Finetrainers v0.1.0 🧪

New models supported

Checkpoints released

Other major changes

What's Changed

FineTrainers v0.0.1 🧪

What's Changed

用户点评

README 原文

finetrainers 🧪

Table of Contents

Quickstart

Using the main branch

Features

News

Support Matrix

Featured Projects 🔥

Acknowledgements

相关项目