Awesome-Model-Merging-Methods-Theories-Applications

A comprehensive list of papers about 'Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunities. ACM Computing Surveys, 2026.'.

---

[!IMPORTANT] Contributions welcome:

Contact us or submit a pull request for unlisted relevant papers, content clarifications, or categorization adjustments, and update relevant information once your paper is accepted. Thank you!

---

💥 News 💥

• 🔥🔥🔥 Our survey, accepted by ACM Computing Surveys, please cite it or the library if helpful.
• 🔥🔥🔥 We flagged papers using models of size $\geq$ 7B (or small-sized mainstream LLMs) in their experiments.

---

Abstract

Model merging is an efficient empowerment technique in the machine learning community that does not require the collection of raw training data and does not require expensive computation. As model merging becomes increasingly prevalent across various fields, it is crucial to understand the available model merging techniques comprehensively. However, there is a significant gap in the literature regarding a systematic and thorough review of these techniques. To address this gap, this survey provides a comprehensive overview of model merging methods and theories, their applications in various domains and settings, and future research directions. Specifically, we first propose a new taxonomic approach that exhaustively discusses existing model merging methods. Secondly, we discuss the application of model merging techniques in large language models, multimodal large language models, and 10+ machine learning subfields, including continual learning, multi-task learning, few-shot learning, etc. Finally, we highlight the remaining challenges of model merging and discuss future research directions.

Citation

If you find our paper or this resource helpful, please consider cite:

@article{yang2026ModelMergingSurvey,
  author = {Yang, Enneng and Shen, Li and Guo, Guibing and Wang, Xingwei and Cao, Xiaochun and Zhang, Jie and Tao, Dacheng},
  title = {Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications, and Opportunities},
  year = {2026},
  issue_date = {June 2026},
  publisher = {Association for Computing Machinery},
  address = {New York, NY, USA},
  volume = {58},
  number = {8},
  issn = {0360-0300},
  url = {https://doi.org/10.1145/3787849},
  doi = {10.1145/3787849},
  journal = {ACM Computing Surveys},
  month = feb,
  articleno = {216},
  numpages = {41}
}

Thanks!

---

Framework

• 💥 News 💥
• Abstract
• Citation
• Framework
• Survey
• Benchmark/Evaluation
• Advanced Methods
  • Pre-Merging Methods
    • Better Fine-tuning
      • Linearization Fine-tuning
      • Subspace Fine-tuning
      • Sharpness-aware Fine-tuning
      • Others
    • Architecture Transformation
    • Weight Alignment
  • During Merging Methods
    • Basic Merging Methods
    • Weighted-based Merging Methods
    • Subspace-based Merging Method (Sparse or Low-rank Subspace)
    • Routing-based Merging Methods (Dynamic Merging)
    • Post-calibration based Methods
  • Other Merging Methods
  • Theories or Analysis of Model Merging
• Application of Model Merging in Foundation Models
  • Model Merging in Large Language Models
    • Human Preference Alignment for LLMs
    • Detoxification of LLMs
    • Knowledge Editing/Unlearning of LLMs
    • Faster Training of LLMs
    • Faster Reasoning of LLMs
    • Improving Computational Efficiency of MoE-based LLM
    • Mixing Datasets via Model Merging
    • LLM Agent Merging
    • Combine the Capabilities of Expert LLMs
  • Model Merging in Multimodal Large Language Models
    • Model Merging for Multimodal Fusion
    • Model Merging for Cross-Modal Knowledge Transfer
    • Combine the Capabilities of Expert MLLMs
  • Model Merging in Image Generative Models
    • Style Mixing in Generative Models
    • Reducing Training Cost of Generative Models
    • Enhancing the Faithfulness (or Generation Quality) of Diffusion Models
    • Deepfake Detection
  • Model Merging in Video Generative Models
    • Enhancing Motion Modeling
  • Model Merging in Embodied Models
• Application of Model Merging in Different Machine Learning Subfields
  • Model Merging in Continual Learning
    • Model Merging to Mitigate Catastrophic Forgetting
  • Model Merging in Multi-Task/Multi-Objective/Multi-Domain/Auxiliary Learning
    • Model Merging for Knowledge Transfer in Multi-Task Learning
    • Model Merging for Knowledge Transfer in Multi-Objective Optimization
    • Model Merging for Knowledge Transfer in Multi-Domain Learning
    • Model Merging for Knowledge Transfer in Auxiliary Learning
  • Model Merging in Out-of-Distribution/Domain Generalization
    • Model Merging for Better Out-of-Distribution Generalization
    • Model Merging for Better Domain Generalization or Domain Adaptation
  • Model Merging in Federated Learning
    • Model Merging for Local Knowledge Aggregation
  • Model Merging in Zero-shot/Few-shot Learning
    • Model Merging for Cross-task Generalization in Zero-shot Learning
    • Model Merging for Cross-task Generalization in Few-shot Learning
  • Model Merging in Adversarial Learning
    • Model Merging as an Attack
    • Model Merging as a Defense or Intellectual Property Protection
• Other Applications
• Contact

---

Survey

Paper Title	Year	Conference/Journal
Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunities	2026	ACM Computing Surveys
Model Merging in the Era of Large Language Models: Methods, Applications, and Future Directions	2026	Arxiv
Scaling Intelligence Through Model Merging: A Comprehensive Survey	2025	Arxiv
Democratizing AI Through Model Fusion: A Comprehensive Review and Future Directions	2025	Arxiv
From Task-Specific Models to Unified Systems: A Review of Model Merging Approaches	2025	Arxiv
SoK: On Finding Common Ground in Loss Landscapes Using Deep Model Merging Techniques	2024	Arxiv
A Survey on Model MoErging: Recycling and Routing Among Specialized Experts for Collaborative Learning	2024	Arxiv
Merge, Ensemble, and Cooperate! A Survey on Collaborative Strategies in the Era of Large Language Models	2024	Arxiv
Learn From Model Beyond Fine-Tuning: A Survey	2023	Arxiv
Deep Model Fusion: A Survey	2023	Arxiv

Benchmark/Evaluation

Paper Title	Year	Conference/Journal	Remark
Model Merging for Medical LVLMs: A Benchmark and a Winner-Take-All Approach	2026	Arxiv
merge-and-rebase	2026	Github	Codebase for model merging, task-vector transport, and configurable fine-tuning across vision and text models. It is built for fast iteration on checkpoint merging, rebasing, and evaluation workflows. Supports both Vison and Language merging.
crdt-merge	2026	Github	CRDT-based distributed model merging with formal convergence guarantees. 25 strategies (SLERP, TIES, DARE, Fisher, evolutionary). Two-layer OR-Set architecture enabling conflict-free multi-node merge.
Conflict-Free Replicated Data Types for Neural Network Model Merging: A Two-Layer Architecture Enabling CRDT-Compliant Model Merging Across 26 Strategies	2026	arXiv	crdt-merge
OptMerge: Unifying Multimodal LLM Capabilities and Modalities via Model Merging	2026	ICLR	Qwen2-VL-7B-Base, InternVL2.5-1B-Instruct, Qwen2.5-VL-32B-Instruct, Vicuna-7B-v1.5
An Empirical Survey of Model Merging Algorithms for Social Bias Mitigation	2025	Arxiv	LLAMA-2-7B, LLAMA-3-8B, LLAMA-3.1-8B, QWEN2-7B
A Systematic Study of Model Merging Techniques in Large Language Models	2025	Arxiv	Llama-3.2-3B-Instruct, Llama-3.1-8B-Instruct, Qwen3-4B, Qwen3-8B
FusionBench: A Comprehensive Benchmark of Deep Model Fusion	2025	JMLR	Mistral-7B-v0.1, MetaMath-Mistral-7B, dolphin-2.1-mistral-7b, speechless-code-mistral-7b-v1.0
Towards Performance Consistency in Multi-Level Model Collaboration	2025	ICCV
Model Merging Scaling Laws in Large Language Models	2025	Arxiv	Qwen2.5 0.5, 1.5, 3, 7, 14, 32, 72B
FBMS: An R Package for Flexible Bayesian Model Selection and Model Averaging	2025	Arxiv
MergeBench: A Benchmark for Merging Domain-Specialized LLMs	2025	Arxiv	Llama-3.2-3B, Llama3.1-8B, Gemma-2-2B and Gemma-2-9B
Mergenetic: a Simple Evolutionary Model Merging Library	2025	System Demonstrations	Mistral-7B
RobustMerge: Parameter-Efficient Model Merging for MLLMs with Direction Robustness	2025	NeurIPS	LLaVA-v1.5-7B
Mix Data or Merge Models? Balancing the Helpfulness, Honesty, and Harmlessness of Large Language Model via Model Merging	2025	Arxiv	Llama-3-8B-Instruct, Mistral-7B-Instruct-v0.2
How to Merge Your Multimodal Models Over Time?	2024	Arxiv
Mix Data or Merge Models? Optimizing for Diverse Multi-Task Learning	2024	Arxiv	Aya 23 8B
A Unified View of Delta Parameter Editing in Post-Trained Large-Scale Models	2024	Arxiv	LLaMA3-8B-Instruct, Qwen2-7B-Instruct, Mistral-7B-Instruct-v0.3,
Model-GLUE: Democratized LLM Scaling for A Large Model Zoo in the Wild	2024	NeurIPS Track on Datasets and Benchmarks	Synthia-7B-v1.2, Llama-2-7b-evolcodealpaca, OpenHermes-7B, pygmalion-2-7b, Llama-2-7b-chat-hf, BeingWell_llama2_7b, MetaMath-7B-V1.0, vicuna-7b-v1.5, Platypus2-7B, GOAT-7B-Community, Llama-2-7b-WikiChat-fused, dolphin-llama2-7b, MetaMath-Llemma-7B, CodeLlama-7b-Instruct-hf, Magicoder-S-CL-7B, CrystalChat
What Matters for Model Merging at Scale?	2024	Arxiv	PaLM-2 (1B, 8B, 24B, 64B), PaLM-2-IT (1B, 8B, 24B, 64B)
Realistic Evaluation of Model Merging for Compositional Generalization	2024	Arxiv
Fine-tuning large language models for domain adaptation: Exploration of training strategies, scaling, model merging and synergistic capabilities	2024	Arxiv	Llama-3.1-8B, Mistral-7B-v0.3
Arcee's MergeKit: A Toolkit for Merging Large Language Models	2024	Arxiv	Llama2-7B-Chat, Meditron-7B

Advanced Methods

Pre-Merging Methods

Better Fine-tuning

Linearization Fine-tuning

Paper Title	Year	Conference/Journal	Remark
Distilling Linearized Behavior into Non-Linear Fine-Tuning for Effective Task Arithmetic	2026	ICML	Llama-3.2-1B-Instruct
Understanding and Enforcing Weight Disentanglement in Task Arithmetic	2026	Arxiv
Tangent Space Fine-Tuning for Directional Preference Alignment in Large Language Models	2026	Arxiv	Llama-3.2-1B-Instruct
Dataless Weight Disentanglement in Task Arithmetic via Kronecker-Factored Approximate Curvature	2026	ICLR
Fine-Tuning Attention Modules Only: Enhancing Weight Disentanglement in Task Arithmetic	2025	ICLR
Tangent Transformers for Composition,Privacy and Removal	2024	ICLR
Parameter Efficient Multi-task Model Fusion with Partial Linearization	2024	ICLR
Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Models	2023	NeurIPS

Subspace Fine-tuning

Paper Title	Year	Conference/Journal	Remark
Unraveling LoRA Interference: Orthogonal Subspaces for Robust Model Merging	2025	Arxiv	Llama3-8B
Efficient Model Editing with Task-Localized Sparse Fine-tuning	2025	ICLR

Sharpness-aware Fine-tuning

Paper Title	Year	Conference/Journal	Remark
Mitigating Parameter Interference in Model Merging via Sharpness-Aware Fine-Tuning	2025	ICLR

Others

Paper Title	Year	Conference/Journal	Remark
CAMFT: Conflict-Aware Mergeable Fine-Tuning for Large Language Models	2026	Arxiv	Llama-3.1-8B-Instruct
Post-Hoc Merging is Not Enough: Many-Shot Model Merging with Loss-Gap Balancing	2026	ICML	Gemma-2-2B, Llama-3.2-3B, Llama-3.1-8B, and Qwen-3-4B
Understanding and Enforcing Weight Disentanglement in Task Arithmetic	2026	CVPR
MergOPT: A Merge-Aware Optimizer for Robust Model Merging	2026	ICLR	Llama3.1-8B-Instruct

Architecture Transformation

Paper Title	Year	Conference/Journal	Remark
Training-Free Transformer Merging via Sequential Local Operator Alignment	2026	Arxiv	ViTB-32, ViTB-16, Roberta-base, LLama-7B, LoRA finetuning
Rethinking Heterogeneous LLM Merging: A Weighted Model Averaging Perspective	2026	Arxiv	Qwen2.5-14B, Qwen2.5-32B, Qwen2.5-32B, Qwen3-8B, Qwen3-14B, Qwen3-32B
Model Assembly Learning with Heterogeneous Layer Weight Merging	2025	ICLR Workshop
Training-free Heterogeneous Model Merging	2025	Arxiv
Knowledge fusion of large language models	2024	ICLR	Llama-2 7B, OpenLLaMA 7B, MPT 7B
Knowledge Fusion of Chat LLMs: A Preliminary Technical Report	2024	Arxiv	NH2-Mixtral-8x7B, NH2-Solar-10.7B, and OpenChat-3.5-7B
On Cross-Layer Alignment for Model Fusion of Heterogeneous Neural Networks	2023	ICASSP
GAN Cocktail: mixing GANs without dataset access	2022	ECCV

Weight Alignment

Paper Title	Year	Conference/Journal	Remark
Scaling Linear Mode Connectivity and Merging to Billion Parameter Pretrained Transformers	2026	Arxiv
Transport and Merge: Cross-Architecture Merging for Large Language Models	2026	Arxiv	LLaMA-3 8B
Symmetry-Aware Graph Metanetwork Autoencoders: Model Merging through Parameter Canonicalization	2025	TAG-DS
Understanding Mode Connectivity via Parameter Space Symmetry	2025	ICML
Update Your Transformer to the Latest Release: Re-Basin of Task Vectors	2025	ICML
Model Assembly Learning with Heterogeneous Layer Weight Merging	2025	ICLR Workshop
Beyond the Permutation Symmetry of Transformers: The Role of Rotation for Model Fusion	2025	Arxiv
The Non-Local Model Merging Problem: Permutation Symmetries and Variance Collapse	2024	Arxiv
Equivariant Deep Weight Space Alignment	2024	ICML
Harmony in diversity: Merging neural networks with canonical correlation analysis	2024	ICML
Transformer fusion with optimal transport	2024	ICLR
Layerwise linear mode connectivity	2024	ICLR
ZipIt! Merging Models from Different Tasks without Training	2024	ICLR
Proving linear mode connectivity of neural networks via optimal transport	2024	AISTATS
Training-Free Pretrained Model Merging	2024	CVPR
Merging LoRAs like Playing LEGO: Pushing the Modularity of LoRA to Extremes Through Rank-Wise Clustering	2024	Arxiv	Llama2-7b, Llama2-13b
C2M3: Cycle-Consistent Multi Model Merging	2024	NeurIPS
PLeaS--Merging Models with Permutations and Least Squares	2024	Arxiv
Rethink Model Re-Basin and the Linear Mode Connectivity	2024	Arxiv
Git Re-Basin: Merging Models modulo Permutation Symmetries	2023	ICLR
Re-basin via implicit Sinkhorn differentiation	2023	CVPR
Plateau in Monotonic Linear Interpolation--A "Biased" View of Loss Landscape for Deep Networks	2023	ICLR
Linear Mode Connectivity of Deep Neural Networks via Permutation Invariance and Renormalization	2023	ICLR
REPAIR: REnormalizing Permuted Activations for Interpolation Repair	2023	ICLR
Going beyond linear mode connectivity: The layerwise linear feature connectivity	2023	NeurIPS
The role of permutation invariance in linear mode connectivity of neural networks	2022	ICLR
What can linear interpolation of neural network loss landscapes tell us?	2022	ICML
Loss Surface Simplexes for Mode Connecting Volumes and Fast Ensembling	2021	ICML
Analyzing Monotonic Linear Interpolation in Neural Network Loss Landscapes	2021	ICML
Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariances	2021	ICML
Linear Mode Connectivity and the Lottery Ticket Hypothesis	2020	ICML
Optimizing mode connectivity via neuron alignment	2020	NeurIPS
Model fusion via optimal transport	2020	NeurIPS
Uniform convergence may be unable to explain generalization in deep learning	2019	NeurIPS
Explaining landscape connectivity of low-cost solutions for multilayer nets	2019	NeurIPS
Essentially no barriers in neural network energy landscape	2018	ICML
Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs	2018	NeurIPS

During Merging Methods

Basic Merging Methods

Paper Title	Year	Conference/Journal	Remark
Composing parameter-efficient modules with arithmetic operation	2023	NeurIPS
Editing models with task arithmetic	2023	ICLR
Model fusion via optimal transport	2020	NeurIPS
Weight averaging for neural networks and local resampling schemes	1996	AAAI Workshop
Acceleration of stochastic approximation by averaging	1992	IAM Journal on Control and Optimization
Animating rotation with quaternion curves (Spherical Linear Interpolation (SLERP) Model Merging)	1985	SIGGRAPH Computer Graphics

Weighted-based Merging Methods

Paper Title	Year	Conference/Journal	Remark
CrossMean: Model Merging by Pairing Merged and Expert Activations	2026	Arxiv
αTransfer: Coefficient Transfer for Efficient Model Merging	2026	Arxiv	Qwen3-0.6B, Qwen3-1.7B, Qwen3-4B
The Missing Coefficients: Bayesian Pairwise Merging for Model Personalization	2026	Arxiv	Qwen3.5-2B, Qwen3-VL-4B, Ministral-3-3B
CoMerge: Conflict-Driven Preference Optimization for Multi-Task Model Merging	2026	Arxiv	Llama-3.1-8B-Instruct
Task-Aware Model Merging via Fisher-Weighted Median	2026	TMLR	Llama-3.2-3B, Llama-3.1-8B, CLIP, GPT2
HyperFix: Combinatorial Nonlinear Correction for Task Vector Merging	2026	Arxiv
EvoGM: Learning to Merge LLMs via Evolutionary Generative Optimization	2026	ICML	Qwen2.5-1.5B, Qwen3-8B
Label-Free Cross-Task LoRA Merging with Null-Space Compression	2026	Arxiv	LLAMA-3 8B, LLAVA-1.5-7B
The Mean is the Mirage: Entropy-Adaptive Model Merging under Heterogeneous Domain Shifts in Medical Imaging	2026	Arxiv
LARV: Data-Free Layer-wise Adaptive Rescaling Veneer for Model Merging	2026	Arxiv
Souper-Model: How Simple Arithmetic Unlocks State-of-the-Art LLM Performance	2025	Arxiv	xLAM-2-70b, CoALM-70B, watt-tool-70B, functionary-medium-70B, xLAM-2-8b, ToolACE-2-8B, watt-tool-8B, BitAgent-8B, CoALM-8B
Superpose Task-specific Features for Model Merging	2025	EMNLP	Llama-2-7B
T3: Test-Time Model Merging in VLMs for Zero-Shot Medical Imaging Analysis	2025	Arxiv
Weight Weaving: Parameter Pooling for Data-Free Model Merging	2025	Arxiv
Expert Merging: Model Merging with Unsupervised Expert Alignment and Importance-Guided Layer Chunking	2025	Arxiv	Mistral-7B, InternVL, Qwen2-VL
Variational Task Vector Composition	2025	NeurIPS
RegMean++: Enhancing Effectiveness and Generalization of Regression Mean for Model Merging	2025	Arxiv
StatsMerging: Statistics-Guided Model Merging via Task-Specific Teacher Distillation	2025	Arxiv
SeMe: Training-Free Language Model Merging via Semantic Alignment	2025	Arxiv
NAN: A Training-Free Solution to Coefficient Estimation in Model Merging	2025	Arxiv	LLaMA2-13B, WizardLM-13B, WizardMath-13B, LLaVA-v1.5-13B, LLaVA-1.6-13B, Math-LLaVA
Leveraging Submodule Linearity Enhances Task Arithmetic Performance in LLMs	2025	ICLR	Llama-2-7B and Llama-2-13B
Layer-Aware Task Arithmetic: Disentangling Task-Specific and Instruction-Following Knowledge	2025	Arxiv	Gemma-2-9B, Llama-3-8B
Sens-Merging: Sensitivity-Guided Parameter Balancing for Merging Large Language Models	2025	Arxiv	LLaMA-2 7B series, Mistral 7B series, LLaMA-2 13B series
RankMean: Module-Level Importance Score for Merging Fine-tuned Large Language Models	2024	ACL
Non-Uniform Parameter-Wise Model Merging	2024	Arxiv
How to Weight Multitask Finetuning? Fast Previews via Bayesian Model-Merging	2024	Arxiv
LiNeS: Post-training Layer Scaling Prevents Forgetting and Enhances Model Merging	2024	Arxiv
Merging in a Bottle: Differentiable Adaptive Merging (DAM) and the Path from Averaging to Automation	2024	Arxiv	shisa-gamma-7b, WizardMath-7B-V1.1, Abel-7B-002, Llama-3-SauerkrautLM-8b-Instruct, Llama-3-Open-Ko-8B, llama-3-sqlcoder-8b, Meta-Llama-3-8B
Knowledge Composition using Task Vectors with Learned Anisotropic Scaling	2024	Arxiv
MetaGPT: Merging Large Language Models Using Model Exclusive Task Arithmetic	2024	EMNLP	LLaMA-2-7B, Mistral-7B, LLaMA-2-13B
Checkpoint Merging via Bayesian Optimization in LLM Pretraining	2024	Arxiv	Baichuan2-220B, Baichuan2-440B, Baichuan2-660B, Baichuan2-1540B, Baichuan2-1760B, Baichuan2-1980B, Baichuan2-2200B, Baichuan2-2420B, DeepSeek-1400B, DeepSeek-1600B, DeepSeek-1800B, DeepSeek-2000B
Arcee’s MergeKit: A Toolkit for Merging Large Language Models	2024	Arxiv	Llama2-7B-Chat, Meditron-7B
Evolutionary optimization of model merging recipes	2024	Arxiv	shisa-gamma-7b-v1, WizardMath-7B-V1.1, Arithmo2-Mistral-7B, Abel-7B-002, Mistral-7B-v0.1, LLaVA-1.6-Mistral-7B
XFT: Unlocking the Power of Code Instruction Tuning by Simply Merging Upcycled Mixture-of-Experts	2024	ACL
AdaMerging: Adaptive Model Merging for Multi-Task Learning	2024	ICLR
Model Merging by Uncertainty-Based Gradient Matching	2024	ICLR
Merging by Matching Models in Task Subspaces	2024	TMLR
Fisher Mask Nodes for Language Model Merging	2024	LREC-COLING
Erasure Coded Neural Network Inference via Fisher Averaging	2024	ISIT
Dataless Knowledge Fusion by Merging Weights of Language Models	2023	ICLR
Merging models with fisher-weighted averaging	2022	NeurIPS

Subspace-based Merging Method (Sparse or Low-rank Subspace)

Paper Title	Year	Conference/Journal	Remark
Separating Common and Unique Directions for Model Merging	2026	Arxiv	Llama-3.1-8B
Model Merging Does Not Require Uniform LoRA Ranks	2026	Arxiv
GRAM: Group-wise Rank-Aware Model Merging via Subspace Alignment)	2026	Arxiv	Qwen2-7B-Instruct
PGSB: Pretrained-Guided Shared Basis for LoRA Model Merging	2026	Arxiv	Llama 3 8B
Predicting and Repairing Merge Collapse in Large Language Models	2026	Arxiv	Mistral-7B, DeepSeek-7B, Qwen2.5-7B
CASS: Contribution-Aware Structured Sparsity for Model Merging	2026	NeurIPS	Qwen2.5-0.5B, Qwen2.5-1.5B
Orthogonal Yet Coupled: Decoupling Geometric Components for Model Merging	2026	Arxiv	Llama-3.1-8B, Qwen2.5-VL-7B-Instruct
Beyond Uniform Subspaces: Spectrum-Aware and Depth-Adaptive Fusion for Multi-Task Model Merging	2026	Arxiv
Not All Task Vectors Need Equal Rank: Energy-Proportional Allocation for Model Merging	2026	Arxiv
Not All Ranks Are Equal: Budget-Aware LoRA Merging Across Tasks	2026	Arxiv
Escaping Low-Dimensional Overlap: Multi-Task Model Merging via High-Dimensional Sparse Disentanglement	2026	Arxiv	Qwen2.5-1.5B and Qwen2.5-7B
CORAM: Coherent Orthogonal Rotation for Model Merging	2026	Arxiv	Llama3.2-3B, Llama-3.1-8B, Qwen2.5-VL-7B-Instruct, Gemma-2-9B
SWIM-merging: Reweighting singular values of weight matrices for efficient test-time model merging	2026	Neurocomputing
CABS+: Efficient and Scalable Model Merging via Conflict-Aware Sparsification and Adaptive Weight Allocation	2026	Arxiv	Mistral-7B-v0.1, WildMarcoroni-Variant1-7B and WestSeverus7B-DPO-v2
CT-Merging: Consensus Directions and Task-Level Scaling for LoRA Adapter Merging	2026	Arxiv
Spectral Rewiring for Exploration, Purification, and Model Merging	2026	Arxiv	DeepScaleR (1.5B), POLARIS (4B), OLMo-3.1-32B-Think, OLMo-3-7B-Base
Essential Subspace Merging for Multi-Task Learning	2026	Arxiv
Sparsity Curse: Understanding RLVR Model Parameter Space from Model Merging	2026	KDD	Qwen-2.5-7B
PACT: Preserving Anchored Cores in Task-vectors for Model Merging	2026	Arxiv
Closed-Form Spectral Regularization for Multi-Task Model Merging	2026	Arxiv	Llama-3.2-3B, Qwen2-VL-7B, InternVL2.5-1B
ResMerge: Residual-based Spectral Merging of Large Language Models	2026	Arxiv	Qwen2.5-7B-Base, Qwen2.5-7B-SimpleRL-Zoo, Open-ReasonerZero-7B (Zero), General-Reasoner-Qwen2.5-7B(Reasoner)
Compress then Merge: From Multiple LoRAs into One Low-Rank Adapter	2026	ICML	LLaMA3-8B
TaDA: Calibrated Probe Gating for Task-Domain LoRA Merging	2026	Arxiv	Llama-2-7b-hf
Model Merging by Output-Space Projection	2026	Arxiv	Llama3.1-8B
Saliency-Aware Model Merging	2026	Arxiv
Model Merging on Loss Landscape: A Geometry Perspective	2026	Arxiv
PivotMerge: Bridging Heterogeneous Multimodal Pre-training via Post-Alignment Model Merging	2026	Arxiv	LLaVA1.5-7B
Towards Adaptive Continual Model Merging via Manifold-Aware Expert Evolution	2026	Arxiv
Evolutionary Negative Module Pruning for Better LoRA Merging	2026	Arxiv
Crowded in B-Space: Calibrating Shared Directions for LoRA Merging	2026	Arxiv	Llama-3.1-8B
Diet Your LLM: Dimension-wise Global Pruning of LLMs via Merging Task-specific Importance Score	2026	Arxiv	Gemma-2 9B, Qwen2.5-7B, Phi-4-mini
DC-Merge: Improving Model Merging with Directional Consistency	2026	CVPR	LLaVA
CoMoL: Efficient Mixture of LoRA Experts via Dynamic Core Space Merging	2026	Arxiv	Qwen3-8B and Llama3.1-8B
Model Merging in the Essential Subspace	2026	Arxiv
Beyond Parameter Arithmetic: Sparse Complementary Fusion for Distribution-Aware Model Merging	2026	Arxiv	Mistral-7B, Qwen2.5-14B, and Qwen2.5-32B
Orthogonal Model Merging	2026	Arxiv	Llama-3.1-8B, Qwen2.5-VL-7B-Instruct, Llama-3.2-3B
When Shared Knowledge Hurts: Spectral Over-Accumulation in Model Merging	2026	Arxiv
Merging Beyond: Streaming LLM Updates via Activation-Guided Rotations	2026	Arxiv	Qwen2.5-7B, Qwen2.5-14B
AdaRank: Adaptive Rank Pruning for Enhanced Model Merging	2026	ICLR
Decomposing Task Vectors for Refined Model Editing	2025	Arxiv
Stay Unique, Stay Efficient: Preserving Model Personality in Multi-Task Merging	2025	Arxiv	Qwen-14B
Towards Reversible Model Merging For Low-rank Weights	2025	Arxiv
Purifying Task Vectors in Knowledge-Aware Subspace for Model Merging	2025	Arxiv	LLaMA-2-7B
RobustMerge: Parameter-Efficient Model Merging for MLLMs with Direction Robustness	2025	NeurIPS	LLaVA
Accurate and Efficient Low-Rank Model Merging in Core Space	2025	NeurIPS
Efficient Multi-Source Knowledge Transfer by Model Merging	2025	Arxiv
One Size Does Not Fit All: A Distribution-Aware Sparsification for More Precise Model Merging	2025	Arxiv
NegMerge: Sign-Consensual Weight Merging for Machine Unlearning	2025	ICML
Subspace-Boosted Model Merging	2025	Arxiv
Training-free LLM Merging for Multi-task Learning	2025	Arxiv
Merging Smarter, Generalizing Better: Enhancing Model Merging on OOD Data	2025	Arxiv
Locate-then-Merge: Neuron-Level Parameter Fusion for Mitigating Catastrophic Forgetting in Multimodal LLMs	2025	Arxiv	Mistral-7B, Llama3-8B
CALM: Consensus-Aware Localized Merging for Multi-Task Learning	2025	ICML
Merge-Friendly Post-Training Quantization for Multi-Target Domain Adaptation	2025	ICML
Adaptive LoRA Merge with Parameter Pruning for Low-Resource Generation	2025	ACL	Llama-3-8B-Instruct
Decom-Renorm-Merge: Model Merging on the Right Space Improves Multitasking	2025	Arxiv	LLaMA3.1-8B
CAT Merging: A Training-Free Approach for Resolving Conflicts in Model Merging	2025	Arxiv
LoRI: Reducing Cross-Task Interference in Multi-Task LowRank Adaptation	2025	Arxiv	Llama-3-8B and Mistral-7B
Task Vector Quantization for Memory-Efficient Model Merging	2025	Arxiv
Disentangling Task Interference within Neurons: Model Merging in Alignment with Neuronal Mechanisms	2025	Arxiv	Llama-2-7b
Exploring Sparse Adapters for Scalable Merging of Parameter Efficient Experts	2025	ICLR 2025 Workshop
LEWIS (LayEr WIse Sparsity) -- A Training Free Guided Model Merging Approach	2025	ICLR 2025 Workshop	Gemma-9b, LLaMA 3.1 8b
CABS: Conflict-Aware and Balanced Sparsification for Enhancing Model Merging	2025	Arxiv	Mistral-7b-v0.1, WildMarcoroni-Variant1-7B and WestSeverus-7B-DPO-v2
Low-Rank and Sparse Model Merging for Multi-Lingual Speech Recognition and Translation	2025	Arxiv
LED-Merging: Mitigating Safety-Utility Conflicts in Model Merging with Location-Election-Disjoint	2025	Arxiv	Llama-3- 8B, Mistral-7B, and Llama2-13B
Parameter Efficient Merging for Multimodal Large Language Models with Complementary Parameter Adaptation	2025	Arxiv
Optimal Brain Iterative Merging: Mitigating Interference in LLM Merging	2025	Arxiv	Llama-2-13b, WizardMath-13B-V1.0, WizardLM13B-V1.2, llama-2-13b-codealpaca
Superpose Singular Features for Model Merging	2025	Arxiv	Llama-2-7B
STAR: Spectral Truncation and Rescale for Model Merging	2025	NAACL	Mistral-7B-Instruct
No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces	2025	Arxiv
Merging Models on the Fly Without Retraining: A Sequential Approach to Scalable Continual Model Merging	2025	NeurIPS
Modeling Multi-Task Model Merging as Adaptive Projective Gradient Descent	2025	Arxiv
Revisiting Weight Averaging for Model Merging	2024	Arxiv
Task Singular Vectors: Reducing Task Interference in Model Merging	2025	CVPR
Less is More: Efficient Model Merging with Binary Task Switch	2024	Arxiv
FREE-Merging: Fourier Transform for Model Merging with Lightweight Experts	2024	Arxiv	Qwen-14B (LoRA), LLaMa2-13B, WizardLM-13B, WizardMath-13B, WizardCoderPython-13B
Beyond Task Vectors: Selective Task Arithmetic Based on Importance Metrics	2024	Arxiv
Parameter Competition Balancing for Model Merging	2024	NeurIPS	Llama-2-7b
Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch	2024	ICML	WizardLM-13B, WizardMath-13B, and llama-2-13b-codealpaca, Mistral-7B
Localizing Task Information for Improved Model Merging and Compression	2024	ICML
Sparse Model Soups: A Recipe for Improved Pruning via Model Averaging	2024	ICLR
Model merging with svd to tie the knots	2024	Arxiv	Llama3-8B
NegMerge: Consensual Weight Negation for Strong Machine Unlearning	2024	Arxiv
Localize-and-Stitch: Efficient Model Merging via Sparse Task Arithmetic	2024	Arxiv
Activated Parameter Locating via Causal Intervention for Model Merging	2024	Arxiv	Llama-2-chat-7B
PAFT: A Parallel Training Paradigm for Effective LLM Fine-Tuning	2024	Arxiv	Mistral-7B-v0.1, Llama-3-8B, Neurotic-7B, MoMo-70B
DELLA-Merging: Reducing Interference in Model Merging through Magnitude-Based Sampling	2024	Arxiv	Llama-2-13b-code-alpaca, WizardLM, Wizard-Math, WizardCoder-Python
EMR-Merging: Tuning-Free High-Performance Model Merging	2024	NeurIPS
DPPA: Pruning Method for Large Language Model to Model Merging	2024	Arxiv	LLaMa 2
Model breadcrumbs: Scaling multi-task model merging with sparse masks	2023	Arxiv
Concrete Subspace Learning based Interference Elimination for Multi-task Model Fusion	2023	Arxiv
ComPEFT: Compression for Communicating Parameter Efficient Updates via Sparsification and Quantization	2023	Arxiv	LLaMA 7B, 13B, 33B, and 65B
Effective and ParameterEfficient Reusing Fine-Tuned Models	2023	Openreview
Resolving Interference When Merging Models	2023	NeurIPS
Task-Specific Skill Localization in Fine-tuned Language Model	2023	ICML

Routing-based Merging Methods (Dynamic Merging)

Paper Title	Year	Conference/Journal	Remark
Learning to Recover Task Experts from a Multi-Task Merged Model	2026	Arxiv
Training-free Task Classification for Multi-Task Model Merging	2026	Arxiv
Dynamic Model Merging Made Slim	2026	Arxiv	Llama-3.1-8B-Instruct, Qwen-2.5-7B-Instruct
Auto-FlexSwitch: Efficient Dynamic Model Merging via Learnable Task Vector Compression	2026	Arxiv
TECS-L (Golden MoE): Dense-to-MoE Expert Splitting Framework	2026	GitHub	Mistral-7B
Fine-Grained Model Merging via Modular Expert Recombination	2026	Arxiv
MIN-Merging: Merge the Important Neurons for Model Merging	2025	Arxiv
SE-Merging: A Self-Enhanced Approach for Dynamic Model Merging	2025	Arxiv
Adaptive Task Vectors for Large Language Models	2025	Arxiv	LLaMA3-8B and Mistral-7B
Dynamic Fisher-weighted Model Merging via Bayesian Optimization	2025	Arxiv
Data-Adaptive Weight-Ensembling for Multi-task Model Fusion	2025	IJCV
MASS: MoErging through Adaptive Subspace Selection	2025	Arxiv
Dynamic Model Merging with Mixture of Weights	2025	TCSVT
CAMEx: Curvature-aware Merging of Experts	2025	ICLR
1bit-Merging: Dynamic Quantized Merging for Large Language Models	2025	Arxiv	LLaMA-2 7B, Mistral 7B, and LLaMA-2 13B
MergeME: Model Merging Techniques for Homogeneous and Heterogeneous MoEs	2025	Arxiv
Mediator: Memory-efficient LLM Merging with Less Parameter Conflicts and Uncertainty Based Routing	2025	Arxiv	Qwen-2.5-7B, LLaMA-3.2-8B
Adapting Foundation Models via Training-free Dynamic Weight Interpolation	2024	NeurIPS 2024 Workshop
Efficient and Effective Weight-Ensembling Mixture of Experts for Multi-Task Model Merging	2024	Arxiv
DaWin: Training-free Dynamic Weight Interpolation for Robust Adaptation	2024	NeurIPS 2024 Workshop
Merging Multi-Task Models via Weight-Ensembling Mixture of Experts	2024	ICML
Learning to Route Among Specialized Experts for Zero-Shot Generalization	2024	ICML
Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policy	2024	ICLR
Soft merging of experts with adaptive routing	2024	TMLR
SMILE: Zero-Shot Sparse Mixture of Low-Rank Experts Construction From Pre-Trained Foundation Models	2024	Arxiv	Mistral-7B-v0.1, MetaMath-Mistral-7B, dolphin-2.1-mistral-7b, speechless-code-mistral-7b-v1.0
Twin-Merging: Dynamic Integration of Modular Expertise in Model Merging	2024	NeurIPS	Qwen-14B
Self-MoE: Towards Compositional Large Language Models with Self-Specialized Experts	2024	Arxiv	Gemma-7B, LLaMA-2 7B & 13B, Mistral 7B, LLaMA-3 8B
Towards Efficient Pareto Set Approximation via Mixture of Experts Based Model Fusion	2024	Arxiv
Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints	2023	ICLR

Post-calibration based Methods

Paper Title	Year	Conference/Journal	Remark
Error-Subspace Calibration for Multi-Task Model Merging	2026	Arxiv	WizardLM-13B-V1.2, LLaMA-2-13B
Merge++: Universal Merge Refinement Through Data-Free Checkpoint Inversion	2026	Arxiv
DARTS: Decoder-Aware Representation Tuning via Surgery for Model Merging	2026	Arxiv	Llama2-7B-Instruct (Meta), WizardMath-7B-v1.0, and Llama-2-7B-Code
SyMerge: From Non-Interference to Synergistic Merging via Single-Layer Adaptation	2026	ICML
FEATCAL: Feature Calibration for Post-Merging Models	2026	Arxiv	Llama-3.1-8B-Instruc
MAGIC: Achieving Superior Model Merging via Magnitude Calibration	2025	Arxiv	OLMo-3-7B
Towards Minimizing Feature Drift in Model Merging: Layer-wise Task Vector Fusion for Adaptive Knowledge Integration	2025	NeurIPS
Multi-Task Model Fusion via Adaptive Merging	2025	ICASSP
Representation Surgery in Model Merging with Probabilistic Modeling	2025	ICML
Parameter-Efficient Interventions for Enhanced Model Merging	2024	Arxiv
MOMA: Masked Orthogonal Matrix Alignment for Zero-Additional-Parameter Model Merging	2024	Arxiv
SurgeryV2: Bridging the Gap Between Model Merging and Multi-Task Learning with Deep Representation Surgery	2024	Arxiv
Representation Surgery for Multi-Task Model Merging	2024	ICML

Other Merging Methods

Paper Title	Year	Conference/Journal	Remark
A Broader Look at Model Merging: Rethinking Implicit Regularization Induced by Task Arithmetic	2026	Arxiv	Qwen3-0.6B, Qwen3-1.7B, Qwen3-4B
Orientation, not magnitude: the causal structure of task-vector interference in merged language models	2026	Arxiv
Rethinking Expert Training for Model Merging with Prompt Learning	2026	Arxiv
TwistedMerge: Certified Higher-Order Diagnostics and Abstention for Model Merging	2026	Arxiv
Model Merging as Probabilistic Inference in Fine-Tuning Parameter Space	2026	Arxiv
Model Merging to Evolution: Parameter Space Exploration for Expert Models	2026	Arxiv	Gemma-2-2b-it
GFFMERGE: Efficient Merging of Graph Neural Force Fields and Beyond	2026	ICML
Access Sets Matter: Budgeting Expert Reads for Scalable Weight-Space Model Merging	2026	ICML Workshop	Qwen3-0.6B
Darwin Family: MRI-Trust-Weighted Evolutionary Merging for Training-Free Scaling of Language-Model Reasoning	2026	Arxiv
Black-Box Optimization of Mixed Binary-Continuous Variables: Challenges and Opportunities in Evolutionary Model Merging	2026	Arxiv
Bayesian Model Merging	2026	Arxiv
Generalizing the Geometry of Model Merging Through Frechet Averages	2026	Arxiv	Llama-3 8B
Differentially Private Model Merging	2026	Arxiv
Task Alignment: A simple and effective proxy for model merging in computer vision	2026	Arxiv
Model Merging via Data-Free Covariance Estimation	2026	Arxiv
Resolving Interference (RI): Disentangling Models for Improved Model Merging	2026	Arxiv
BD-Merging: Bias-Aware Dynamic Model Merging with Evidence-Guided Contrastive Learning	2026	Arxiv
ACE-Merging: Data-Free Model Merging with Adaptive Covariance Estimation	2026	Arxiv
Training-Free Cross-Architecture Merging for Graph Neural Networks	2026	Arxiv
Gradient-Sign Masking for Task Vector Transport Across Pre-Trained Models	2026	ICLR	Flan-T5
Transporting Task Vectors across Different Architectures without Training	2026	ICML
MergePipe: A Budget-Aware Parameter Management System for Scalable LLM Merging	2026	Arxiv	Llama3.1-8B, Llama-3.2-3B, Qwen3-0.6B, Qwen3-1.7B, and Qwen3-8B
DisTaC: Conditioning Task Vectors via Distillation for Robust Model Merging	2026	ICLR
Sparsity-Aware Evolution for Model Merging	2026	Arxiv
AutoMerge: Search-Based Model Merging Framework for Effective Model Reuse	2026	Arxiv	Llama2-7B-Chat, Llama2-7B-Code
Model Merging via Multi-Teacher Knowledge Distillation	2025	Arxiv
Bridging Training and Merging Through Momentum-Aware Optimization	2025	Arxiv
From Coefficients to Directions: Rethinking Model Merging with Directional Alignment	2025	Arxiv
Escaping Optimization Stagnation: Taking Steps Beyond Task Arithmetic via Difference Vectors	2025	Arxiv
Model Merging with Functional Dual Anchors	2025	Arxiv
Black-box Model Merging for Language-Model-as-a-Service with Massive Model Repositories	2025	Arxiv
Rethinking Layer-wise Model Merging through Chain of Merges	2025	Arxiv	Llama 3-8B
Competition and Attraction Improve Model Fusion	2025	Arxiv	WizardMath 7B v1.0, AgentEvol 7B
PSO-Merging: Merging Models Based on Particle Swarm Optimization	2025	Arxiv	Llama-3-8B, Llama-2-13B, and Mistral-7B-v0.3
DisTaC: Conditioning Task Vectors via Distillation for Robust Model Merging	2025	Arxiv
Navigating the Accuracy-Size Trade-Off with Flexible Model Merging	2025	Arxiv
Efficient Multi-Task Inferencing: Model Merging with Gromov-Wasserstein Feature Alignment	2025	Arxiv
Reinforced Model Merging	2025	Arxiv
FW-Merging: Scaling Model Merging with Frank-Wolfe Optimization	2025	Arxiv	LLaMA2-7B
Whoever Started the Interference Should End It: Guiding Data-Free Model Merging via Task Vectors	2025	Arxiv	WizardLM-13B (LM), WizardMath-13B (Math), and llama-2-13bcodealpaca (Code)
GNNMERGE: Merging of GNN Models Without Accessing Training Data	2025	Arxiv
MERGE3: Efficient Evolutionary Merging on Consumer-grade GPUs	2025	ICML	Mistral-7B
Activation-Informed Merging of Large Language Models	2025	Arxiv	Llama-2-13b, WizardLM-13B, WizardMath-13B, llama-2-13b-code-alpaca
Scalable Model Merging with Progressive Layer-wise Distillation	2025	Arxiv	WizardLM-13B, WizardMath-13B and llama-2-13b-code-alpaca
Fine, I’ll Merge It Myself: A Multi-Fidelity Framework for Automated Model Merging	2025	Arxiv	Llama-2-13, WizardLM13B, WizardMath-13, llama-2-13b-code-alpaca
Task Arithmetic in Trust Region: A Training-Free Model Merging Approach to Navigate Knowledge Conflicts	2025	ICLR
Fine-tuning Aligned Classifiers for Merging Outputs: Towards a Superior Evaluation Protocol in Model Merging	2024	Arxiv
Multi-Task Model Merging via Adaptive Weight Disentanglement	2024	Arxiv
Rethinking Weight-Averaged Model-merging	2024	Arxiv
ATM: Improving Model Merging by Alternating Tuning and Merging	2024	Arxiv
HM3: Hierarchical Multi-Objective Model Merging for Pretrained Models	2024	Arxiv	Llama-2-7B-Chat, WizardMath-7B, CodeLlama-7B
Weight Scope Alignment: A Frustratingly Easy Method for Model Merging	2024	Arxiv
It’s Morphing Time: Unleashing the Potential of Multiple LLMs via Multi-objective Optimization	2024	Arxiv	Qwen1.5-7B-Chat, Liberated-Qwen1.5-7B, firefly-qwen1.5-en-7B
Toward Data Efficient Model Merging between Different Datasets without Performance Degradation	2024	JMLR
SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scaling	2023	Arxiv	SOLAR 10