mergekit

mergekit

mergekit is a toolkit for merging pre-trained language models. mergekit uses an out-of-core approach to perform unreasonably elaborate merges in resource-constrained situations. Merges can be run entirely on CPU or accelerated with as little as 8 GB of VRAM. Many merging algorithms are supported, with more coming as they catch my attention.

Contents

• Why Merge Models?
• Features
• Installation
• Community & Support
  • Contributing
  • Community Tools
• Usage
• Merge Configuration
  • Parameter Specification
  • Tokenizer Configuration
  • Chat Template Configuration
  • Examples
• Merge Methods
• LoRA Extraction
• Mixture of Experts Merging
• Evolutionary Merge Methods
• Multi-Stage Merging (mergekit-multi)
• Raw PyTorch Model Merging (mergekit-pytorch)
• Tokenizer Transplantation (mergekit-tokensurgeon)
• Citation

Why Merge Models?

Model merging is a powerful technique that allows combining the strengths of different models without the computational overhead of ensembling or the need for additional training. By operating directly in the weight space of models, merging can:

• Combine multiple specialized models into a single versatile model
• Transfer capabilities between models without access to training data
• Find optimal trade-offs between different model behaviors
• Improve performance while maintaining inference costs
• Create new capabilities through creative model combinations

Unlike traditional ensembling which requires running multiple models, merged models maintain the same inference cost as a single model while often achieving comparable or superior performance.

Features

Key features of mergekit include:

• Supports Llama, Mistral, GPT-NeoX, StableLM, and more
• Many merge methods
• GPU or CPU execution
• Lazy loading of tensors for low memory use
• Interpolated gradients for parameter values (inspired by Gryphe's BlockMerge_Gradient script)
• Piecewise assembly of language models from layers ("Frankenmerging")
• Mixture of Experts merging
• LORA extraction
• Evolutionary merge methods
• Multi-stage merging for complex workflows.
• Merging of raw PyTorch models (mergekit-pytorch).

Installation

git clone https://github.com/arcee-ai/mergekit.git
cd mergekit

pip install -e .  # install the package and make scripts available

If the above fails with the error of:

ERROR: File "setup.py" or "setup.cfg" not found. Directory cannot be installed in editable mode:
(A "pyproject.toml" file was found, but editable mode currently requires a setuptools-based build.)

You may need to upgrade pip to > 21.3 with the command python3 -m pip install --upgrade pip.

Community & Support

• Issues: GitHub Issues
• Discussions: Arcee Discord

Contributing

We welcome contributions to mergekit! If you have ideas for new merge methods, features, or other improvements, please check out our contributing guide for details on how to get started.

Community Tools

• FrankensteinAI: For those who prefer a browser-based experience without local setup or hardware wrangling, the team at FrankensteinAI has built a hosted platform powered by mergekit. Also features a community gallery and leaderboard for sharing and comparing merged models.

Usage

The script mergekit-yaml is the main entry point for mergekit. It takes a YAML configuration file and an output path, like so:

mergekit-yaml path/to/your/config.yml ./output-model-directory [--cuda] [--lazy-unpickle] [--allow-crimes] [... other options]

This will run the merge and write your merged model to ./output-model-directory.

For more information on the arguments accepted by mergekit-yaml run the command mergekit-yaml --help.

In-memory Python API

Use merge_tensors for one output tensor, or merge_state_dicts for modules and state dictionaries already in memory:

from mergekit.merge_methods import merge_state_dicts, merge_tensors

merged_tensor = merge_tensors(
    [tensor_a, tensor_b], "linear", parameters={"weight": [0.25, 0.75]}
)
merged_weights = merge_state_dicts(
    [model_a, model_b], "linear", parameters={"weight": [0.25, 0.75]}
)

Both accept a method name or a method object, plus dtype and out_dtype. For base-aware methods, use base_index with merge_tensors or base with merge_state_dicts. See Defining Merge Methods for custom methods and batching.

Uploading to Huggingface

When you have a merged model you're happy with, you may want to share it on the Hugging Face Hub. mergekit generates a README.md for your merge with some basic information for a model card. You can edit it to include more details about your merge, like giving it a good name or explaining what it's good at; rewrite it entirely; or use the generated README.md as-is. It is also possible to edit your README.md online once it has been uploaded to the Hub.

Once you're happy with your model card and merged model, you can upload it to the Hugging Face Hub using the huggingface_hub Python library.

# log in to huggingface with an access token (must have write permission)
huggingface-cli login
# upload your model
huggingface-cli upload your_hf_username/my-cool-model ./output-model-directory .

The documentation for huggingface_hub goes into more detail about other options for uploading.

Merge Configuration

Merge configurations are YAML documents specifying the operations to perform in order to produce your merged model. Below are the primary elements of a configuration file:

• merge_method: Specifies the method to use for merging models. See Merge Methods for a list.
• slices: Defines slices of layers from different models to be used. This field is mutually exclusive with models.
• models: Defines entire models to be used for merging. This field is mutually exclusive with slices.
• base_model: Specifies the base model used in some merging methods.
• parameters: Holds various parameters such as weights and densities, which can also be specified at different levels of the configuration.
• dtype: Casts inputs to this dtype before merging, including downcasting when requested. If omitted, corresponding weights are promoted to a common dtype independently: matching bfloat16 weights stay bfloat16, while bfloat16 with float16 or float32 promotes to float32. Float64 inputs retain float64 precision.
• out_dtype: Casts merged weights after computation; it does not change input precision. If omitted, the method's output dtype is retained.
• tokenizer or tokenizer_source: Determines how to construct a tokenizer for the merged model.
• chat_template: Specifies a chat template for the merged model.

Methods control their intermediate precision independently of PyTorch autocast. Linear and SLERP compute in float32 for float16, bfloat16, and float32 inputs, and float64 for float64 inputs, then cast back to the aligned input dtype. Normalized linear merges require weights with a nonzero sum in the working precision; a zero sum produces NaN or infinity through tensor division. Nearly cancelling weights can lose accuracy. Architecture-specific forced dtypes take precedence over dtype and out_dtype for those weights.

Parameter Specification

Parameters are flexible and can be set with varying precedence. They can be specified conditionally using tensor name filters, which allows finer control such as differentiating between attention heads and fully connected layers.

Parameters can be specified as:

• Scalars: Numbers, booleans, or strings, according to the merge parameter's type.
• Gradients: Lists of values. Numeric endpoints are validated and interpolated; booleans and strings select discrete steps. Scientific notation such as [1e-5, 1e-3] is supported for numeric parameters.

The parameters can be set at different levels, with decreasing precedence as follows:

1. slices.*.sources.parameters - applying to a specific input slice
2. slices.*.parameters - applying to a specific output slice
3. models.*.parameters or input_model_parameters - applying to any tensors coming from specific input models
4. parameters - catchall

Tokenizer Configuration

The tokenizer behavior can be configured in two ways: using the new tokenizer field (recommended) or the legacy tokenizer_source field (maintained for backward compatibility). These fields are mutually exclusive - you should use one or the other, not both.

Modern Configuration (tokenizer)

The tokenizer field provides fine-grained control over vocabulary and embeddings:

By default, inputs with different vocabulary sizes must be aligned using this configuration (or tokenizer_source) before merging. Use source: base to retain the base vocabulary, source: union to combine vocabularies, or a model path to select its vocabulary. All non-vocabulary dimensions must match regardless of tokenizer configuration. Alignment applies to every vocabulary-indexed parameter, including output biases. Architecture definitions identify these using vocabulary_axis: a nonnegative axis index (0 for conventional embedding matrices and output biases), or null for tensors not indexed by token IDs. Positional, token-type, and vision embeddings are not vocabulary-indexed.

tokenizer:
  source: "union"  # or "base" or a specific model path
  tokens:          # Optional: configure specific tokens
    <token_name>:
      source: ...  # Specify embedding source
      force: false # Optional: force this embedding for all models
  pad_to_multiple_of: null  # Optional: pad vocabulary size

Tokenizer Source

The source field determines the vocabulary of the output model:

• union: Combine vocabularies from all input models (default)
• base: Use vocabulary from the base model
• "path/to/model": Use vocabulary from a specific model

Token Embedding Handling

When a tokenizer is configured, each input model's embedding matrix is adjusted to match the output vocabulary before being passed to the merge method. For tokens a model already has, its own embedding is used. For tokens a model is missing, a fallback embedding is assigned using these rules:

• If the base model has the token, use the base model's embedding
• If only one model has the token, use that model's embedding
• Otherwise, use an average of all available embeddings

The merge method then combines these per-model embeddings (original and filled-in) to produce the final output. This means the final embedding for a token present in multiple models is determined by your merge method (SLERP, linear, TIES, etc.), not simply taken from one model.

You can override these defaults for specific tokens. Any tokens listed here that don't already exist in the output vocabulary will be added automatically, making this useful for introducing new special tokens.

tokenizer:
  source: union
  tokens:
    # Use embedding from a specific model
    <|im_start|>:
      source: "path/to/chatml/model"

    # Force a specific embedding for all models
    <|special|>:
      source: "path/to/model"
      force: true

    # Map a token to another model's token embedding
    <|renamed_token|>:
      source:
        kind: "model_token"
        model: "path/to/model"
        token: "<|original_token|>"  # or use token_id: 1234

    # Use a zero embedding
    <|unused|>:
      source:
        kind: "zero"

Practical Example

Here's how you might preserve both Llama 3 Instruct and ChatML prompt formats when merging models:

tokenizer:
  source: union
  tokens:
    # ChatML tokens
    <|im_start|>:
      source: "chatml_model"
    <|im_end|>:
      source: "chatml_model"

    # Llama 3 tokens - force original embeddings
    <|start_header_id|>:
      source: "llama3_model"
      force: true
    <|end_header_id|>:
      source: "llama3_model"
      force: true
    <|eot_id|>:
      source: "llama3_model"
      force: true

Legacy Configuration (tokenizer_source)

For backward compatibility, the tokenizer_source field is still supported:

tokenizer_source: "union"  # or "base" or a model path

This provides basic tokenizer selection but lacks the fine-grained control of the modern tokenizer field.

Unsafe embedding truncation

mergekit-yaml config.yml output/ --unsafe-truncate-embeddings opts into tokenizer-unaware truncation. It discards trailing slices along each parameter's vocabulary_axis to match the smallest input; all other dimensions must match. Token IDs are not checked; every retained ID must have the same meaning in every input. Use tokenizer configuration to align tokens when this is not guaranteed.

• Disabled by default and independent of --allow-crimes.
• If tokenizer or tokenizer_source is configured, tokenizer alignment takes precedence and this option has no effect.
• Each truncation logs the tensor name and original and retained shapes.
• Configs and tokenizers are copied without adjusting vocabulary sizes or token IDs. They may disagree with the truncated weights, preventing the output from loading or generating correctly.

For direct Python callers, mergekit.merge_methods.preprocessing.truncate_vocabulary(group) performs the same preprocessing on a TensorGroup with vocabulary_axis set. It returns borrowed views, preserving input IDs and base designation. Merge methods require matching tensor shapes.

Chat Template Configuration

The optional chat_template field allows overriding the chat template used for the merged model.

chat_template: "auto"  # or a template name or Jinja2 template

Options include:

• "auto": Automatically select the most common template among input models
• Built-in templates: "alpaca", "chatml", "llama3", "mistral", "exaone"
• A Jinja2 template string for custom formatting

Examples

Several examples of merge configurations are available in examples/.

Merge Methods

mergekit offers many methods for merging models, each with its own strengths and weaknesses. Choosing the right method depends on your specific goals, the relationship between the models you're merging, and the desired characteristics of the final model.

For detailed explanations, parameter descriptions, and use cases for each method, please see our Merge Method Guide.

Method Overview

Method (value)	Core Idea	# Models	Base Model	Key Strengths / Use Cases
Linear (linear)	Simple weighted average of model parameters.	≥2	-	Averaging similar checkpoints, model soups.
SLERP (slerp)	Spherical linear interpolation between two models.	2	✓	Smoothly transitioning between two models.
NuSLERP (nuslerp)	Enhanced SLERP with flexible weighting.	2	*	More intuitive SLERP; task vector SLERP.
Multi-SLERP (multislerp)	Barycentric SLERP for multiple models.	≥2	*	Spherical interpolation for >2 models.
Karcher Mean (karcher)	Riemannian barycenter of model parameters.	≥2	-	Geometrically sound averaging on manifolds.
Task Arithmetic (task_arithmetic)	Linearly combine "task vectors" (differences from a base).	≥2	✓	Transferring/combining fine-tuned skills.
TIES (ties)	Task arithmetic + sparsification & sign consensus.	≥2	✓	Merging many models, reducing interference.
DARE (dare_linear, dare_ties)	Task arithmetic + random pruning & rescaling.	≥2	✓	Robust skill retention, similar to TIES.
DELLA (della, della_linear)	Task arithmetic + adaptive magnitude-based pruning.	≥2	✓	Prioritizing important changes, reducing interference.
Model Breadcrumbs (breadcrumbs, breadcrumbs_ties)	Task arithmetic + outlier removal (small & large diffs).	≥2	✓	Refining task vectors by removing extreme changes.
SCE (sce)	Task arithmetic + adaptive matrix-level weighting based on variance.	≥2	✓	Dynamically weighting models based on parameter variance.
Model Stock (model_stock)	Geometric weight calculation for linear interpolation.	≥3	✓	Finding good linear interpolation weights for many checkpoints.
Nearswap (nearswap)	Interpolate where parameters are similar.	2	✓	Selective merging based on parameter similarity.
Arcee Fusion (arcee_fusion)	Dynamic thresholding for fusing important changes.	2	✓	Identifying and merging salient features.
Passthrough (passthrough)	Directly copies tensors from a single input model.	1	-	Frankenmerging, layer stacking, model surgery.

Key for Base Model Column:

• ✓: Required - One of the input models must be designated as the base_model.
• *: Optional - One of the input models can be designated as the base_model.
• -: Not Applicable - base_model has no effect on this method.

LoRA Extraction

Mergekit allows extracting PEFT-compatible low-rank approximations of finetuned models.

Usage

mergekit-extract-lora --model finetuned_model_id_or_path --base-model base_model_id_or_path --out-path output_path [--no-lazy-unpickle] [--cuda] [--max-rank=desired_rank] [--sv-epsilon=tol]

Mixture of Experts Merging

The mergekit-moe script supports merging multiple dense models into a mixture of experts, either for direct use or for further training. For more details see the mergekit-moe documentation.

Evolutionary Merge Methods

See docs/evolve.md for details.

Multi-Stage Merging (mergekit-multi)

mergekit-multi enables the execution of complex, multi-stage model merging workflows. You can define multiple merge configurations in a single YAML file, where later merges can use the outputs of earlier ones as inputs. This is useful for building up sophisticated models through a series of targeted merges.

See the mergekit-multi documentation for usage details and examples.

Raw PyTorch Model Merging (mergekit-pytorch)

For merging arbitrary PyTorch models (not necessarily Hugging Face Transformers), mergekit-pytorch provides a way to apply mergekit's algorithms directly to .pt or .safetensors checkpoints. The configuration is similar to the YAML format used in mergekit-yaml, but does not support layer slicing or tokenizer configuration.

Usage

mergekit-pytorch path/to/your/raw_config.yml ./output_pytorch_model_directory [options]

Use mergekit-pytorch --help for detailed options.

Tokenizer Transplantation (mergekit-tokensurgeon)

mergekit-tokensurgeon is a specialized tool for transplanting tokenizers between models, allowing you to align the vocabulary of one model with another. This is particularly useful for cheaply producing draft models for speculative decoding or for cross-tokenizer knowledge distillation. See the documentation for more details and how to use it.

Vocabulary-dependent biases and other auxiliary parameters are remapped by exact token identity; entries for new tokens are initialized to zero. Embedding matrices use the selected approximation method.

Citation

If you find mergekit useful in your research, please consider citing the paper:

@inproceedings{goddard-etal-2024-arcees,
    title = "Arcee{'}s {M}erge{K}it: A Toolkit for Merging Large Language Models",
    author = "Goddard, Charles  and
      Siriwardhana, Shamane  and
      Ehghaghi, Malikeh  and
      Meyers, Luke  and
      Karpukhin, Vladimir  and
      Benedict, Brian  and
      McQuade, Mark  and
      Solawetz, Jacob",
    editor = "Dernoncourt, Franck  and
      Preo{\c{t}}iuc-Pietro, Daniel  and
      Shimorina, Anastasia",
    booktitle = "Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track",
    month = nov,
    year = "2024",
    address = "Miami, Florida, US",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2024.emnlp-industry.36",
    doi = "10.18653/v1/2024.emnlp-industry.36",
    pages = "477--485",
    abstract = "The rapid growth of open-source language models provides the opportunity to merge model checkpoints, combining their parameters to improve performance and versatility. Advances in transfer learning have led to numerous task-specific models, which model merging can integrate into powerful multitask models without additional training. MergeKit is an open-source library designed to support this process with an efficient and extensible framework suitable for any hardware. It has facilitated the merging of thousands of models, contributing to some of the world{'}s most powerful open-source model checkpoints. The library is accessible at: https://github.com/arcee-ai/mergekit.",
}