EasyEdit

---

✨ EasyEdit blog published! We have also published a blog post titled "Take Control of What Your LLM Knows and Does — with the EasyEdit Tool Series", where you can learn more about how EasyEdit empowers you to control and edit your LLM’s knowledge and behavior.  

Table of Contents

• Table of Contents
• 🔔News
• Editing Demo
• Knowledge Editing
  • Task Definition
    • Knowledge insert
    • Knowledge update
    • Knowledge erase
  • Comparisons of the different technologies
  • Evaluation
• 🌟Overview
  • Current Implementation
  • Quick Start on Some Works
  • Tutorial notebook
• Requirements
  • 🔧Pip Installation
  • ⚡uv Installation
  • Editing GPU memory usage
• 📌Use EasyEdit
  • BaseEditor
    • Introduction by a Simple Example
  • Evaluation
  • Trainer
• Use EasyEdit with KnowEdit
  • Dataset
  • Usage
• Editing Performance
• Citation
• 🎉Contributors
  • Other Related Projects

🔔News

• 2026-07-13, 🎉 EasyEdit has been selected for the CCF ODTC open source incentive program.
• 2026-07-08, 🚀 EasyEdit has received a major upgrade, bringing Transformers 5.x compatibility, new models support, more reliable multi-GPU editing, enhanced multimodal pipelines, a unified EasyEdit2 steering workflow, expanded vLLM support, and faster multi-worker deployment.
• 2026-03-04, 🎉 SteerEval is released — a hierarchical benchmark for evaluating LLM controllability across behavioral domains and granularity levels, with an automated data synthesis pipeline.
• 2025-10-12, ⚡ LightMem is released — a lightweight and efficient memory framework empowering LLMs and AI agents with long-term memory capabilities!
• 2025-10-02, 👑 SimIE has arrived — a general framework for lifelong model editing, which restores the strong performance of parameter-modifying methods from standard model editing in a lifelong context!
• 2025-09-10, 🎉🎉 EasyEdit2 Paper has been accepted by the EMNLP 2025 System Demonstration Track.
• 2025-07-24, 🚀🚀the EasyEdit has added three new unstructured long-form knowledge editing datasets AKEW, LEME and UNKE. In addition, EasyEdit also incorporated two of the currently most popular unstructured editing methods UNKE and AnyEdit. EasyEdit2 also supports the representation steering method RePS and provides initial support for AxBench-style evaluation.
• 2025-06-07, 👑 UltraEdit has arrived — powered by a lifelong normalization strategy that continuously updates feature statistics across turns, it can edit 20K samples on a 7B model in just 5 minutes and scales stably to millions !
• 2025-06-05, 🌟🌟the EasyEdit has added a new model editing algorithm CORE, designed to strengthen context robustness by minimizing context-sensitive variance in hidden states of the model for edited knowledge.
• 2025-05-28, 🌟🌟the EasyEdit has added a new model editing algorithm NAMET, which introduces noise during memory extraction via a one-line modification to MEMIT. Thanks to @ybdai7 for contribution!
• 2025-05-15, 🚀🚀We released a new blog Reflection on Knowledge Editing: Charting the Next Steps discussing the next step for knowledge editing research.
• 2025-04-03, 🚀🚀We have upgraded EasyEdit and introduced EasyEdit2! EasyEdit2 provides an easy-to-use steering framework for editing large language models with precision and flexibility. This upgrade enhances the framework’s capabilities, making it more robust and adaptable for various steering methods. More details can be found in our paper, in the README and on our website.

• 2025-03-04, 🌟🌟In addition to the original token-level teacher-forcing paradigm for evaluation, EasyEdit has integrated a new evaluation method, following the paper titled "The Mirage of Model Editing: Revisiting Evaluation in the Wild". You can use this script to quickly launch this evaluation approach, which better aligns with real-world requirements. Special thanks to @WanliYoung for contribution!

• 2025-01-03, We have updated the evaluation method in paper "CKnowEdit: A New Chinese Knowledge Editing Dataset for Linguistics, Facts, and Logic Error Correction in LLMs", adopting an LLM-as-a-judge approach that is more aligned with real-world scenarios and practical applications. Both the experimental results and the running scripts have been updated accordingly in readme.

• 2024-11-19, we update the Table 4 results in the paper "A Comprehensive Study of Knowledge Editing for Large Language Models" after optimizing certain methods (related to AdaLoRA) and fixing computational bugs (related to ROME and MEMIT) in the EasyEdit (More details in https://github.com/zjunlp/EasyEdit/issues/427). These improvements have led to better results than before. We will continue updating this paper and welcome everyone to discuss and exchange ideas.

• 2024-11-11, 🎉🎉the paper on model editing for LLMs4Code, "Model Editing for LLMs4Code: How Far are We?", has been accepted by ICSE 2025! This work proposes a benchmark for LLMs4Code editing, CLMEEval, which is built upon EasyEdit!

• 2024-11-09, we fixed a bug regarding the KnowEdit results in the https://github.com/zjunlp/EasyEdit/issues/390. Thanks for the help of @StarLooo to help us with it.

• 2024-10-24, EasyEdit has added two new knowledge editing methods, AlphaEdit. In addition, we have fixed several bugs.

• 2024-10-23, the EasyEdit integrates constrained decoding methods from steering editing to mitigate hallucination in LLM and MLLM, with detailed information available in DoLa and DeCo.

• 2024-09-26, 🎉🎉 our paper "WISE: Rethinking the Knowledge Memory for Lifelong Model Editing of Large Language Models" has been accepted by NeurIPS 2024.

• 2024-09-20, 🎉🎉 our papers: "Knowledge Mechanisms in Large Language Models: A Survey and Perspective" and "Editing Conceptual Knowledge for Large Language Models" have been accepted by EMNLP 2024 Findings.

• 2024-07-29, the EasyEdit has added a new model editing algorithm EMMET, which generalizes ROME to the batch setting. This essentially allows making batched edits using the ROME loss function.

• 2024-07-23, we release a new paper: "Knowledge Mechanisms in Large Language Models: A Survey and Perspective", which reviews how knowledge is acquired, utilized, and evolves in large language models. This survey may provide the fundamental mechanisms for precisely and efficiently manipulating (editing) knowledge in LLMs.

• 2024-06-04, 🎉🎉 EasyEdit Paper has been accepted by the ACL 2024 System Demonstration Track.

• 2024-06-03, we released a paper titled "WISE: Rethinking the Knowledge Memory for Lifelong Model Editing of Large Language Models", along with introducing a new editing task: Continuous Knowledge Editing and correspondding lifelong editing method called WISE.

• 2024-04-24, EasyEdit announced support for the ROME method for Llama3-8B. Users are advised to update their transformers package to version 4.40.0.

• 2024-03-29, EasyEdit introduced rollback support for GRACE. For a detailed introduction, refer to the EasyEdit documentation. Future updates will gradually include rollback support for other methods.

• 2024-03-22, a new paper titled "Detoxifying Large Language Models via Knowledge Editing" was released, along with a new dataset named SafeEdit and a new detoxification method called DINM.

• 2024-03-12, another paper titled "Editing Conceptual Knowledge for Large Language Models" was released, introducing a new dataset named ConceptEdit.

• 2024-03-01, EasyEdit added support for a new method called FT-M. This method involves training a specific MLP layer using cross-entropy loss on the target answer and masking the original text. It outperforms the FT-L implementation in ROME. The author of issue https://github.com/zjunlp/EasyEdit/issues/173 is thanked for their advice.

• 2024-02-27, EasyEdit added support for a new method called InstructEdit, with technical details provided in the paper "InstructEdit: Instruction-based Knowledge Editing for Large Language Models".

• 2024-02-09, the EasyEdit has added the support for the Dynamic LoRA model editing method MELO'AAAI24.
• 2024-02-06, we release a new paper: "EasyInstruct: An Easy-to-use Instruction Processing Framework for Large Language Models" with an HF demo EasyInstruct.
• 2024-02-06, we release a preliminary tool EasyDetect for LLM hallucination detection，with a demo.
• 2024-01-24, the EasyEdit has added the support for editing Mistral-7B (manually update transformers==4.34.0), we have also fixed some bugs in evaluating MEND (slightly influence the performance).
• 2024-01-16, the EasyEdit has added the support for the precise model editing method PMET'AAAI24.
• 2024-01-03, we release a new paper:"A Comprehensive Study of Knowledge Editing for Large Language Models" with a new benchmark KnowEdit! KnowEdit is constructed by re-organizing and cleaning existing datasests including WikiBio, ZsRE, WikiData Counterfact, WikiData Recent, convsent, Sanitation with new train/val/test spliting. Special thanks to the builders and maintainers of the those datasets.We are looking forward to any comments or discussions on this topic :)
• 2023-12-06, the EasyEdit has added the support for the lifelong model editing method GRACE'NeurIPS24.
• 2023-11-18, our tutorial "Knowledge Editing for Large Language Models" has been accepted by COLING 2024.
• 2023-10-25, our tutorial "Knowledge Editing for Large Language Models" has been accepted by AAAI 2024.
• 2023-10-24, the EasyEdit has added the support for efficient editing of Baichuan2, ChatGLM2, InternLM, QWen and fixed several bugs for a better user experience.
• 2023-10-14, we release the MultimodalEditor based on the paper "Can We Edit Multimodal Large Language Models?".
• 2023-10-13, we release the paper "Can We Edit Multimodal Large Language Models?" accepted by EMNLP 2023.
• 2023-10-08, our paper "Editing Large Language Models: Problems, Methods, and Opportunities" has been accepted by EMNLP 2023.
• 2023-10-07, the EasyEdit has added the support for editing models with multiple GPUs, using huggingface Accelerate.
• 2023-9-21, the EasyEdit has added the support for Parameter-Efficient Fine-Tuning through AdaLoRA to inject knowledge into the LLM.
• 2023-8-31, the EasyEdit has added the support for official fine-tuning API for gpt-3.5-turbo to customize ChatGPT for your editing cases.
• 2023-8-15, we release the paper "EasyEdit: An Easy-to-use Knowledge Editing Framework for Large Language Models."
• 2023-7-12, we release version 0.0.1, supporting several knowledge editing techniques for LLMs. EasyEdit helps to better align LLMs with changing needs and values of users.
• 2023-5-22, we release the paper "Editing Large Language Models: Problems, Methods, and Opportunities" and provide a paper list at PaperList.
• 2023-3-25, the EasyEdit project has been launched and is under development.

---

A Comprehensive Study of Knowledge Editing for Large Language Models [paper][benchmark][code]

IJCAI 2024 Tutorial Google Drive

COLING 2024 Tutorial Google Drive

AAAI 2024 Tutorial Google Drive

AACL 2023 Tutorial [Google Drive] [Baidu Pan]

Editing Demo

There is a demonstration of editing. The GIF file is created by Terminalizer.

We provide a handy Jupyter Notebook! It allows you to edit a LLM's knowledge of the US president, switching from Biden to Trump and even back to Biden. This includes methods like WISE, AlphaEdit, AdaLoRA, and Prompt-based editing.

Knowledge Editing

Task Definition

Deployed models may still make unpredictable errors. For example, LLMs notoriously hallucinate, perpetuate bias, and factually decay, so we should be able to adjust specific behaviors of pre-trained models.

Knowledge editing aims to adjust base model's $(f_\theta)$ behavior on the particular edit descriptor $[x_e, y_e]$​​​ efficiently.

Multi Setting

Single Knowledge Editing

Evaluating the performance of the model after a single edit. The model reloads the original weights (e.g. LoRA discards the adapter weights) after a single edit. You should set sequential_edit=False

$$\theta' \leftarrow \text{arg} \min\limits_{\theta} (\Vert f_\theta(x_e) - y_e \Vert)$$

Continuous Knowledge Editing

This requires sequentially editing, and evaluation is performed after all knowledge updates have been applied:

$$\theta' \leftarrow \text{arg} \min\limits_{\theta} \sum_{e=1}^{\Vert X_e \Vert} (\Vert f_\theta(x_e) - y_e \Vert)$$

It makes parameter adjustments for $(x_e, y_e)$, where $x_e \in X_e$ and $f_\theta'(x_e) = y_e$. Here, $X_e$​ represents the whole edit set. To enable continuous editing, you can set sequential_edit=True: README (for more details).

Multi Scenario

Knowledge insert

• Inject knowledge that LLMs have not seen before. such as:
  • How many times has Messi won the World Cup? 0 $\rightarrow$ 1:

Knowledge update

• Update outdated knowledge. such as:
  • The president of USA: Donald Trump $\rightarrow$ Joe Biden:

Knowledge erase

• Erase sensitive information. such as:
  • The phone number of someone is XXXX $\rightarrow$ __

Without influencing the model behavior on unrelated samples, the ultimate goal is to create an edited model $(f_\theta')$​​.

Editing Task for Image Captioning and Visual Question Answering. README

The proposed task takes the preliminary attempt to edit LLMs' personalities by editing their opinions on specific topics, given that an individual's opinions can reflect aspects of their personality traits. We draw upon the established BIG FIVE theory as a basis for constructing our dataset and assessing the LLMs' personality expressions. README

Evaluation

Logits-based

• ES: evaluating the editing success rate based on the logits of pre-generated text.
• DD: evaluating whether the model changes opinions on other topics based on the logits of pre-generated text.

Generation-based

• Acc: the accuracy of the generated text after editing the model on target personality.
• TPEI: measuring whether generated opinion text from the edited model leans more towards the target personality.
• PAE: utilizing GPT-4 to evaluate the personality traits in generated text.

While for assessing Acc and TPEI, you can download the trained classifier from here.

Comparisons of different technologies

Evaluation

The knowledge editing process generally impacts the predictions for a broad set of inputs that are closely associated with the edit example, called the editing scope.

A successful edit should adjust the model’s behavior within the editing scope while remaining unrelated inputs:

$$ f_{\theta_{e}}(x) = \begin{cases} y_e & \text{if } x \in I(x_e,y_e) \ f_{\theta}(x) & \text{if } x \in O(x_e, y_e) \end{cases} $$

• Reliability: the success rate of editing with a given editing descriptor
• Generalization: the success rate of editing within the editing scope
• Locality: whether the model's output changes after editing for unrelated inputs
• Portability: the success rate of editing for reasoning/application(one hop, synonym, logical generalization)
• Efficiency: time and memory consumption

🌟Overview

EasyEdit is a Python package for edit Large Language Models (LLM) like GPT-J, Llama, GPT-NEO, GPT2, T5(support models from 1B to 65B), the objective of which is to alter the behavior of LLMs efficiently within a specific domain without negatively impacting performance across other inputs. It is designed to be easy to use and easy to extend.

• EasyEdit contains a unified framework for Editor, Method and Evaluate, respectively representing the editing scenario, editing technique, and evaluation method.

• Each Knowledge Editing scenario comprises of three components:

  • Editor: such as BaseEditor(Factual Knowledge and Generation Editor) for LM, MultiModalEditor(MultiModal Knowledge).
  • Method: the specific knowledge editing technique used(such as ROME, MEND, ..).
  • Evaluate: Metrics for evaluating knowledge editing performance.
    • Reliability, Generalization, Locality, Portability

• The current supported knowledge editing techniques are as follows:

  • Memory-based: SERAC, IKE, GRACE, MELO, WISE
  • Meta-learning: MEND, InstructEdit, MALMEN
  • Locate-then-edit: KN, ROME, MEMIT, PMET, DINM, R-ROME, EMMET
  • FT-L

  Note 1: Due to the limited compatibility of this toolkit, some knowledge editing methods including T-Patcher, KE, CaliNet are not supported.

  Note 2: Similarly, the MALMEN method is only partially supported due to the same reasons and will continue to be improved.

Current Implementation

You can choose different editing methods according to your specific needs. GPT series includes GPT-2, GPT-J and GPT-NEO.

Method	T5	GPT series	LlaMA	Baichuan	ChatGLM	InternLM	Qwen	Mistral
FT	✅	✅	✅	✅	✅	✅	✅	✅
AdaLoRA			✅		✅
SERAC	✅	✅	✅
IKE	✅	✅	✅	✅	✅	✅	✅	✅
MEND	✅	✅	✅	✅	✅	✅	✅	✅
KN	✅	✅	✅	✅	✅	✅	✅	✅
ROME		✅	✅	✅	✅	✅	✅	✅
r-ROME		✅	✅	✅	✅	✅	✅	✅
MEMIT		✅	✅	✅	✅	✅	✅	✅
EMMET		✅	✅
GRACE		✅	✅
MELO		✅
PMET		✅	✅
InstructEdit		✅	✅
DINM		✅	✅					✅
WISE		✅	✅	✅			✅
Defer		✅	✅					✅
AlphaEdit		✅	✅
UltraEdit		✅	✅				✅	✅

❗️❗️ If you intend to use Mistral, please update the transformers library to version 4.34.0 manually. You can use the following code: pip install transformers==4.34.0.

We also support some knowledge editing methods for LMMs (Large Multimodal Models). Please refer to MultimodalEdit.md for more details.

Method	MiniGPT-4	BLIP-2	LLaVA-OneVision	Qwen2-VL
MEND	✅	✅
SERAC	✅	✅
IKE			✅	✅
LoRA			✅	✅
GRACE			✅	✅
WISE			✅	✅

Quick Start on Some Works

Work	Description	Path
InstructEdit	InstructEdit: Instruction-based Knowledge Editing for Large Language Models	Quick Start
DINM	Detoxifying Large Language Models via Knowledge Editing	Quick Start
WISE	WISE: Rethinking the Knowledge Memory for Lifelong Model Editing of Large Language Models	Quick Start
ConceptEdit	Editing Conceptual Knowledge for Large Language Models	Quick Start
MMEdit	Can We Edit Multimodal Large Language Models?	Quick Start
PersonalityEdit	Editing Personality For Large Language Models	Quick Start
PROMPT	PROMPT-based knowledge editing methods	Quick Start

Dataset

Benchmark: KnowEdit [Hugging Face][WiseModel][ModelScope]

❗️❗️ To be noted, KnowEdit is constructed by re-organizing and extending existing datasests including WikiBio, ZsRE, WikiDataCounterfact, WikiDataRecent, convsent, Sanitation to make a comprehensive evaluation for knowledge editing. Special thanks to the builders and maintainers of the those datasets.

Please note that Counterfact and WikiDataCounterfact are not the same dataset.

We provide detailed scripts for user to easily use KnowEdit, please refer to examples.

• ZsRE: is a context-free question-answering task. Given a question based on the subject and relation, the model is expected to provide the correct object as the answer.
• Wikirecent: This dataset specifically focuses on triplets that have been recently inserted into WikiData after July 2022.
• WikiBio: The original dataset was created by prompting GPT-3 to generate 238 Wikipedia-style biographies using subjects from the WikiBio.
• WikiDatacounterfact: Since tail entities are often not captured by models, and therefore are not suitable for testing modification edits, RippleEdit collects triplets about popular entities, where the subject corresponds to one of the top-viewed pages in Wikipedia.
• Convsent: This is a sentiment editing task that assesses the model's ability to modify a dialog agent's sentiment on a specific topic without affecting its responses to other topics.
• Sanitation: This dataset specifically addresses privacy concerns associated with learned language models.

knowedit
├── WikiBio
│   ├── wikibio-test-all.json
│   └── wikibio-train-all.json
├── ZsRE
│   └── ZsRE-test-all.json
├── wiki_counterfact
│   ├── test_cf.json
│   └── train_cf.json
├── convsent
│   ├── blender_test.json
│   ├── blender_train.json
│   └── blender_val.json
├── convsent
│   ├── trivia_qa_test.json
│   └── trivia_qa_train.json
└── wiki_recent
    ├── recent_test.json
    └── recent_train.json

---

Datasets for Chinese Knowledge: CKnowEdit

dataset	HuggingFace	WiseModel	ModelScope	Description
CKnowEdit	[HuggingFace]	[WiseModel]	[ModelScope]	dataset for editing Chinese Knowledge

• Here, you can follow CKnowEdit.md to find more details about CKnowEdit and run Chinese knowledge editing experiments.

CKnowEdit is a high-quality Chinese-language dataset for knowledge editing which is highly characterized by the Chinese language, with all data sourced from Chinese knowledge bases. It is meticulously designed to more deeply discern the nuances and challenges inherent in the comprehension of the Chinese language by current LLMs, providing a robust resource for refining Chinese-specific knowledge within LLMs.

The field descriptions for the data in CKnowEdit are as follows:

"prompt": query inputed to the model (str)
"target_old": the incorrect response previously generated by the model (str)
"target_new": the accurate answer of the prompt (str)
"portability_prompt": new prompts related to the target knowledge (list or None)
"portability_answer": accurate answers corresponding to the portability_prompt (list or None)
"locality_prompt": new prompts unrelated to the target knowledge (list or None)
"locality_answer": accurate answers corresponding to the locality_prompt (list or None)
"rephrase": alternative ways to phrase the original prompt (list)

CknowEdit
├── Chinese Literary Knowledge
│   ├── Ancient Poetry
│   ├── Proverbs
│   └── Idioms
├── Chinese Linguistic Knowledge
│   ├── Phonetic Notation
│   └── Classical Chinese
├── Chinese Geographical Knowledge
└── Ruozhiba

---

Datasets for Factual Knowledge

dataset	Google Drive	BaiduNetDisk	Description
ZsRE plus	[Google Drive]	[BaiduNetDisk]	Question Answering dataset using question rephrasings
Counterfact plus	[Google Drive]	[BaiduNetDisk]	Counterfact dataset using Entity replacement

We provide zsre and counterfact datasets to verify the effectiveness of knowledge editing. You can download them here. [Google Drive], [BaiduNetDisk].

• For locality, in addition to testing unrelated instances, we also provide tests on distracting (reference: Detecting Edit Failures...), other attribution, and other downstream tasks (such as commonsense reasoning).
• For portability, it tests whether the model can apply edited instances for inference. We provide evaluations for one-hop reasoning, subject alias, and inverse relation (eg, a one-to-one relationship between spouses should be bidirectionally edited).

editing-data
├── counterfact
│   ├── counterfact-edit.json
│   ├── counterfact-train.json
│   └── counterfact-val.json
├── locality
│   ├── Commonsense Task
│   │   ├── piqa_valid-labels.lst
│   │   └── piqa_valid.jsonl
│   ├── Distracting Neighbor
│   │   └── counterfact_distracting_neighbor.json
│   └── Other Attribution
│       └── counterfact_other_attribution.json
├── portability
│   ├── Inverse Relation
│   │   └── zsre_inverse_relation.json
│   ├── One Hop
│   │   ├── counterfact_portability_gpt4.json
│   │   └── zsre_mend_eval_portability_gpt4.json
│   └── Subject Replace
│       ├── counterfact_subject_replace.json
│       └── zsre_subject_replace.json
└── zsre
    ├── zsre_mend_eval.json
    ├── zsre_mend_train_10000.json
    └── zsre_mend_train.json

• counterfact: original counterfact dataset using Entity replacement
• zsre: original question answering dataset using question rephrasings
• locality (evaluation for locality, see details in this paper)
  • Commonsense Task: evaluation for other downstream tasks such as commonsense task
  • Distracting Neighbor: test on distracting neighborhood (reference: Detecting Edit Failures...)
  • Other Attribution
• portability
  • Inverse Relation: evaluation for one-to-one relationship such as spouse
  • One Hop: evaluation for one-hop reasoning
  • Subject Replace: evaluation for synonym replacement

---

Datasets for Conceptual Knowledge: ConceptEdit

dataset	Google Drive	HuggingFace Dataset	Description
ConceptEdit	[Google Drive]	[HuggingFace Dataset]	dataset for editing conceptual knowledge

• Here, you can follow ConceptEdit.md to run concept editing experiments.

data
└──concept_data.json
    ├──final_gpt2_inter.json
    ├──final_gpt2_intra.json
    ├──final_gptj_inter.json
    ├──final_gptj_intra.json
    ├──final_llama2chat_inter.json
    ├──final_llama2chat_intra.json
    ├──final_mistral_inter.json
    └──final_mistral_intra.json

Concept Specific Evaluation Metrics

• Instance Change: capturing the intricacies of these instance-level changes
• Concept Consistency: the semantic similarity of generated concept definition

---

Datasets for Multimodal Knowledge: MMEdit

dataset	Google Drive	BaiduNetDisk	Description
E-IC	[Google Drive]	[BaiduNetDisk]	dataset for editing Image Captioning
E-VQA	[Google Drive]	[BaiduNetDisk]	dataset for editing Visual Question Answering
ComprehendEdit	[Google Drive]	[BaiduNetDisk]	dataset for editing Visual Question Answering

• All images used in E-IC and E-VQA are available for download at Google Drive
• For locality, it is the same as factual editing in order to measure whether unrelated facts retain their outputs.
• For multimodal locality, it assesses the impact of editing on the visual module, which is similar to regular locality.
• The details of the impact of editing on other in-domain samples are in ComprehendEdit.

editing-data
├── caption
│   ├── caption_train_edit.json
│   └── caption_eval_edit.json
├── locality
│   ├── NQ dataset
│   │   ├── train.json
│   │   └── validation.json
├── multimodal_locality
│   ├── OK-VQA dataset
│   │   ├── okvqa_loc.json
└── vqa
    ├── vqa_train.json
    └── vqa_eval.json

• Multimodal locality (evaluation for multimodal locality, see dataset's details in this paper)

---

Datasets for detoxifying LLMs: SafeEdit

dataset	HuggingFace Dataset	Description
SafeEdit	[HuggingFace Dataset]	dataset for detoxifying LLMs

• Here, you can follow SafeEdit.md to run detoxification editing experiments.

data
└──SafeEdit_train.json
└──SafeEdit_val.json
└──SafeEdit_test.json
    

Detoxifying Specific Evaluation Metrics

• Defense Duccess (DS): the detoxification success rate of edited LLM for adversarial input (attack prompt + harmful question), which is used to modify LLM.
• Defense Generalization (DG): the detoxification success rate of edited LLM for out-of-domain malicious inputs.
• General Performance: the side effects for unrelated task performance.

Tutorial notebook

Method	Description	GPT-2	LlaMA
IKE	In-Context Learning (ICL) Edit	[Colab-gpt2]	[Colab-llama]
ROME	Locate-Then-Edit Neurons	[Colab-gpt2]	[Colab-llama]
MEMIT	Locate-Then-Edit Neurons	[Colab-gpt2]	[Colab-llama]

---

Datasets for Real-World Lifelong Knowledge Editing: WikiBigEdit

dataset	HuggingFace Dataset	Description
WikiBigEdit	[HuggingFace Dataset]	500k knowledge edits from Wikidata

• Here, you can follow run_WikiBigEdit.py to run WikiBigEdit edits using EasyEdit.

• If you want to learn more about WikiBigEdit, please refer to the following paper: Understanding the Limits of Lifelong Knowledge Editing in LLMs

Requirements

🔧Pip Installation

Note: Please use Python 3.9+ for EasyEdit To get started with pip, simply install conda and run:

git clone https://github.com/zjunlp/EasyEdit.git
cd EasyEdit
conda create -n EasyEdit python=3.10
conda activate EasyEdit
pip install -r requirements.txt

⚡uv Installation

You can also use uv for faster dependency installation:

git clone https://github.com/zjunlp/EasyEdit.git
cd EasyEdit
conda create -n EasyEdit python=3.10
conda activate EasyEdit
pip install uv
uv pip install --torch-backend cu128 -r requirements.txt

If you are installing for a CPU-only environment, replace the last command with:

uv pip install --torch-backend cpu -r requirements.txt

After installation, you can verify the environment with:

uv pip check

Editing GPU memory usage

Our results are all based on the default configuration

	llama-2-7B	chatglm2	gpt-j-6b	gpt-xl
FT	60GB	58GB	55GB	7GB
SERAC	42GB	32GB	31GB	10GB
IKE	52GB	38GB	38GB	10GB
MEND	46GB	37GB	37GB	13GB
KN	42GB	39GB	40GB	12GB
ROME	31GB	29GB	27GB	10GB
MEMIT	33GB	31GB	31GB	11GB
AdaLoRA	29GB	24GB	25GB	8GB
GRACE	27GB		23GB	6GB
WISE	34GB		27GB	7GB

📌Use EasyEdit

• Edit large language models(LLMs) around 5 seconds

• Following example shows you how to perform editing with EasyEdit. More examples and tutorials can be found at examples

BaseEditor

BaseEditoris the class for Language Modality Knowledge Editing. You can choose the appropriate editing method based on your specific needs.

• Due to different transformer versions and different GPU models, the editing results may fluctuate slightly.

Introduction by a Simple Example

With the modularity and flexibility of EasyEdit, you can easily use it to edit model.

Step1: Define a PLM as the object to be edited. Choose the PLM to be edited. EasyEdit supports partial models(T5, GPTJ, GPT-NEO, LlaMA so far) retrievable on HuggingFace. The corresponding configuration file directory is hparams/YUOR_METHOD/YOUR_MODEL.YAML, such as hparams/MEND/gpt2-xl.yaml, set the corresponding model_name to select the object for knowledge editing.

model_name: gpt2-xl
model_class: GPT2LMHeadModel
tokenizer_class: GPT2Tokenizer
tokenizer_name: gpt2-xl
model_parallel: false # true for multi-GPU editing

Step2: Choose the appropriate Knowledge Editing Method

## In this case, we use MEND method, so you should import `MENDHyperParams`
from easyeditor import MENDHyperParams
## Loading config from hparams/MEMIT/gpt2-xl.yaml
hparams = MENDHyperParams.from_hparams('./hparams/MEND/gpt2-xl')

Step3: Provide the edit descriptor and edit target

## edit descriptor: prompt that you want to edit
prompts = [
    'What university did Watts Humphrey attend?',
    'Which family does Ramalinaceae belong to',
    'What role does Denny Herzig play in football?'
]
## You can set `ground_truth` to None !!!(or set to original output)
ground_truth = ['Illinois Institute of Technology', 'Lecanorales', 'defender']
## edit target: expected output
target_new = ['University of Michigan', 'Lamiinae', 'winger']

Step4: Combine them into a BaseEditor EasyEdit provides a simple and unified way to init Editor, like huggingface: from_hparams.

## Construct Language Model Editor
editor = BaseEditor.from_hparams(hparams)

Step5: Provide the data for evaluation Note that the data for portability and locality are both optional(set to None for basic editing success rate evaluation only). The data format for both is a dict, for each measurement dimension, you need to provide the corresponding prompt and its corresponding ground truth. Here is an example of the data:

locality_inputs = {
    'neighborhood':{
        'prompt': ['Joseph Fischhof, the', 'Larry Bird is a professional', 'In Forssa, they understand'],
        'ground_truth': ['piano', 'basketball', 'Finnish']
    },
    'distracting': {
        'prompt': ['Ray Charles, the violin Hauschka plays the instrument', 'Grant Hill is a professional soccer Magic Johnson is a professional', 'The law in Ikaalinen declares the language Swedish In Loviisa, the language spoken is'],
        'ground_truth': ['piano', 'basketball', 'Finnish']
    }
}

In the above example, we evaluate the performance of the editing methods about "neighborhood" and "distracting".

Step6: Edit and Evaluation Done! We can conduct Edit and Evaluation for your model to be edited. The edit function will return a series of metrics related to the editing process as well as the modified model weights. [sequential_edit=True for continuous editing]

metrics, edited_model, _ = editor.edit(
    prompts=prompts,
    ground_truth=ground_truth,
    target_new=target_new,
    locality_inputs=locality_inputs,
    sequential_edit=False # True: start continuous editing ✈️
)
## metrics: edit success, rephrase success, locality e.g.
## edited_model: post-edit model

The maximum input length for EasyEdit is 512. If this length is exceeded, you will encounter the error "CUDA error: device-side assert triggered." You can modify the maximum length in the following file:LINK

Step7: RollBack In sequential editing, if you are not satisfied with the outcome of one of your edits and you do not wish to lose your previous edits, you can use the rollback feature to undo your previous edit. Currently, we only support the GRACE method. All you need to do is a single line of code, using the edit_key to revert your edit.

editor.rolllback('edit_key')

In EasyEdit, we default to using target_new as the edit_key

Evaluation

We specify the return metrics as dict format, including model prediction evaluations before and after editing. For each edit, it will include the following metrics:

• rewrite_acc $\rightarrow$ Reliablilty
• rephrase_acc $\rightarrow$ Generalization
• locality $\rightarrow$ Locality
• portablility $\rightarrow$ Portablility

{
    "post": {
        "rewrite_acc": ,
        "rephrase_acc": ,
        "locality": {
            "YOUR_LOCALITY_KEY": ,
            //...
        },
        "portablility": {
            "YOUR_PORTABILITY_KEY": ,
            //...
        },
    },
    "pre": {
        "rewrite_acc": ,
        "rephrase_acc": ,
        "portablility": {
            "YOUR_PORTABILITY_KEY": ,
            //...
        },
    }
}

• For evaluation for Reliablilty, you only need to provide the corresponding editing prompts and editing target_new.
• For evaluation for Generalization, rephrase_prompts are required.
• For evaluation for Locality and Portablility, you need to define the name of the corresponding metric, as well as prompts and ground_truth.

  • Note: the length needs to be equal to the edit prompts

Trainer

• meta-learning based: MEND
• memory-based routing: SERAC

For above editing methods, pre-training of corresponding meta-networks or classifiers is required. Therefore, in EasyEdit, we provide a unified framework for pretraining the relevant network structures. Take the training MEND for example:

• Step 1 and Step 2 are the same as the example above, which involves selecting the appropriate editing model and editing method.

Step3: Provide the edit training set The currently supported and available datasets are: zsre and counterfact(Google Drive). Please place them in the "data" directory and initialize the dataset_class (ZsreDataset for zsre and CounterFactDataset for counterfact) to load the corresponding training set.

train_ds = ZsreDataset('./data/zsre_mend_train.json', config=training_hparams)
eval_ds = ZsreDataset('./data/zsre_mend_eval.json', config=training_hparams)

Step4: Combine them into a Trainer

trainer = EditTrainer(
    config=training_hparams,
    train_set=train_ds,
    val_set=eval_ds
)

Step5: Run and Edit Done! We can conduct Run and Evaluation.

trainer.run()

• Run: The CHECKPOINT will be saved to the path results_dir.
• Edit: Set the archive field in the hparams file to CHECKPOINT. EasyEdit will automatically load the corresponding pre-trained weights during the editing process(Go to edit).

Training Example

from easyeditor import EditTrainer, MENDTrainingHparams, ZsreDataset

training_hparams = MENDTrainingHparams.from_hparams('hparams/TRAINING/MEND/llama-7b.yaml')
train_ds = ZsreDataset('./data/zsre/zsre_mend_train.json', config=training_hparams)
eval_ds = ZsreDataset('./data/zsre/zsre_mend_eval.json', config=training_hparams)
trainer = EditTrainer(
    config=training_hparams,
    train_set=train_ds,
    val_set=eval_ds
)
trainer.run()

Use EasyEdit with KnowEdit

Dataset

KnowEdit is a benchmark dataset of knowledge editing for LLMs. You can easily obtain KnowEdit from HuggingFace, HuggingFace, and ModelScope.

| dataset | HuggingFace| HuggingFace | ModelScope | | :--------: | :-----------------------------------------------------------------------------------------------: | :---