需先登录 Hugging Face 并下载预训练模型到 trained_d2l 才能运行示例；Python API 只支持非批处理输入，批处理需改用 hypernet.py；示例依赖 data/sakana_wiki.txt 等本地数据文件。
curl -LsSf https://astral.sh/uv/install.sh | sh
./install.sh
uv run huggingface-cli login
uv run huggingface-cli download SakanaAI/doc-to-lora --local-dir trained_d2l --include "*/" 硬件要求按 README 说明
README 未说明显存或 GPU 型号要求；示例代码使用 model.device 与 torch.load，未提及 CPU 运行方式。
用超网络把文档内容即时内化成 LoRA，让 LLM 记住事实信息
用超网络直接生成 LoRA 权重，让大模型把事实性知识记进参数里，不用微调也能更新模型记忆。
核心能力
- 用超网络从文档生成 LoRA 权重并注入模型
- 调用 internalize 后生成结果受文档内容影响
- 提供 reset 接口清除已内化的信息
- 附带交互 Demo 与 NIAH 等实验脚本
- 提供预训练 checkpoint 与 Python API
容易踩的坑
内容由模型根据 README 生成，可能不完整或不准确，以官方文档为准。看原文 README
暂无 release 数据（可能没有发版，或尚未同步）。
用户点评
登录用户的真实使用记录，可按版本和硬件筛选共 0 条
还没有人点评。第一个说出真实体验的人最有价值。
🛠️ Installation
curl -LsSf https://astral.sh/uv/install.sh | sh
./install.sh
🤗 Pre-Trained Models
uv run huggingface-cli login
uv run huggingface-cli download SakanaAI/doc-to-lora --local-dir trained_d2l --include "*/"
🚀 Python API Usage
# caveat: this interface only supports non-batched inputs
# for batched inference please see `src/ctx_to_lora/modeling/hypernet.py`
import torch
from ctx_to_lora.model_loading import get_tokenizer
from ctx_to_lora.modeling.hypernet import ModulatedPretrainedModel
# model loading
checkpoint_path = "trained_d2l/gemma_demo/checkpoint-80000/pytorch_model.bin"
state_dict = torch.load(checkpoint_path, weights_only=False)
model = ModulatedPretrainedModel.from_state_dict(
state_dict, train=False, use_sequence_packing=False
)
model.reset()
tokenizer = get_tokenizer(model.base_model.name_or_path)
# prepare data
doc = open("data/sakana_wiki.txt", "r").read()
chat = [{"role": "user", "content": "Tell me about Sakana AI."}]
chat_ids = tokenizer.apply_chat_template(
chat,
add_special_tokens=False,
return_attention_mask=False,
add_generation_prompt=True,
return_tensors="pt",
).to(model.device)
# calls after internalization will be influenced by internalized info
model.internalize(doc)
outputs = model.generate(input_ids=chat_ids, max_new_tokens=512)
print(tokenizer.decode(outputs[0]))
# remove internalized info
# model.reset()
# without internalized info, the model will halucinate
# outputs = model.generate(input_ids=chat_ids, max_new_tokens=512)
# print(tokenizer.decode(outputs[0]))
🎮 Interactive Demo
uv run demo/app.py
🧪 Experimental Scripts
To run any of the following scripts, use uv run $PATH_TO_SCRIPT from the root of this project.
| Experiment | Data prep | Training | Evaluation | Notes |
|---|---|---|---|---|
| Main experiment | scripts/main_exp/0-download_data.sh | scripts/main_exp/1-train.sh | scripts/main_exp/eval/*.sh | Downloading data is fastest; regenerate only if you need fresh synthetic data. Evaluation scripts reproduce the main paper metrics. |
| NIAH | scripts/niah/0-gen_data.sh | scripts/niah/1-train.sh | scripts/niah/2-eval.sh | Run the scripts in order; data generation only needs to happen once |
🔬 Self-Generated Data Viewer
After downloading/generating the data, we can see samples of the data using this script.
uv run webui/self_gen_viewer.py
See more info at webui/SELF_GEN_VIEWER.md.
📚 Citation
@inproceedings{charakorn2026doctolora,
title ={Doc-to-Lo{RA}: Learning to Instantly Internalize Contexts},
author ={Rujikorn Charakorn and Edoardo Cetin and Shinnosuke Uesaka and Robert Tjarko Lange},
booktitle ={Forty-third International Conference on Machine Learning},
year ={2026},
url ={https://openreview.net/forum?id=iW1oBBO72S}
}