SimpleMem

Efficient Lifelong Memory for LLM Agents — Text & Multimodal

Store, compress, and retrieve long-term memories with semantic lossless compression. Now with multimodal support for text, image, audio & video.

🇨🇳 中文 • 🇯🇵 日本語 • 🇰🇷 한국어 • 🇪🇸 Español • 🇫🇷 Français • 🇩🇪 Deutsch • 🇧🇷 Português 🇷🇺 Русский • 🇸🇦 العربية • 🇮🇹 Italiano • 🇻🇳 Tiếng Việt • 🇹🇷 Türkçe

🚀 Quick Start • 🌟 Overview • 📦 Installation • 🔌 MCP Server • 📊 Reproduce • 📝 Citation

🔥 News

• [05/21/2026] 📦 Unified simplemem package — one import, auto-routing! SimpleMem, Omni-SimpleMem, and EvolveMem now live in a single package. from simplemem import SimpleMem auto-selects the text or multimodal backend from the first method you call, and simplemem.optimize(...) taps EvolveMem's self-evolution loop. Install in one step with pip install -e ..
• [05/14/2026] 🧬 EvolveMem (v3.0) — Self-Evolving Memory via AutoResearch! The retrieval infrastructure itself now self-evolves through LLM-driven closed-loop diagnosis. On LoCoMo, EvolveMem outperforms the strongest baseline by +25.7% relative; on MemBench, by +18.9% relative. The system discovers entirely new retrieval dimensions not present in the original design. View EvolveMem →
• [04/02/2026] 🧠 Omni-SimpleMem (v2.0) — Multimodal Memory is Here! SimpleMem now supports text, image, audio & video memory. Achieving new SOTA on LoCoMo (F1=0.613, +47%) and Mem-Gallery (F1=0.810, +51%) over previous best. View Omni-SimpleMem →
• [02/09/2026] 🚀 Cross-Session Memory — Outperforming Claude-Mem by 64%! View Cross-Session Documentation →
• [01/20/2026] 📦 SimpleMem is now available on PyPI! Install via pip install simplemem. View Package Usage Guide →
• [01/14/2026] 🎉 SimpleMem MCP Server is LIVE! Cloud-hosted at mcp.simplemem.cloud. View MCP Documentation →
• [01/05/2026] SimpleMem paper was released on arXiv!

---

📑 Table of Contents

• 🚀 Quick Start
• 🌟 Overview
• 📦 Installation
• 🐳 Docker
• 🔌 MCP Server
• 📊 Reproduce Paper Results
• 🗺️ Roadmap
• 📝 Citation

---

🚀 Quick Start

🧠 Understanding the Basic Workflow

At a high level, SimpleMem works as a long-term memory system for LLM-based agents. The workflow consists of three simple steps:

1. Store information – Dialogues or facts are processed and converted into structured, atomic memories.
2. Index memory – Stored memories are organized using semantic embeddings and structured metadata.
3. Retrieve relevant memory – When a query is made, SimpleMem retrieves the most relevant stored information based on meaning rather than keywords.

This design allows LLM agents to maintain context, recall past information efficiently, and avoid repeatedly processing redundant history.

🎓 Basic Usage

SimpleMem ships as a single simplemem package. The default mode="auto" automatically detects which backend to use based on what you call — no manual configuration needed:

from simplemem import SimpleMem

mem = SimpleMem()  # mode="auto" — backend chosen by first call

The first method you call determines the backend:

First call	Backend selected	Why
add_dialogue()	Text (SimpleMem)	Dialogue-based API → text mode
add_text() / add_image() / add_audio() / add_video()	Omni (Omni-SimpleMem)	Multimodal API → omni mode

📝 Auto → Text (pure text input)

from simplemem import SimpleMem

mem = SimpleMem()  # auto mode

# add_dialogue() → text backend auto-selected
mem.add_dialogue(
    "Alice",
    "Bob, let's meet at Starbucks tomorrow at 2pm",
    "2025-11-15T14:30:00",
)
mem.add_dialogue(
    "Bob",
    "Sure, I'll bring the market analysis report",
    "2025-11-15T14:31:00",
)
mem.finalize()

answer = mem.ask("When and where will Alice and Bob meet?")
# → "16 November 2025 at 2:00 PM at Starbucks"

🧠 Auto → Omni (multimodal input)

from simplemem import SimpleMem

mem = SimpleMem()  # auto mode

# add_image() → omni backend auto-selected
mem.add_text(
    "User loves hiking in the Rocky Mountains.",
    tags=["session_id:D1"],
)
mem.add_image("photo.jpg", tags=["session_id:D1"])
mem.add_audio("voice_note.wav", tags=["session_id:D1"])

result = mem.query("What does the user enjoy?", top_k=5)
for item in result.items:
    print(item["summary"])

mem.close()

💡 Tip: Auto mode picks the lightest backend that fits your data. You can still use mode="text" or mode="omni" explicitly if you prefer.

---

🧬 Advanced: Optimize Retrieval Config

Tune retrieval hyperparameters offline on your own dev set, then deploy the resulting Config for inference. This is a thin wrapper around EvolveMem's self-evolution loop:

import simplemem
from simplemem import SimpleMem, load_config

# mem is a finalized SimpleMem instance with memories already built
dev_questions = [
    ("When is the meeting?", "2pm tomorrow at Starbucks"),
    ("What should Bob prepare?", "market analysis report"),
]
config = simplemem.optimize(mem, dev_questions, max_rounds=3)
config.save("my_config.json")

# Later, deploy with the optimized config
config = load_config("my_config.json")
mem = SimpleMem(config=config)

EvolveMem runs an LLM-driven Evaluate → Diagnose → Propose → Guard cycle over your dev questions, adjusting global retrieval flags (top_k, fusion mode, answer verification, reflection rounds, ...). For the full standalone version with benchmark adapters and per-category overrides, see EvolveMem/.

---

🚄 Advanced: Parallel Processing

For large-scale dialogue processing, enable parallel mode:

from simplemem import create

mem = create(
    mode="text",
    clear_db=True,
    enable_parallel_processing=True,  # ⚡ Parallel memory building
    max_parallel_workers=8,
    enable_parallel_retrieval=True,   # 🔍 Parallel query execution
    max_retrieval_workers=4
)

💡 Pro Tip: Parallel processing significantly reduces latency for batch operations!

---

🌟 Overview

SimpleMem is a unified memory stack for LLM agents, built on one principle: store semantically lossless memory at high information density, so an agent recalls more while spending far fewer tokens. The package brings together three works that share this principle but attack different parts of the problem.

📝 SimpleMem: the efficiency core (text)

Most memory systems force a bad trade-off. They either passively accumulate raw interaction history (redundant, token-hungry) or run expensive reasoning loops to filter noise (slow, costly). SimpleMem instead compresses interactions through a three-stage pipeline:

Stage	What it does
1. Semantic Structured Compression	Distills unstructured interactions into compact memory units (self-contained facts with resolved coreferences and absolute timestamps), each indexed through multiple complementary views for flexible retrieval.
2. Online Semantic Synthesis	Merges related context within a session into unified abstract representations, removing redundancy as memory is built rather than at query time.
3. Intent-Aware Retrieval Planning	Infers the search intent behind a query to decide what to retrieve and assemble a precise, compact context.

On the LoCoMo benchmark this delivers a 26.4% average F1 gain over prior systems while cutting inference-time token consumption by roughly 30x. Mechanism details (hybrid index layers, compression examples, retrieval planning): SimpleMem text memory →.

🧠 Omni-SimpleMem: multimodal memory (text, image, audio, video)

Omni-SimpleMem extends the compression-first philosophy to four modalities, built on three principles: Selective Ingestion (entropy-driven filtering per modality), Progressive Retrieval (hybrid FAISS + BM25 with pyramid token-budget expansion), and Knowledge Graph Augmentation (multi-hop cross-modal reasoning). Rather than being hand-designed, its architecture was discovered by an autonomous research pipeline that ran around 50 experiments across two benchmarks, diagnosing failure modes, proposing architectural changes, and even repairing data-pipeline bugs with no human in the inner loop. Tellingly, the bug fixes and architectural changes each contributed more than all hyperparameter tuning combined, taking the system from a naive baseline to state-of-the-art on both LoCoMo and Mem-Gallery. Full docs: Omni-SimpleMem →.

🧬 EvolveMem: self-evolving retrieval

EvolveMem closes a blind spot shared by almost every memory system: the stored content evolves, but the retrieval machinery (scoring functions, fusion strategies, answer-generation policies) stays frozen after deployment. EvolveMem runs a closed-loop AutoResearch process (Evaluate → Diagnose → Propose → Guard → Repeat) in which an LLM diagnoses per-question failures and proposes configuration changes, guarded by automatic rollback on regression and exploration incentives during stagnation. It discovers new retrieval dimensions (query decomposition, entity-swap, answer verification) not in the original design, improves LoCoMo by 25.7% relative over the strongest baseline, and its evolved configurations transfer positively across benchmarks. Full docs: EvolveMem →.

How they fit together

from simplemem import SimpleMem gives you the text core with automatic routing to the multimodal backend, and simplemem.optimize(...) taps EvolveMem to tune retrieval for your own data. One package, one mental model: compress losslessly, retrieve by intent, and let the system keep improving itself.

---

📦 Installation

📝 Notes for First-Time Users

• Ensure you are using Python 3.10+ in your active environment, not just installed globally.
• An OpenAI-compatible API key must be configured before running any memory construction or retrieval, otherwise initialization may fail.
• When using non-OpenAI providers (e.g., Qwen or Azure OpenAI), verify both the model name and OPENAI_BASE_URL in config.py.
• For large dialogue datasets, enabling parallel processing can significantly reduce memory construction time.

📋 Requirements

• 🐍 Python 3.10+
• 🔑 OpenAI-compatible API (OpenAI, Qwen, Azure OpenAI, etc.)

🛠️ Setup

# 📥 Clone repository
git clone https://github.com/aiming-lab/SimpleMem.git
cd SimpleMem

# 📦 Install dependencies (pinned versions)
pip install -r requirements.txt

# — OR — install as an editable package
pip install -e .                  # default: text + multimodal + evolver
pip install -e ".[server]"        # + MCP / HTTP server (mcp, fastapi, ...)
pip install -e ".[all]"           # everything, including dev tools

# ⚙️ Configure API settings
cp config.py.example config.py
# Edit config.py with your API key and preferences

⚙️ Configuration Example

# config.py
OPENAI_API_KEY = "your-api-key"
OPENAI_BASE_URL = None  # or custom endpoint for Qwen/Azure

LLM_MODEL = "gpt-4.1-mini"
EMBEDDING_MODEL = "Qwen/Qwen3-Embedding-0.6B"  # State-of-the-art retrieval

Because SimpleMem talks to any OpenAI-compatible endpoint, you can point it at a managed provider such as Atlas Cloud by setting OPENAI_BASE_URL:

# config.py — using Atlas Cloud as an OpenAI-compatible backend
OPENAI_API_KEY = "your-atlascloud-api-key"
OPENAI_BASE_URL = "https://api.atlascloud.ai/v1"

LLM_MODEL = "deepseek-ai/deepseek-v4-pro"   # a reasoning model — keep max tokens generous
EMBEDDING_MODEL = "Qwen/Qwen3-Embedding-0.6B"

deepseek-ai/deepseek-v4-pro is a reasoning model, so leave enough output budget; if responses come back empty, raise the token limit (>= 512). Any other OpenAI-compatible chat model on Atlas Cloud (e.g. Qwen/Qwen3-Next-80B-A3B-Instruct, zai-org/glm-5, moonshotai/kimi-k2.6) works the same way.

---

🐳 Run with Docker

The MCP Server can be run in Docker for a consistent, isolated environment. Data (LanceDB and user DB) is persisted in a host volume.

Prerequisites

• Docker and Docker Compose

Quick run

# From the repository root
docker compose up -d

• Web UI: http://localhost:8000/
• REST API: http://localhost:8000/api/
• MCP (SSE): http://localhost:8000/mcp/sse?token=<TOKEN>

Data is stored in ./data on the host (created automatically).

Custom configuration

1. Copy the environment template and edit it:

   cp .env.example .env
   # Edit .env: set JWT_SECRET_KEY, ENCRYPTION_KEY, LLM_PROVIDER, model URLs, etc.
   

2. Run with the env file:

   docker compose --env-file .env up -d
   

Using Ollama on the host

When LLM_PROVIDER=ollama and Ollama runs on your machine (not in Docker), set in .env:

LLM_PROVIDER=ollama
OLLAMA_BASE_URL=http://host.docker.internal:11434/v1

On Linux, host.docker.internal is enabled automatically via the Compose file.

Using Requesty

Requesty is an OpenAI-compatible router. To use it, set in .env:

LLM_PROVIDER=requesty
REQUESTY_BASE_URL=https://router.requesty.ai/v1

Provide your Requesty API key (from app.requesty.ai/api-keys) when registering. Models use the same provider/model naming as OpenRouter (e.g. openai/gpt-4.1-mini). See the Requesty docs for available models.

Useful commands

docker compose logs -f simplemem   # Follow logs
docker compose down                 # Stop and remove containers

📖 For self-hosting the MCP server (Docker or bare metal), see MCP Documentation.

---

🔌 MCP Server (text memory)

SimpleMem is available as a cloud-hosted memory service via the Model Context Protocol (MCP), enabling seamless integration with AI assistants like Claude Desktop, Cursor, and other MCP-compatible clients.

🌐 Cloud Service: mcp.simplemem.cloud — or self-host the MCP server locally using Docker.

Key Features

Feature	Description
Streamable HTTP	MCP 2025-03-26 protocol with JSON-RPC 2.0
Multi-tenant Isolation	Per-user data tables with token authentication
Hybrid Retrieval	Semantic search + keyword matching + metadata filtering
Production Optimized	Faster response times with OpenRouter integration

Quick Configuration

{
  "mcpServers": {
    "simplemem": {
      "url": "https://mcp.simplemem.cloud/mcp",
      "headers": {
        "Authorization": "Bearer YOUR_TOKEN"
      }
    }
  }
}

📖 For detailed setup instructions and self-hosting guide, see MCP Documentation

---

📊 Reproduce Paper Results

Reproduce the LoCoMo / MemBench / Mem-Gallery numbers from the papers. Each pillar has its own benchmark runner in its own directory. Install the benchmark extras first: pip install -e ".[benchmark]".

📝 SimpleMem (text) — LoCoMo

Run from the repository root:

python test_locomo10.py                       # full LoCoMo benchmark
python test_locomo10.py --num-samples 5       # quick subset
python test_locomo10.py --result-file my_results.json

🧬 EvolveMem — self-evolution + LoCoMo / MemBench

Run from the EvolveMem/ directory (see EvolveMem/README.md):

cd EvolveMem
python run_evolution.py --data data/locomo10.json --max-rounds 7
python run_benchmark.py locomo --sample 0 --initial weak --max-rounds 3
python run_benchmark.py membench --agent FirstAgent --max-rounds 3

🧠 Omni-SimpleMem — LoCoMo / Mem-Gallery

Run from the OmniSimpleMem/ directory (see OmniSimpleMem/README.md):

cd OmniSimpleMem
python benchmarks/locomo/run_locomo.py --data-path /path/to/locomo10.json --model gpt-4o

---

🗺️ Roadmap

Current capability by integration channel:

Capability	Python (pip install)	MCP server (Claude Desktop, Cursor, ...)
Text memory	✅	✅
Multimodal (image / audio / video)	✅	⬜ planned
optimize() self-evolving retrieval	✅	⬜ planned

Planned work to close the gap (the MCP server is a standalone multi-tenant text service; these are real features, not doc fixes):

• Multimodal over MCP. Add memory_add_image / memory_add_audio / memory_add_video tools. Needs a file-upload path (base64 or URL, since MCP cannot pass local file paths), a multi-tenant adaptation of the Omni-SimpleMem storage backend, and server-side vision/audio model access.
• EvolveMem over MCP. Expose optimize() as an MCP tool. More tractable than multimodal (text in, JSON config out, no file transport), but the MCP retriever currently honors only semantic_top_k / keyword_top_k of the ~10 dimensions EvolveMem evolves. Requires extending the MCP retriever to support the remaining knobs (structured top_k, fusion mode/weights, entity swap, query decomposition, answer verification), an adapter to run the evolution loop over a tenant's stored memories, per-tenant config persistence, and async execution (the loop is LLM-heavy and would time out a synchronous request).
• Docker inherits both automatically once the MCP server supports them (add multimodal deps to the image and an Omni storage volume).

For full multimodal and self-evolving retrieval today, use the Python API (see Quick Start).

---

📝 Citation

If you use SimpleMem in your research, please cite:

@article{simplemem2026,
  title={SimpleMem: Efficient Lifelong Memory for LLM Agents},
  author={Liu, Jiaqi and Su, Yaofeng and Xia, Peng and Zhou, Yiyang and Han, Siwei and  Zheng, Zeyu and Xie, Cihang and Ding, Mingyu and Yao, Huaxiu},
  journal={arXiv preprint arXiv:2601.02553},
  year={2026},
  url={https://arxiv.org/abs/2601.02553}
}

@article{evolvemem2026,
  title={EvolveMem: Self-Evolving Memory Architecture via AutoResearch for LLM Agents},
  author={Liu, Jiaqi and Ye, Xinyu and Xia, Peng and Zheng, Zeyu and Xie, Cihang and Ding, Mingyu and Yao, Huaxiu},
  journal={arXiv preprint arXiv:2605.13941},
  year={2026},
  url={https://arxiv.org/abs/2605.13941}
}

@article{omnisimplemem2026,
  title   = {Omni-SimpleMem: Autoresearch-Guided Discovery of Lifelong Multimodal Agent Memory},
  author  = {Liu, Jiaqi and Ling, Zipeng and Qiu, Shi and Liu, Yanqing and Han, Siwei and Xia, Peng and Tu, Haoqin and Zheng, Zeyu and Xie, Cihang and Fleming, Charles and Ding, Mingyu and Yao, Huaxiu},
  journal = {arXiv preprint arXiv:2604.01007},
  year    = {2026},
}

---

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

---

🙏 Acknowledgments

We would like to thank the following projects and teams:

• 🔍 Embedding Model: Qwen3-Embedding - State-of-the-art retrieval performance
• 🗄️ Vector Database: LanceDB - High-performance columnar storage
• 📊 Benchmark: LoCoMo - Long-context memory evaluation framework