mempalace

Agents get lighter ways in: a 3-tool MCP server with Palace Query Language, shared-brain rules for host:harness:project identities, and optional native exact-vector search. New installs follow XDG, and storage fixes close a silent write-loss path, a macOS integrity refusal and several hangs.

Upgrade notes

• mempalace rules no longer accepts --agent. Pass --host, --harness and --project, then re-render any installed shared-brain block. (#2508)
• get_collection() refuses names MemPalace never reads. Any name other than the configured drawers collection or mempalace_closets raises CollectionNameMismatchError; maintenance scripts can pass _skip_name_check=True. (#2446)
• The MCP event listing returns the newest events first when no cursor is given. mempalace_event_list and palace_coordinate now default to newest-first unless since_event_id or order is passed; mempalace logstream list and Logstream.list_events() still default to oldest-first. (#2497)
• New installs keep config and palace under ~/.config/mempalace (or $XDG_CONFIG_HOME/mempalace). Existing ~/.mempalace installs are unchanged. (#148)

Performance & Architecture (Dual-Track Rust Engine)

• Optional native Rust exact-vector acceleration (rust_exact). Reads the existing SQLite format into a contiguous float buffer, uses Rayon for large scans, and releases the Python GIL. Local and external writes invalidate the native cache. Loading is collection-scoped and decodes blobs safely; serial and parallel scans preserve deterministic tie ordering. Complex filters and absent native extensions fall back to Python.

• Standalone native vector CLI and optional wheels. GitHub releases include native wheels and platform executables. The CLI accepts JSON embedding vectors via --vector or stdin; it does not embed text or require Python. Platform runtime libraries may still be required.

• NumPy top-k optimization. Cache vector norms and partition distances before sorting the selected results, preserving row-order ties at the cutoff.

• Read-only search operations stop contending with palace writer leases. searcher.py and _open_search_collection now explicitly open collections with read_only=True. Read-only queries connect in SQLite WAL reader mode without attempting schema initialization or writer lease locks, allowing CLI and agent searches to run concurrently with an active background MCP server or writer process without raising MineAlreadyRunning.

Refactors

• MCP server is a package, not an 8 000-line module. mempalace.mcp_server stays the public import path (TOOLS, handle_request, main, every tool_* handler, and the process-global names tests monkeypatch). Implementation lives in domain files under mempalace/mcp_server/ so PRs can target drawers, KG, coordination, protocol, or HTTP without colliding on one god-file. Fragments exec into the package namespace — a physical split, not a behaviour change.
• CLI is a package, not a 4 000-line module. mempalace.cli stays the public import path (main, every cmd_* handler, and the names tests monkeypatch). Implementation lives in domain files under mempalace/cli/ so PRs can target init, mine, search, repair, logstream, or argparse without colliding on one god-file. Fragments exec into the package namespace — a physical split, not a behaviour change.
• MCP server tests are split by domain. tests/test_mcp_server.py is now tests/mcp/ (protocol, read, search, write, diary, KG, guards, stale-library) so community MCP PRs do not collide on one 8 600-line test file. Shared helpers live in tests/_mcp_server_helpers.py.
• Searcher is a package, not a 2 200-line module. mempalace.searcher stays the public import path (search, search_memories, SearchError, and the helpers tests monkeypatch). Implementation lives under mempalace/searcher/ (ranking, filters, sqlite BM25, candidates, q

v3.9.0 — one hub for every agent

One hub now serves every agent. CLI search reuses the live palace index instead of loading another multi-gigabyte HNSW copy, concurrent searches stop blocking each other, and skill-first coordination plus opt-in release awareness make shared-brain fleets easier to operate safely.

Shared hub and search

• mempalace search reuses a healthy live Hub. On the reported 4.99-million-vector palace, two direct CLI searches held about 18.6 GB of private RSS. Forwardable searches now reuse the Hub collection while preserving filters, ranking, output, authentication, configuration negotiation, and safe fallback behavior. (#2379)
• Hub reads run concurrently. Searches no longer serialize behind one exclusive HTTP dispatch lock; reads share a writer-preferring lock while mutations remain exclusive. (#2398)
• Search results are clearer and faster to enrich. Public similarity is the raw vector score, closet boost remains in effective ranking, source lookups are indexed, and duplicate enriched passages are removed. (#2399, #2143)
• A private-palace benchmark is included for product search evaluation without exposing private queries or corpus text. (#2400)

Agent coordination and operations

• Skill-first setup and task handoff. The guided setup skill can bootstrap private or shared-brain installations, render canonical rules, assign stable fleet identities, and validate logstream readiness. Task creation now records an immutable base commit and complete definition of done before controlled launch. (#2374, #2361)
• Release awareness is explicit and opt-in. Agents can surface cached update state and prepare an exact upgrade plan without blocking, auto-installing, or silently enabling network checks. (#2376)
• Logstream gains topic routing and reverse pagination, making focused coordination and newest-first inspection practical without weakening append-only cursor guarantees. (#2397)

Correctness and resource safety

• Unreadable configuration is never overwritten. Invalid files are preserved, writes are atomic where possible, symlinks remain intact, and UTF-8 project configuration is handled consistently. (#2357, #2353, #2369)
• Drawer mutations purge stale closet entries, single conversation files mine correctly through a running Hub, and absent SQLite databases no longer report a false clean integrity verdict. (#2355, #2282, #2291)
• Qdrant bounded reads stop scanning whole collections, and EmbeddingGemma internal batches can be reduced for long drawers on memory-constrained hardware. (#2384, #2383)

A big thank-you to everyone who contributed. This release includes 29 merged PRs and seven first-time contributors.

Install: pip install -U mempalace==3.9.0

What's Changed

• docs: refresh the release entry-point check and plugin tool counts by @igorls in https://github.com/MemPalace/mempalace/pull/2348
• docs(shared-brain): teach the from-zero guide about logstream watch by @igorls in https://github.com/MemPalace/mempalace/pull/2349
• docs(coordination): wake for replies, and keep announcements silent by @igorls in https://github.com/MemPalace/mempalace/pull/2352
• fix(mcp): report an absent integrity verdict instead of a clean one (#2290) by @mvalentsev in https://github.com/MemPalace/mempalace/pull/2291
• docs: add branching model to CONTRIBUTING.md by @jjcav84 in https://github.com/MemPalace/mempalace/pull/2246
• fix(sqlite_exact): cross-handle cache invalidation and filtered id lookups by @igorls in https://github.com/MemPalace/mempalace/pull/2362
• fix(search): return distinct passages after closet enrichment by @fatkobra in https://github.com/MemPalace/mempalace/pull/2143
• fix(graph): report complete palace graph stats by @MichaelWave369 in https://github.com/MemPalace/mempalace/pull/2288
• feat(cli): mempalace rules — render the canonical shared-brain rules block by @igorls in https://github.com/MemPalace/mempalace/pull/2361
• fix(mcp):

v3.8.0 — fast large palaces, lean MCP proxies, wakeable agents

Large palaces get fast and stay small in 3.8.0. Chroma metadata tools no longer cold-load HNSW, sqlite_exact stops scanning and hydrating the whole palace for common reads, long-running Chroma servers stop rebuilding their client on their own writes, and proxied MCP sessions no longer import a storage stack they never use. Agents also gain a persistent logstream watch primitive so coordination can wake a harness instead of relying on repeated five-minute waits.

Performance

• Chroma metadata reads skip the vector index. On a 1.7 GB / 165k-drawer palace, wing-scoped list_drawers(limit=20) fell from 2.31 s and 1148 MB peak RSS to 0.01 s and 86 MB. list_drawers and tunnel reads now query Chroma's SQLite metadata directly, push filters into SQL, and hydrate only the requested previews. (#2314)
• sqlite_exact common reads no longer walk the whole palace. On a live 167k-drawer / 166k-closet palace, mempalace_status fell from 6997 ms to 1045 ms and warm mempalace_search from 6364 ms to 210–1600 ms. Vector ranking is batched and cached, top-k documents are hydrated afterward, equality filters paginate in SQL, taxonomy results are cached, and wing / room / hall gain indexed generated columns without rewriting the 1.6 GB embedding table. (#2308, #2311, #2313)
• Long-running Chroma servers stay flat instead of reloading HNSW on their own writes. Across eight searches, measured RSS changed from 951 MB → 3522 MB before to 940 MB → 965 MB after. Replaced clients are closed, while genuine external database changes still trigger a rebuild. (#2307)
• Hub-proxied MCP sessions are thin. A plain stdio proxy now imports only the forwarding path and lazily loads the full server only if local fallback is needed. A proxied session runs at roughly 17 MB, with import weight falling from about 80 MB to 24 MB—avoiding roughly 3.9 GB across a 50-agent fleet. (#2312)

Agent coordination

• mempalace logstream watch provides a durable wake-up loop. It owns cursor advancement, supports repeated type filters, agent self-exclusion, persistent state files, idle exit, and NDJSON follow mode. Cursorless first runs start at the live tip; restarts may replay an event but do not silently miss one. (#2315)
• The monitoring protocol now documents the safe resume cursor. Cross-replica events are appended in arrival order, so since_event_id—not a wall-clock timestamp—is the resumable high-water mark. Tool descriptions and the shared coordination protocol now put that rule where agents use it. (#2315)

Integrity and operations

• sync --apply no longer deletes drawers whose source file it could not reach. One Path.exists() separated keep from remove, so an unreadable directory, a path component that is a regular file, and a volume that is not mounted all answered the same as a deletion — a project mined from a mounted volume could lose every drawer to one sync --apply while the volume was away, and come back empty. Removal now asks for corroboration rather than a probe: a file that is not at its path is removable only while the palace can still see a source of its own, a regular file, in that same directory. Everything uncorroborated lands in a new unresolved bucket, never pruned and reported the way removals are. (#2320)
• EmbeddingGemma cannot silently store NaN or zero vectors through CoreML auto-selection on Apple Silicon. Auto mode avoids the affected provider/model pair, and every vector is checked for a finite, non-zero norm before use. CUDA, DirectML, and other models are unchanged. (#2283)
• FTS5 auto-heal verifies and repairs the content table before rebuilding the index. It uses Chroma's metadata copy as authority where possible and declines to guess when integrity cannot be established. (#2279)
• Mesh status is consistent across transports. The hub publishes a token-free, last-k

v3.7.1 — mine completeness, reconnect integrity, clean shutdown

Post-3.7.0 integrity patch: ingest no longer hangs on non-regular files, incomplete mines can be retried instead of permanently skipped, chromadb reconnect no longer rewinds the HNSW index, and MCP releases the writer lease on SIGTERM/SIGHUP.

Bug Fixes

• Ingest commands no longer hang on a named pipe. os.walk and glob list a FIFO as an ordinary filename and MemPalace decides what to read from the suffix, so a pipe called notes.md in a mined directory wedged mine in the kernel forever: opening a FIFO for reading waits for a writer that never arrives, and the S_ISREG refusal written on the next line could never run. mine --mode convos, sweep, init, compress and split blocked the same way through their own readers. The four affected opens now pass O_NONBLOCK, which makes the existing type check reachable — a pipe is refused on its mode, with or without a live writer — and the discovery walks drop non-regular entries with a SKIP: <name> (not a regular file) line, so the readers that use a plain open() never see one. Regular files read back byte-identical; the one case where the flag is not inert, a reader breaking a write lease, re-checks the file type and retries without it rather than dropping the file. mine --mode extract was already immune through its zero-size gate. (#2221)
• Project re-mine no longer silently skips a partial or interrupted file. Four related gaps in process_file: (1) multi-batch upserts now stamp every drawer with chunk_total so file_already_mined can tell "N of N committed" from "crashed after batch 1"; (2) source_mtime comes from the same fstat as the content read, so an append between read and a later re-stat cannot permanently hide the new tail; (3) a failed stale-drawer purge aborts the file instead of half-overwriting; (4) closets are purged even when the re-mine ends with zero drawers. A mid-file upsert failure also deletes the partial drawers and closets for that source before re-raising, so the next mine retries instead of treating the incomplete set as complete. (#2088, #2122, #2151)
• Conversation mine completeness matches the project path. Convo drawers now stamp chunk_total; a mid-batch upsert failure deletes that source's partial drawers before re-raising; prefetch_mined_set omits incomplete groups so the bulk "already filed" skip cannot permanently strand missing exchanges from an interrupted transcript mine. (#2183)
• Stale chromadb System cache is cleared on palace reconnect. After a peer or rebuild changes chroma.sqlite3 on disk, both mcp_server._get_client and ChromaBackend._client drop chromadb's path-keyed SharedSystemClient cache before reopening — otherwise the stale in-memory HNSW segment is reused and can persist an outdated index over the peer's writes (index count going backwards). (#2002, #2028, #2026, #2032)
• MCP releases the palace writer lease on SIGTERM/SIGHUP. The lease was only released via atexit, which CPython skips on those signals' default disposition. SSH disconnect (SIGHUP) and container/systemd stop (SIGTERM) therefore left mine_palace_*.lock naming a dead PID until a contender's liveness check reclaimed it. main() now installs handlers that exit through sys.exit, so the existing atexit release path runs. (#2205)

A big thank-you to everyone who contributed to this release. Contributors and first-time contributors are credited below.

What's Changed

• fix(tests): harden hybrid search against empty Windows Chroma reads by @igorls in https://github.com/MemPalace/mempalace/pull/2220
• docs(changelog): lead 3.7.0 with agent logstream + tighten notes by @igorls in https://github.com/MemPalace/mempalace/pull/2219
• fix: 3.7.1 critical patch — hang, re-mine honesty, stale Chroma cache by @igorls in https://github.com/MemPalace/mempalace/pull/2228
• fix(mcp_server): reset chromadb System cache on staleness reconnect (#20