llm-wiki-compiler

atomicstrataMIT活跃维护

把零散原始资料编译成互相链接的 Markdown 知识库,基于 Karpathy 的 LLM Wiki 思路,适合给 AI Agent 提供结构化上下文。

The knowledge compiler. Raw sources in, interlinked wiki out. Inspired by Karpathy's LLM Wiki pattern.

v1.5.0发布于 2026-10-19 20:15 UTC2,250star +6 / 7 天239fork同步于 2026-11-10 17:46 UTC

怎么上手

由 README 提炼,执行前请确认
npm install -g llm-wiki-compiler
export ANTHROPIC_API_KEY=sk-...
llmwiki quickstart ./notes.md
硬件要求按 README 说明

README 未说明

下载:源码 zip·GitHub Releases(安装包)本站不镜像安装包

项目说明

由 README 自动翻译提炼

把原始资料编译成可引用、可审阅的互联 Markdown 知识库,供人和 Agent 复用

把零散原始资料编译成互相链接的 Markdown 知识库,基于 Karpathy 的 LLM Wiki 思路,适合给 AI Agent 提供结构化上下文。

核心能力

  • 两阶段 LLM 流水线先抽取概念再生成类型化页面
  • 段落与论断标注来源文件及行号,lint 校验引用
  • 语义检索、BM25 重排与 wikilink 图扩展组成证据包
  • 用 profile.json 声明实体、关系、生命周期与审核门
  • 支持导出导入 Open Knowledge Format 等格式

容易踩的坑

需要自备 LLM 提供方 API Key,否则无法编译;模板安装只适用于新建或空项目;OKF 导入默认进入审核候选而非直接生效

内容由模型根据 README 生成,可能不完整或不准确,以官方文档为准。看原文 README

最近发版

5 个版本v1.5.0
  1. v1.5.01.5.0最新正式版2026-10-19 · 2026-10-19 20:15 UTC

    llmwiki 1.5.0 — Wiki behavior separated from storage

    npm install --global llm-wiki-compiler now installs 1.5.0 (npm latest). The two supporting packages are pinned to the same version and install with it. This release ships the same code as 1.5.0-rc.1; only the version and release documentation changed since the candidate.

    Highlights

    • Same wiki, cleaner foundation. Compilation, retrieval, review, artifacts and project state now reach the disk through focused storage boundaries, and today's local-disk layout is their first implementation. Locking, atomic writes, path confinement, symlink handling and recovery stay in the local implementation. Your wiki keeps the same layout, file formats, defaults, and CLI, SDK and MCP entry points, with no migration or storage setting. This is groundwork; it does not add hosted storage, a remote backend, role-based permissions or a public storage-plugin API.
    • Converse gateways. LLMWIKI_PROVIDER=converse connects to a bearer-authenticated endpoint speaking the Bedrock Converse protocols, with streaming and structured extraction. Its AWS protocol packages are optional peers, loaded only when selected.
    • Pages that fit the source. llmwiki compile --no-pages-limit (also scoped refresh, SDK noPagesLimit: true) drops the requested 3–8 concepts per source.
    • A viewer that keeps up. llmwiki view refreshes its snapshot after a compile without a restart and reads LLMWIKI_VIEW_PORT when --port is not given.

    Fixes

    • Wikilink markup in an extracted concept name no longer creates a duplicate page.
    • GPT-6 Luna models work with the OpenAI provider.
    • A named pipe in sources/ no longer hangs freshness checks.
    • MCP wiki tools and the whole stdio server keep progress output off the JSON-RPC stream.

    Upgrade notes

    • No migration is needed. Back up a wiki you care about before upgrading, as for any release.
    • Prompt version moves from v6 to v7. Upgrading alone does not recompile unchanged sources; a source that needs extraction again is extracted afresh.
    • Converse is opt-in: install @aws-sdk/client-bedrock-runtime and @smithy/node-http-handler only if you select it.
    • The MCP Registry entry io.github.atomicstrata/llmwiki is published for 1.5.0 after the npm release.

    Thank you

    • @LorenzoGentile — Converse gateway provider (#282), --no-pages-limit extraction (#277, merged as #285), wikilink concept-title normalization (#278) and GPT-6 Luna request parameters (#279).
    • @Erick0412-dev — viewer snapshot refresh (#275) and LLMWIKI_VIEW_PORT (#274).
    • @vishalhabib99 — quiet MCP wiki tools over stdio (#270).

    Credit includes original implementations completed or extended during maintainer review.

    Full details: CHANGELOG · Upgrade notes · Full diff: https://github.com/atomicstrata/llm-wiki-compiler/compare/v1.4.2...v1.5.0

  2. v1.5.0-rc.11.5.0-rc.1pre-release2026-10-15 · 2026-10-15 17:36 UTC
    查看更新说明

    A cleaner foundation for your existing wiki

    1.5.0-rc.1 is the first release candidate for llmwiki 1.5. It is available on npm's next channel for early testing; 1.4.2 remains the stable release on latest.

    The main change separates wiki behavior from local filesystem operations. Compilation, retrieval, review and project state work through focused storage interfaces, with the existing disk layout as their implementation. Existing wikis need no migration or new setting. This creates a foundation for alternative storage without shipping a remote backend, hosted service, role-based permissions or a public storage-plugin API yet.

    Also in this candidate

    • Converse gateways: use bearer-authenticated endpoints speaking the Converse protocols.
    • Source-driven extraction: compile --no-pages-limit removes the requested concept count so the model can follow the substance of a source. Generated pages still need review.
    • Viewer improvements: refresh wiki snapshots without restarting the viewer and configure its port with LLMWIKI_VIEW_PORT.
    • Correctness fixes: normalize extracted wikilink titles, support GPT-6 Luna request parameters, keep MCP stdout clean, and avoid blocking freshness checks on nonregular source files.

    Try it

    npm install --global [email protected]
    

    The matching core and local-workflow packages install automatically. Back up important wikis before trying a preview, and exercise your existing CLI, SDK or MCP workflows. Please report regressions before the stable release.

    Every newly generated page records prompt version v7, including default runs. Upgrading alone does not recompile unchanged sources; extraction snapshots from the earlier prompt version are not reused when extraction is needed again. The MCP Registry entry is not updated for this candidate.

    Thank you

    Thanks to @LorenzoGentile for the Converse provider (#282), source-driven extraction (#277, integrated as #285), wikilink normalization (#278), and GPT-6 Luna support (#279); @Erick0412-dev for viewer refresh (#275) and port configuration (#274); and @vishalhabib99 for quiet MCP wiki tools (#270). Their contributions make this release more useful and reliable across everyday workflows.

    See the full changelog, upgrade notes, and architecture.

  3. v1.4.21.4.22026-10-03 · 2026-10-03 00:34 UTC
    查看更新说明

    llmwiki 1.4.2 fixes llmwiki compile with the codex-agent provider.

    • In 1.4.1, concept extraction through codex-agent failed with invalid_json_schema, because Codex accepts only strict output schemas. The provider now sends Codex a strict copy of each tool schema, and optional fields keep their meaning.
    • Thanks to @frankstallone for the report and the diagnosis.

    No other changes. Install [email protected]; the supporting packages use matching versions.

  4. v1.4.11.4.12026-10-02 · 2026-10-02 04:14 UTC
    查看更新说明

    llmwiki 1.4.1 makes it easier for agents to find the compiler and use an existing wiki as evidence.

    • A bundled Agent Skill explains when to use llmwiki and guides agents to inspect the wiki and retrieve cited context before generating or writing.
    • Searchable npm metadata, a versioned MCP Registry manifest, and clearer MCP descriptions improve discovery and tool selection.
    • Updated setup documentation explains installation, provider requirements, and review boundaries.
    • The README highlights 1.4.0 benefits; the changelog consolidates its release history and contributor credits; the architecture guide reflects the released implementation.

    Existing MCP tool names, inputs, and execution behavior are unchanged. Install [email protected]; the supporting packages use matching versions.

  5. v1.4.01.4.02026-10-02 · 2026-10-02 00:35 UTC
    查看更新说明

    llmwiki 1.4.0 is available on npm latest. Continue installing the standard package; its matching core and local-workflow dependencies are installed automatically.

    npm install --global [email protected]
    llmwiki --version
    

    Highlights

    • Review generated answers before publication and approve candidate batches with shared finalization.
    • Reuse source extraction and prompt prefixes, with grounded query reporting and embedding fallback enabled by default.
    • Keep personal/project Claude coding instructions out of generated wiki pages and answers while retaining existing Claude login.
    • Clarify target-first wiki-link aliases and ship an optional read-only source-review cockpit.
    • Split the SDK implementation into matching compiler, core, and local-workflow packages while retaining the standard entry point.

    Upgrade notes

    Experimental artifact and workflow types can require consumer updates. Newly generated concept pages record prompt version v6; upgrading alone does not regenerate unchanged sources. Use compile --review to inspect source-triggered updates before approval.

    Read the SDK upgrade notes and full changelog.

    Validation and limits

    The stable source passed hosted build/tests, type checks, codebase health, release preflight, and registry/fresh-install verification. Its runtime is unchanged from RC3, which passed live synthetic-fixture acceptance with an existing Claude login and Voyage embeddings: compile, review approval, cited answers, and semantic retrieval without fallback. An actual 1.3.0 fixture upgraded to RC3 preserved unchanged concept pages, accepted a source update with v6 provenance, and answered old and new facts with resolved citations.

    The existing source-line citation lint rule can warn on query answers that use valid wiki-page citations. Native Windows and broad provider/platform coverage are outside the live acceptance scope.

用户点评

登录用户的真实使用记录,可按版本和硬件筛选
共 0 条

    还没有人点评。第一个说出真实体验的人最有价值。

    README 原文

    未翻译

    llmwiki

    CI npm docs license

    New in 1.5 — Wiki behavior separated from storage.

    1.5.0 is the stable release on npm latest: npm install --global llm-wiki-compiler.

    • Same wiki, cleaner foundation. Compilation, retrieval and review now describe what they need from storage instead of touching files directly. Your existing wiki keeps its layout and works as before, with no migration.
    • Converse gateways. Use a bearer-authenticated Converse endpoint as your provider.
    • Pages that fit the source. llmwiki compile --no-pages-limit drops the fixed concept count.
    • A viewer that keeps up. llmwiki view refreshes after a compile without a restart and reads its port from LLMWIKI_VIEW_PORT.

    What's in 1.5.0 · Changelog

    New in 1.4 — Review answers before publishing.

    Review generated answers and their citations before they become pages, approve a batch of pages in one operation, and trace an answer back to the pages that informed it. Release notes · Upgrade guide

    New in 1.3 — A fresh look for your wiki.

    Meet Scientific Clay, with soft surfaces and rounded typography, and Minimal, which follows your system’s light or dark setting. Switch instantly between four themes, including Nebula Light and Dark.

    The same page in a demonstration wiki. Click either screenshot for a closer look.

    Recursive source folders, path exclusions, project-specific compile instructions, and storage for larger embedding indexes.

    New in 1.2 — Explore your records and their evidence.

    Browse profile-defined categories, declared fields, connected records, provenance, and supporting source passages in the local viewer.

    New in 1.1 — Publish and share domain templates.

    Create signed template distributions, discover them through explicitly trusted catalogs, and install or update them with compatibility checks.

    New in 1.0 — Configurable Lifecycle Profiles.

    Build a knowledge system around the way you work. Define your records, relationships, review gates, and workflows in one validated profile. Start with AutoSci for research or Newsroom for editorial work, or create your own.

    Explore Configurable Lifecycle Profiles →


    What llmwiki does

    Compile raw sources into an interlinked, citation-traceable markdown wiki that agents and humans can browse, query, lint, export, and reuse. The default profile preserves the classic concepts-and-queries layout; optional profiles add domain-specific types and workflows without adding domain branches to the compiler.

    llmwiki implements the LLM Wiki pattern: instead of re-discovering knowledge from raw files at query time, compile it once into durable pages that accumulate structure, provenance, review state, and retrieval metadata over time.

    When to use this repo

    Use llmwiki when you need a persistent knowledge base from raw material:

    • Compile papers, notes, READMEs, transcripts, PDFs, images, or web pages into typed wiki pages.
    • Give agents a stable, citation-aware context pack instead of a pile of loose files.
    • Keep generated knowledge auditable with source citations, review queues, freshness checks, and quality gates.
    • Browse the result locally, query it from the CLI, expose it over MCP, or embed it through the SDK.
    • Exchange compiled knowledge with other tools using Open Knowledge Format (OKF), JSON, JSON-LD, GraphML, Marp, and llms.txt.

    Do not use llmwiki as a general static-site generator, a heavy ontology database, or a replacement for ad-hoc search over fast-changing raw logs. It is strongest when source knowledge is worth compiling, reviewing, and reusing.

    What you get

    • Compiled wiki, not chunks. A two-phase LLM pipeline extracts concepts, then generates typed pages: concept, entity, comparison, and overview.
    • Configurable Lifecycle Profiles. A fail-closed .llmwiki/profile.json can declare entity schemas, typed relations, lifecycle state machines, transition requirements, workflows, artifacts, connectors, content tiers, and retrieval policy.
    • Installable domain templates. llmwiki template init autosci creates a research project with papers, ideas, experiments, manuscripts, evidence artifacts, workflows, and Crossref import. newsroom demonstrates the same machinery for editorial work.
    • Runtime trust gates. Relation, evidence, artifact, and human/agent gates are enforced by the write path rather than left as prompt conventions; standing lint detects drift after the fact.
    • Citation-traceable output. Paragraphs and claims cite source files and line ranges, and llmwiki lint validates the links.
    • Hybrid retrieval. Semantic chunk search, BM25 reranking, and wikilink graph expansion build compact evidence packs for queries and agents.
    • Local viewer. llmwiki view opens a read-only browser UI with search, page metadata, graph exploration, source-freshness badges, and citation chips.
    • Review policy. Generated pages can be auto-held for review when confidence, contradiction, schema, or provenance rules trip.
    • Freshness repair. llmwiki lint and llmwiki next surface stale/orphaned pages; llmwiki refresh --stale repairs changed knowledge without compiling unrelated new sources.
    • Eval harness. llmwiki eval reports health score, a per-page health distribution that flags the worst pages, wikilink-graph health, citation coverage/precision, corpus stats, regression deltas, and optional judge-model citation support.
    • MCP server. llmwiki serve exposes ingest, compile, query, lint, read, status, eval, context-pack, and OKF exchange tools to MCP-compatible agents.
    • SDK. createWiki({ root }) drives ingest, compile, query, context, status, export, eval, and OKF import/export from TypeScript without shelling out.
    • Open Knowledge Format exchange. Export and import OKF bundles for portable, markdown-native knowledge exchange. External OKF imports are staged through the review queue by default; trusted bundles can be written live explicitly.
    • Other portable exports. Export JSON, JSON-LD, GraphML, Marp slides, and llms.txt for downstream systems.
    • Provider portable. Anthropic, Claude Agent SDK local login, OpenAI Codex CLI local login, OpenAI-compatible servers, Ollama, GitHub Copilot, Atlas Cloud, OrcaRouter, Converse-compatible gateways, and local OpenAI-compatible runtimes.

    Configurable Lifecycle Profiles (CLP)

    CLP turns llmwiki's knowledge compiler into a reusable substrate for domain-specific knowledge systems. A validated .llmwiki/profile.json is the single contract for:

    • typed entities, fields, and directed relations;
    • lifecycle states, transition evidence, and trust gates;
    • multi-stage workflows and declared actions;
    • hash-pinned artifacts and first-party connector bindings; and
    • content tiers and retrieval behavior.

    These rules are enforced by the runtime, not left as prompt conventions. The CLI, SDK, MCP server, viewer, context builder, lint, status, export, and OKF exchange surfaces all operate from the same profile contract. Invalid profiles and writes that bypass a declared gate fail closed.

    CLP is backward-compatible by construction: a project without .llmwiki/profile.json uses the built-in default concepts-and-queries profile and preserves the pre-1.0 behavior. You can start three ways — scaffold your own profile, install a built-in or local template, or install a signed template from a trusted tap:

    # author your own profile, one entity type at a time
    llmwiki profile init research --entity paper
    
    # or install a built-in or local declarative template
    llmwiki template list
    llmwiki template inspect autosci
    llmwiki template init autosci
    
    llmwiki profile validate
    llmwiki workflow list
    

    autosci is a practical research system with papers, ideas, experiments, manuscripts, evidence artifacts, workflows, and Crossref ingestion. newsroom applies the same generic machinery to articles, desks, bylines, and editorial workflows. Templates contain configuration and examples, never executable plugin code.

    Templates can also be distributed securely. Publishers build signed, offline distributions with llmwiki template publish — Ed25519 signing, key rotation, and package revocation — and verify them with template publish verify. Consumers add explicitly trusted taps, discover and inspect signed catalogs, and install or update templates with continuity, revocation, and compatibility checks enforced under lock.

    Read the CLP concept guide, follow the AutoSci research workflow, or explore the Newsroom editorial workflow.

    Karpathy's LLM Wiki pattern

    Andrej Karpathy described the LLM Wiki pattern as a way to turn raw material into compiled knowledge that future agents can reuse. llmwiki is a concrete compiler for that pattern.

    The key shift is moving work from query time to compile time. Traditional RAG repeatedly retrieves raw chunks and asks the model to reconstruct relationships for each question. llmwiki first turns sources into typed, interlinked pages with citations, metadata, and review state. Queries, context packs, exports, and MCP tools then operate over that compiled artifact.

    That makes llmwiki useful when knowledge should compound: concepts shared across sources become one page, saved answers become future context, stale pages can be detected and repaired, and agents can consume a stable evidence pack instead of re-reading the same raw files from scratch.

    See docs/concepts/karpathy-pattern.mdx for the deeper explanation.

    Agent decision guide

    Use llmwiki for a reusable corpus of documents, research, or project knowledge. Start with wiki_status and get_context_pack when you need evidence for your own reasoning. A one-off file read or live web search does not need a wiki.

    The npm package includes an Agent Skill. See the MCP setup guide for installation. The MCP Registry identifier is io.github.atomicstrata/llmwiki; the npm package is llm-wiki-compiler and the executable is llmwiki.

    Choose the entry point that matches the task:

    GoalUse
    Create a wiki from one source and inspect itllmwiki quickstart <source>
    Start a typed research or editorial projectllmwiki template list, then llmwiki template init autosci|newsroom
    Inspect or validate the active domain contractllmwiki profile show and llmwiki profile validate
    Run a declared lifecycle workflowllmwiki workflow list, then llmwiki workflow start <id>
    Write or verify a profile-declared artifactllmwiki artifact write ... and llmwiki artifact verify <ref>
    Import external records through a connectorllmwiki connector list, then llmwiki connector run <id> --input key=value
    Add more files or URLsllmwiki ingest <url-or-file>
    Compile or recompile changed sourcesllmwiki compile
    Add project-specific writing guidancellmwiki compile --instructions ./SOUL.md; pass the option on each instructed compile
    Remove a bad source and its derived pagesllmwiki rm <source>
    Hold generated pages for human approvalllmwiki compile --review or review policy config
    Ask grounded questionsllmwiki query "question"
    Save an answer back into the wikillmwiki query "question" --save
    Build an evidence pack for another agentllmwiki context "<task>" --json or MCP get_context_pack
    Inspect the compiled knowledge basellmwiki view --open
    Check broken links, citations, confidence, freshness, and qualityllmwiki lint and llmwiki eval
    Repair stale compiled pagesllmwiki refresh --stale --dry-run, then llmwiki refresh --stale
    Drive llmwiki from an agentllmwiki serve --root <project>
    Drive llmwiki from TypeScriptcreateWiki({ root })
    Export for another systemllmwiki export --target <format>
    Export an Open Knowledge Format bundlellmwiki export --target okf --out <dir>
    Import an Open Knowledge Format bundlellmwiki import --okf <dir> --dry-run, then review/approve

    Quick start

    npm install -g llm-wiki-compiler
    
    export ANTHROPIC_API_KEY=sk-...
    # or choose another provider:
    # export LLMWIKI_PROVIDER=openai
    # export OPENAI_API_KEY=sk-...
    
    llmwiki quickstart ./notes.md
    llmwiki query "what are the key ideas?"
    llmwiki view --open
    

    quickstart ingests one source, compiles pages, and opens the viewer. Inside an existing project, run llmwiki next when you want the safest next action.

    To start with a domain model instead of the default concepts-and-queries layout:

    mkdir research-wiki && cd research-wiki
    llmwiki template inspect autosci
    llmwiki template init autosci
    llmwiki profile validate
    llmwiki workflow list
    

    Template installation is for a new or empty typed project. It materializes the chosen profile into .llmwiki/profile.json; normal project loading never depends on a template registry or lockfile.

    Demo

    Try it on any article or document:

    mkdir my-wiki && cd my-wiki
    llmwiki quickstart https://en.wikipedia.org/wiki/Andrej_Karpathy
    llmwiki query "What terms did Andrej coin?"
    

    The examples/basic/ directory includes a small pre-generated wiki you can inspect without an API key.

    Core commands

    CommandWhat it does
    llmwiki ingest <url-or-file>Fetch a URL or copy a local file into sources/.
    llmwiki ingest-session <path>Import exported Claude, Codex, or Cursor sessions into sources/.
    llmwiki quickstart <source>Ingest, compile, and optionally open the viewer in one step.
    llmwiki compileIncrementally extract concepts and generate wiki pages.
    llmwiki rm <source> [--dry-run]Delete a source and the concept pages derived exclusively from it.
    llmwiki refresh --stale [--dry-run]Recompile changed owners of stale pages and clean selected orphaned ownership.
    llmwiki template list|inspect|initDiscover and install validated declarative profile templates.
    llmwiki profile init|show|validate|diffCreate a minimal profile, inspect it, validate it, or assess profile changes.
    llmwiki workflow ...Discover and drive profile-declared workflows, stages, gates, and outputs locally, retaining run state between invocations.
    llmwiki artifact write|verifyWrite trusted profile-declared artifacts and verify hash-pinned references.
    llmwiki connector list|runDiscover first-party connectors and stage external records for review.
    llmwiki review list/show/approve/rejectInspect and manage held candidates.
    llmwiki query "question" [--save]Ask questions against the compiled wiki, optionally saving the answer.
    llmwiki context "<prompt>" --jsonBuild a citation-aware evidence pack for agents.
    llmwiki view [--open]Start the read-only local browser viewer.
    llmwiki status [--json]Report page/source counts, stale and orphaned pages, pending work, and state health.
    llmwiki lintValidate wiki structure, citations, links, metadata, and freshness.
    llmwiki eval [--suite fast|full]Measure wiki quality and optional citation support.
    llmwiki export --target <format>Export the wiki to portable formats, including Open Knowledge Format (okf).
    llmwiki import --okf <dir> [--dry-run] [--trusted]Import an Open Knowledge Format bundle, staged for review by default.
    llmwiki serve --root <dir>Start the MCP server.

    Full command docs live in docs/cli/.

    Open Knowledge Format

    llmwiki is an Open Knowledge Format (OKF) producer and consumer. OKF is a Google Cloud initiative for sharing compiled knowledge as portable markdown files with structured frontmatter.

    llmwiki export --target okf --out ./dist/okf
    llmwiki import --okf ./dist/okf --dry-run
    llmwiki import --okf ./dist/okf
    

    OKF import is intentionally review-first: untrusted bundles become review candidates, not live wiki pages. The importer preserves foreign OKF metadata, stores llmwiki provenance under x-llmwiki, and re-exports imported pages honestly after local edits, including safe original nested paths.

    See docs/guides/open-knowledge-format.mdx, docs/cli/export.mdx, and docs/cli/import.mdx.

    What llmwiki creates

    A project has raw inputs in sources/, compiled markdown in wiki/, and compiler state under .llmwiki/:

    sources/
      raw source files
    wiki/
      concepts/      compiled pages
      queries/       saved answers
      <entity>/      profile-declared typed pages
      graph/         typed relation and audit-event stores
      outputs/       derived workflow projections
      index.md       generated TOC
    .llmwiki/
      profile.json   active domain contract
      template-lock.json  advisory install provenance
      config.json    review policy and source selection
      schema.json    page-kind/cross-link policy
      state.json     source hashes and ownership
      candidates/    held review candidates
      workflows/     signed workflow run state
      eval/          quality history and thresholds
    artifacts/       hash-pinned profile-declared files and manifests
    log.md           activity journal
    

    Compiled pages are plain markdown with YAML frontmatter, plus enough metadata for agents to reason about citations, freshness, confidence, contradictions, and review state. See docs/concepts/wiki-model.mdx.

    Sources are top-level Markdown files by default. Opt into nested source folders and exclusions in project config. Excluding a compiled source retires its contribution on the next ordinary compile without deleting the source file.

    Agent integration

    MCP

    Run:

    llmwiki serve --root /path/to/wiki-project
    

    MCP clients can ingest sources, compile, query, search pages, read pages, lint, run eval, inspect status, request context packs, and exchange OKF bundles. Read-only tools work without provider credentials; LLM-backed tools validate provider credentials at call time. The run_eval tool runs its fast suite without a provider; its full suite (which LLM-judges citation support) requires one.

    See docs/guides/mcp-agent-integration.mdx.

    SDK

    import { createWiki } from "llm-wiki-compiler";
    
    const wiki = createWiki({ root: "/path/to/wiki-project" });
    await wiki.ingest({ source: "./notes.md" });
    await wiki.compile();
    const answer = await wiki.query({ question: "What changed?" });
    

    See docs/guides/sdk.mdx. llm-wiki-compiler is the supported entry point; its scoped supporting packages are implementation dependencies. The local workflow engine and an application's own coordinator are two tiers, not a migration path: llmwiki workflow runs a profile-declared workflow through local invocations with persisted state between sessions. An external coordinator calls the same SDK without using local workflow execution methods; the supporting engine dependency remains installed. See SDK package selection, SDK upgrade notes, and When to use the local engine.

    Configuration

    Minimum requirement: Node.js 24 or newer.

    The default provider is Anthropic:

    export ANTHROPIC_API_KEY=sk-...
    

    Provider selection is environment-driven:

    ProviderTypical setup
    AnthropicANTHROPIC_API_KEY or ANTHROPIC_AUTH_TOKEN
    Claude Agent SDKLocal Claude Code login, LLMWIKI_PROVIDER=claude-agent
    OpenAI Codex CLILocal Codex login / ChatGPT subscription, LLMWIKI_PROVIDER=codex-agent; explicit embedding provider required
    OpenAI-compatibleLLMWIKI_PROVIDER=openai, OPENAI_API_KEY, optional OPENAI_BASE_URL
    Converse-compatible gatewayLLMWIKI_PROVIDER=converse, explicit LLMWIKI_MODEL, LLMWIKI_CONVERSE_BASE_URL, and LLMWIKI_CONVERSE_API_KEY; separate embedding provider required when enabled
    OllamaLLMWIKI_PROVIDER=ollama, OLLAMA_HOST
    GitHub CopilotLLMWIKI_PROVIDER=copilot, GITHUB_TOKEN=$(gh auth token)
    Atlas CloudLLMWIKI_PROVIDER=atlascloud, ATLASCLOUD_API_KEY
    OrcaRouterLLMWIKI_PROVIDER=orcarouter, ORCAROUTER_API_KEY

    See docs/configuration/providers.mdx and docs/configuration/environment-variables.mdx.

    Quality and safety model

    llmwiki is designed for auditable generated knowledge:

    • Review before write. Use compile --review or .llmwiki/config.json review policy to hold risky pages as candidates.
    • Profile floors are runtime checks. Field contracts, lifecycle transitions, relation counts, evidence, and artifact requirements are enforced across page, lifecycle, workflow, import, and approval write surfaces.
    • External connector data is untrusted. First-party connectors use confined fetches and stage fenced review candidates; approval is pinned to the exact body the operator reviewed.
    • Artifacts are content-addressed evidence. Artifact reads and writes are path-confined, size-capped, schema-checked, and verified against hash-pinned references.
    • Fail-closed config. Invalid review-policy config aborts compile instead of silently disabling review.
    • Source confinement. Source snippets and import/export paths are confined to the project.
    • Freshness is explicit. Pages can be fresh, stale, orphaned, or unverified; stale pages are flagged and repairable. The JSON export is active-page-only: it carries freshness for live pages (fresh/stale/unverified); computed-orphaned pages (all sources deleted) surface only as lint and viewer signals and are dropped from the export.
    • Imported compiled knowledge is staged by default. External bundles go through the review queue unless explicitly trusted.
    • CI gates are supported. llmwiki lint and llmwiki eval can enforce quality thresholds.

    See docs/configuration/review-policy.mdx, docs/troubleshooting/stale-pages.mdx, and docs/guides/ci-quality-gates.mdx.

    Scale and what works

    llmwiki is still early software, but it is no longer a toy pipeline for a handful of notes.

    • Incremental compilation means unchanged sources do not flow back through the LLM.
    • Parallel compile runs concept extraction and page generation concurrently under a configurable cap (--concurrency / LLMWIKI_COMPILE_CONCURRENCY), cutting wall-clock on large compiles.
    • Chunk-level embeddings narrow large wikis before BM25 reranking and graph expansion.
    • Content-hash-aware embedding updates avoid recomputing vectors for unchanged pages and chunks.
    • Batch embedding sends page and chunk vectors to the provider in batches rather than one request at a time, cutting latency on cold starts and large refreshes.
    • Binary embedding storage automatically handles stores above the 64 MiB JSON limit; smaller stores can opt in. Binary selection persists, and retrieval still loads the index into memory. See limits, recovery and downgrade guidance.
    • Cached citation judgements make repeated eval --suite full runs cheaper.
    • Lexical fallback keeps query/context workflows usable when the active provider has no embedding endpoint.
    • Prompt budgeting and ingest truncation metadata make large sources explicit instead of silently pretending they fit.

    The current sweet spot is a durable project or domain wiki: research folders, codebase docs, team handbooks, standards, design notes, decision logs, or curated source packs. The less ideal fit is a high-churn firehose where raw search is enough and compiled structure would go stale faster than it can be reviewed.

    Documentation

    The full docs site source is in docs/:

    Preview the docs locally with Node 24:

    cd docs
    volta run --node 24 npx mint dev --port 3001
    

    Current release

    Stable release 1.5.0: The current release on the npm latest dist-tag. It ships the same code as the 1.5.0-rc.1 candidate.

    • Wiki behavior is separated from local file I/O. The code that decides what a wiki does, such as which sources changed, what a page links to, or whether a review candidate can be approved, used to open, lock and rename files inline. Those rules now work through focused storage boundaries, and the current local-disk layout is their first implementation. Locking, atomic writes, path confinement, symlink handling and recovery stay in the local implementation. For you, nothing changes: the same directory layout, file formats, defaults, and CLI, SDK and MCP entry points, with no migration or storage setting. This is groundwork for alternative storage and for testing wiki logic independently of the disk; it does not add hosted storage, a remote backend, role-based permissions, or a public storage-plugin API. See ARCHITECTURE.md.
    • Converse gateway provider. LLMWIKI_PROVIDER=converse connects to a bearer-authenticated endpoint speaking the Converse protocols, with streaming and structured extraction. Its AWS protocol packages are optional and load only when you select it. See provider setup.
    • --no-pages-limit extraction. llmwiki compile --no-pages-limit removes the requested 3–8 concepts per source, so a source yields the pages it supports.
    • Viewer refresh and port. llmwiki view rebuilds its snapshot in the background after a compile, stops refreshing cleanly on close, and reads LLMWIKI_VIEW_PORT when --port is not given.
    • Correctness fixes. Wikilink markup in an extracted concept name no longer creates a duplicate page, and GPT-6 Luna models work with the OpenAI provider. MCP wiki tools keep progress output off the stdio protocol stream, and a named pipe in sources/ no longer hangs freshness checks.
    • Prompt version v7. Every newly generated page records prompt version v7, including pages from default runs. Unchanged sources are not recompiled by upgrading alone.
    npm install --global llm-wiki-compiler
    

    The supporting packages are pinned to the same version and install with it. The MCP Registry entry io.github.atomicstrata/llmwiki is published for 1.5.0 after the npm release. Existing projects need no migration; read the upgrade notes for the prompt-version change and the opt-in Converse dependencies.

    Released 1.4.2: Fixes llmwiki compile with the codex-agent provider and carries the 1.4.0 features: answer review before publishing, batch approval, extraction reuse, page-level answer citations, and the split-package SDK with matching supporting dependencies.

    Experimental artifact types and workflow statuses can require source updates; read the SDK upgrade notes before upgrading from 1.3. See the 1.4.0 release summary and changelog for migration details, contributor credits, and limits.

    Released 1.3.0:

    • Four viewer themes, with Scientific Clay as the new default and Minimal following your system setting. Saved light/dark preferences migrate to Nebula Light/Dark.
    • Recursive source discovery, literal path exclusions, and explicit project-specific compile instructions.
    • Bounded binary storage for larger embedding indexes, configurable completion budgets, and protected gateway request extensions.
    • OrcaRouter embedding routing, retryable rule-language changes, and a public test type-check ratchet.

    See the 1.3.0 release summary and changelog for migration details, contributor credits, and limits.

    Released 1.2.0:

    • Profile-aware viewer with document categories, record relationships, provenance, source previews, verified local artifact access, responsive navigation and fitted graphs.
    • Additional providers and OpenAI-compatible reasoning-model controls, independent embedding configuration, and SDK embedding opt-out.
    • Optional Sources sections and additive SDK system policies with prompt-change invalidation and per-page provenance.
    • Shared-page recovery after source removal, citation line-list fixes, and conservative abbreviated-wikilink repair.

    Windows path fixes are included, but native Windows validation and CI remain outstanding. Full filesystem-confinement guarantees on Windows remain unverified. Use a supported Linux environment for untrusted projects; see #217.

    Released 1.1.0:

    • Template distribution ecosystem: publishers author signed, offline distributions with template publish init | add | build | rotate | revoke (Ed25519 signing, key rotation, package revocation) and verify them with template publish verify.
    • Consumers configure explicitly trusted template taps, discover and inspect signed catalogs, and install or update templates with continuity, revocation, and compatibility checks under lock.
    • llmwiki status command: a readable snapshot of page and source counts, last compile, stale and orphaned pages, pending changes, the review queue, active profile, and state-file health.

    Released 1.0.0:

    • Configurable Lifecycle Profiles across CLI, SDK, MCP, viewer, context, lint, status, export, and profile-aware OKF exchange.
    • Built-in autosci and newsroom templates, typed workflows and actions, first-class artifacts, typed relations and runtime lifecycle gates, plus a hardened first-party connector substrate with Crossref.
    • Typed-page semantic search and retrieval controls, batch embeddings, parallel compile, and fail-closed state recovery.

    See CHANGELOG.md for release history.

    Companion: Atomic Memory

    llmwiki and Atomic Memory are complementary open context infrastructure:

    • llmwiki compiles source material into durable, inspectable knowledge.
    • Atomic Memory gives agents runtime memory that is searchable, scoped, correctable, and inspectable.

    Use them independently or together. The @atomicmemory/llmwiki bridge imports llmwiki export --target json --project-id <id> as durable memory records.

    Contributing

    Contributions are welcome. If llmwiki is missing something you need, open an issue or PR and describe the workflow you are trying to support - need-driven improvements are often the best ones. If you want to contribute more generally, roadmap items are a good place to start. For larger changes to core compile, review, import/export, or retrieval semantics, please start with an issue or design discussion so we can align on the contract first.

    Before committing code changes, run:

    npx tsc --noEmit
    npm run build
    npm test
    npm run fallow:ci
    

    See CONTRIBUTING.md.

    License

    MIT

    Disclaimer

    No LLMs were harmed in the making of this repo.