cocoindex

What's Changed

• fix(engine): a live component that calls operator.update() before any update_full() no longer loses what it mounted on its first run. Its existence row was only written at its parent's commit, so the first child's eager existence write filed it as a directory node, and the parent's commit then tombstoned every child mounted so far. Watch-only feeds (Kafka, Iggy, any LiveMapFeed with only watch()) hit this on a first run against a backlog; later runs were unaffected. Items a first run lost come back when the source delivers them again by @georgeh0 in https://github.com/cocoindex-io/cocoindex/pull/2673
• fix(engine): a memoized function whose tracked state changed while its memo stayed valid had its target states and nested memo entries deleted at commit; they are now kept. A component's memo refresh now writes only to the row it validated, so a concurrent run of the same component can no longer be overwritten under it by @georgeh0 in https://github.com/cocoindex-io/cocoindex/pull/2660
• fix(litellm): LiteLLMEmbedder accepts encoding_format="base64" responses (a much smaller payload over the wire), probes the right dimension for them, and refuses dtype options (vLLM embed_dtype, Voyage output_dtype) that would decode into a wrong vector. The default float path is unchanged by @johannespfrang-h-d in https://github.com/cocoindex-io/cocoindex/pull/2455
• Docs site redesign
  • The docs follow cocoindex.io's design system. The examples catalog filters by category, source, target, model and level, and each of the 32 walkthroughs opens with a "What you'll build" scene drawn from the example's own files by @badmonster0 in https://github.com/cocoindex-io/cocoindex/pull/2639 and https://github.com/cocoindex-io/cocoindex/pull/2640
  • Core Concepts has the pipeline playground on the page and a shorter reading path by @badmonster0 in https://github.com/cocoindex-io/cocoindex/pull/2653, https://github.com/cocoindex-io/cocoindex/pull/2654 and https://github.com/cocoindex-io/cocoindex/pull/2656
  • Navigation and reading on phones, search, tabs and accessibility fixes by @badmonster0 in https://github.com/cocoindex-io/cocoindex/pull/2655, https://github.com/cocoindex-io/cocoindex/pull/2645, https://github.com/cocoindex-io/cocoindex/pull/2644 and https://github.com/cocoindex-io/cocoindex/pull/2641
  • Clearer programming guide concepts, navigation and example copy by @badmonster0 in https://github.com/cocoindex-io/cocoindex/pull/2648 and https://github.com/cocoindex-io/cocoindex/pull/2652
  • The map() section is easier to find in search by @rchiang in https://github.com/cocoindex-io/cocoindex/pull/2325

New Contributors

• @rchiang made their first contribution in https://github.com/cocoindex-io/cocoindex/pull/2325
• @johannespfrang-h-d made their first contribution in https://github.com/cocoindex-io/cocoindex/pull/2455

Full Changelog: https://github.com/cocoindex-io/cocoindex/compare/v1.1.0...v1.1.1

What's Changed

Connector key fixes: no more collisions or lost writes

Every source item and target state has a key. CocoIndex uses the key to match an object with its previous run, and to decide what to create, update or delete. Prompted by the Google Drive fix in #2485, this release audits how every Python connector builds its keys. It fixes each case where the connector gave two different objects one key, or where a key that changed for the same object made CocoIndex delete that object. Two keys you choose can also reach one object, such as overlapping directories or two ContextKeys for one database. CocoIndex can't detect that, so it stays your responsibility; the docs now point out where it can happen.

Two different objects shared one key, so one overwrote or deleted the other:

• Google Drive: files with the same name, even in one folder. Items are now keyed by file ID by @Sujit-1509 in https://github.com/cocoindex-io/cocoindex/pull/2485
• S3, Azure Blob and OCI: object keys that differ only in a doubled or leading /, a ./ segment, or \ versus /; and, with prefix="docs", docs/a.md and docsa.md. Paths in different buckets also compared equal by @georgeh0 in https://github.com/cocoindex-io/cocoindex/pull/2601
• Kafka: the same message key on two topics of one topic_as_map feed. A tombstone on one topic deleted the other topic's outputs by @georgeh0 in https://github.com/cocoindex-io/cocoindex/pull/2597
• Neo4j and FalkorDB: a node label and a relationship type with the same name, and derived edge ids of different endpoints (1 vs "1", ids containing _). The fix comes as new node and relationship target APIs (see "When you upgrade") by @georgeh0 in https://github.com/cocoindex-io/cocoindex/pull/2612
• SurrealDB: derived relation IDs of different endpoints, e.g. doc:'chunk_1' and doc_chunk:1 linked to the same tag by @georgeh0 in https://github.com/cocoindex-io/cocoindex/pull/2599
• zvec: 1 and "1" in an Any, object or union key column became one document by @georgeh0 in https://github.com/cocoindex-io/cocoindex/pull/2608 and https://github.com/cocoindex-io/cocoindex/pull/2613
• Postgres: two vector indexes on one table whose names PostgreSQL truncates to the same 63 bytes, so only one of them could exist by @georgeh0 in https://github.com/cocoindex-io/cocoindex/pull/2603 and https://github.com/cocoindex-io/cocoindex/pull/2615
• Valkey: a payload key or FieldDef named vector, the hash field that holds the document's vector by @georgeh0 in https://github.com/cocoindex-io/cocoindex/pull/2606

One object's key changed between runs, and the delete for the old key removed what the new key had just written:

• localfs: a case-only rename (on macOS and Windows), or ./a.txt changed to a.txt. The file stayed missing until its content changed by @georgeh0 in https://github.com/cocoindex-io/cocoindex/pull/2598
• Qdrant: a UUID point ID written in another text form (upper case, no hyphens, URN) by @georgeh0 in https://github.com/cocoindex-io/cocoindex/pull/2602
• Kafka target: a message key that switched between str and bytes. Its tombstone erased the record on a compacted topic by @georgeh0 in https://github.com/cocoindex-io/cocoindex/pull/2597
• Neo4j and FalkorDB: a renamed vector index by @georgeh0 in https://github.com/cocoindex-io/cocoindex/pull/2611

A delete missed its object, leaving stale data behind:

• LanceDB: every row delete failed for a key column that needs quoting (doc-id), or that holds binary, fixed-size binary or the int64 minimum by @georgeh0 in https://github.com/cocoindex-io/cocoindex/pull/2600
• Doris: keys containing \. Each update appended a duplicate row, removed items stayed, and a trailing \ broke the statement by @georgeh0 in https://github.com/cocoindex-io/cocoindex/pull/2605
• Neo4j and FalkorDB: an edge with an explicit id whose endpoints changed. Neo4j kept the old edge beside the new one, and FalkorDB failed with `unique constraint

What's Changed

• Security: exclude litellm 1.107.0, which was published to PyPI with a credential stealer and backdoor, from every litellm requirement (the litellm, entity_resolution_llm and all extras, and the examples). The locked version stays 1.96.2, which is unaffected by @georgeh0 in https://github.com/cocoindex-io/cocoindex/pull/2576
• Preview works on real pipelines and writes nothing
  • Preview now handles target states under a container (rows in a Postgres / Qdrant / LanceDB table, and so on), so it no longer stops at once on most pipelines. A preview no longer stores, deletes or refreshes memos and no longer writes committed state, so it can't make the next real update skip work. Connector authors: TargetActionSink.from_fn(fn, child_handler=...) / from_async_fn(...) replace from_fn_with_children / from_async_fn_with_children, which now raise DeprecationWarning and don't support preview. Neo4j, FalkorDB and SurrealDB ConnectionFactory keep one connection per event loop and gain close() (neo4j, falkordb). FalkorDB now requires falkordb>=1.6.0 by @badmonster0 in https://github.com/cocoindex-io/cocoindex/pull/2510
• feat(engine): max_inflight_components now bounds every running component. A parent lends its slot to one child at a time, so wide fan-outs stay within the limit and a hierarchy of any depth can't deadlock by @georgeh0 in https://github.com/cocoindex-io/cocoindex/pull/2486
• feat(core): coco.logic_propagated_to_callers(fn) makes a callback's logic count toward its callers' memos, even when it is called through a library function that uses logic_tracking="self". Editing the callback now invalidates those memos by @georgeh0 in https://github.com/cocoindex-io/cocoindex/pull/2483 and https://github.com/cocoindex-io/cocoindex/pull/2529 (coco.logic_tracked remains as a deprecated alias)
• feat(code): Nix syntax support for splitting and structural code patterns by @mulatta in https://github.com/cocoindex-io/cocoindex/pull/2434
• Rust SDK fixes
  • Memoized calls join the enclosing caller's frame, so a memoized caller no longer returns stale values or deletes its children's outputs on a cache hit. Mounting inside #[function(memo)] is rejected, as in Python. use_mount! / mount_each! key by memo identity, so touching a file no longer re-runs every component. Connectors use one sink per handler instead of one apply per row by @badmonster0 in https://github.com/cocoindex-io/cocoindex/pull/2503
  • App::drop_state and normal updates now remove targets declared through root providers (directory files, Postgres/SQLite/LanceDB/Qdrant/… tables) when they're no longer declared. Breaking: the neo4j/falkordb table and relation target functions take &ContextKey<Graph> instead of &Graph by @badmonster0 in https://github.com/cocoindex-io/cocoindex/pull/2512
• fix(graph): cocoindex drop and un-declaring a table delete the Neo4j/FalkorDB nodes and relationships the app wrote, and a primary-key change rebuilds the table instead of leaving old-key nodes behind by @georgeh0 in https://github.com/cocoindex-io/cocoindex/pull/2498
• fix(neo4j): an apply batch retries on transient errors such as DeadlockDetected instead of failing the component by @georgeh0 in https://github.com/cocoindex-io/cocoindex/pull/2491
• fix(ops): LiteLLMEmbedder embeds text starting with data:…;base64,, gs:// or files/ as text rather than as a file reference. A request litellm refuses to build no longer retries until the 10-minute deadline; the batch is split to isolate the bad text by @georgeh0 in https://github.com/cocoindex-io/cocoindex/pull/2532
• fix(code_match): regex prefilters no longer drop nested alternatives, which made searches miss matches. regex_required_literals is now public by @georgeh0 in https://github.com/cocoindex-io/cocoindex/pull/2525
• fix(fn): @coco.fn(batching=...) / runner=... are now type errors (use @coco.fn.as_async), and a lone max_batch_size= on

What's Changed

• feat(env): add Environment.close() to release the LMDB environment explicitly, so the same db path can be reopened without waiting for every Python reference to be collected (fixes EnvAlreadyOpened on free-threaded Python 3.14, where threads started inside a component kept the environment alive) by @georgeh0 in https://github.com/cocoindex-io/cocoindex/pull/2481
• feat(memo): a memo state function can return recollect_after_run=True in MemoStateOutcome to have its state collected again after a re-run, for states that depend on what the function body read by @georgeh0 in https://github.com/cocoindex-io/cocoindex/pull/2474
• Rust SDK ergonomics: #[cocoindex::function(memo, batching)] with per-item memoization, the context_key! macro, and connector schemas derived from Rust row types (SchemaFields, TableSchema::from_row) with vector dimensions resolved at runtime from the embedder by @tomz-alt in https://github.com/cocoindex-io/cocoindex/pull/2287
• GPUPool rewritten in Rust; it can now acquire multiple whole GPUs at once, and no longer loses wake-ups across event loops by @YaxinCheng in https://github.com/cocoindex-io/cocoindex/pull/2277
• Performance for large target updates
  • Much less memory per declared target state: declared states are released right after pre-commit, and Python row values are held in encoded form (a 1.14 M-row component previously held ~1.33 GiB of row dicts) by @georgeh0 in https://github.com/cocoindex-io/cocoindex/pull/2469
  • fingerprint_object / memo fingerprints compute plain data natively (~9× faster per row); existing fingerprints are unchanged by @georgeh0 in https://github.com/cocoindex-io/cocoindex/pull/2470
  • Target handlers can set tracks_value_fingerprint = True to skip reconcile() for unchanged values, so an update where most rows are unchanged runs no per-row Python at commit by @georgeh0 in https://github.com/cocoindex-io/cocoindex/pull/2472
  • Postgres: upserts go through INSERT ... SELECT FROM unnest(arrays) for tables with builtin scalar/array columns by @georgeh0 in https://github.com/cocoindex-io/cocoindex/pull/2471
  • Postgres: multi-column-key deletes use IN (VALUES ...) instead of an OR chain whose planning time grew quadratically by @georgeh0 in https://github.com/cocoindex-io/cocoindex/pull/2458
• fix(core): when one component's write fails (e.g. a target connector's reconcile() raises), only that component fails, not every component batched into the same transaction (including other apps sharing the Environment) by @georgeh0 in https://github.com/cocoindex-io/cocoindex/pull/2451
• fix(core): scoped coco.exception_handlers now receive failures of mount_each items from a LiveMap (and auto_refresh items) by @georgeh0 in https://github.com/cocoindex-io/cocoindex/pull/2478
• fix(core): preview no longer reports actions more than once when its write batch re-runs after the LMDB map grows by @georgeh0 in https://github.com/cocoindex-io/cocoindex/pull/2447
• fix(surrealdb): statement errors are raised instead of swallowed; values sent as typed CBOR; string/integer/array record IDs; record writes use UPSERT ... MERGE and add TableTarget.declare_fields for multi-owner records; HNSW-only vector indexes for SurrealDB 3 by @martinschaer in https://github.com/cocoindex-io/cocoindex/pull/2439
• fix(valkey): un-declaring an index or running App.drop() now also deletes its document hashes by @georgeh0 in https://github.com/cocoindex-io/cocoindex/pull/2419
• fix(valkey): CocoIndex connections are identifiable in the server's CLIENT LIST (LIB-NAME cocoindex); requires valkey-glide>=2.5.2 by @AbhayMahalle in https://github.com/cocoindex-io/cocoindex/pull/2443
• fix(localfs): recursive walk_dir stops at symlink cycles instead of walking forever by @Sujit-1509 in https://github.com/cocoindex-io/cocoindex/pull/2456
• fix(amazon_s3): zero-length reads and partial reads of empty objects return empty bytes