YantrikDB

Headline

db.reembed(new_embedder_name, options) — engine-side embedder migration. Re-encode every memory under a new embedder + atomically swap the active HNSW index, while concurrent recalls continue serving and concurrent writes queue (rather than block).

Closes #41. Shipped as PR #42 — 22 commits across 12 substantive checkpoints, 1492 tests passing (default features) + 1481 slim + 1499 with embedder-download, 0 regressions.

What ships

A 5-phase state machine — Probing → Encoding → Rebuilding → Swapping → Verifying → Completed — invoked via:

let report = db.reembed("potion-base-32M", ReembedOptions::default())?;
// report.generation       — new active generation
// report.encoded_count    — rows re-encoded under the new embedder
// report.build_hwm        — covers_through_seq captured at the cutover barrier
// report.duration         — wall-clock runtime

Or with progress + custom HNSW params + namespace scope:

let opts = ReembedOptions {
    namespace: Some("default".to_string()),
    progress_cb: Some(Box::new(|p| println!("{:?} {}/{:?}", p.phase, p.processed, p.total))),
    batch_size: 512,
    hnsw_m: Some(32),
    write_policy: ReembedWritePolicy::Queue,  // default
    ..Default::default()
};
let report = db.reembed("potion-base-32M", opts)?;

The Queue write policy means concurrent writers during the cutover don't block — they're queued to oplog (as TEXT) and the post-swap materializer re-encodes them under the new embedder.

Correctness invariants locked

4 brainstorm sessions with gpt-5.5 caught 10+ critical correctness bugs in the design space before any shipped:

• No read-side dim split-brain — SearchState is the single atomic publication unit (provenance + embedder + dim + generation + covers_through_seq + HNSW params + vec_index). ArcSwap-wrapped.
• No stale-writer commit — record_text runs the writer-revalidation loop: embed outside the WriteRouter guard, acquire guard, re-check (generation, digest), retry if a reembed swap completed under the embed step.
• No compactor generation rollback — try_publish_search_state is the only SearchState mutator; rejects strictly-lesser-generation publishes via ArcSwap::compare_and_swap.
• Durable crash recovery — meta.active_generation + memories.embedding_generation form the single durable linearization point. open() reads them and rebuilds SearchState; staging columns from a crashed mid-reembed are discarded if SQL didn't commit, or carried forward if it did.
• SQL embedding boundary — DurableEmbeddingStore module is the only sanctioned path through which engine code reads memories.embedding. A compile-test (recall_rs_has_no_raw_sql_embedding_reads) fails the build if a future refactor reintroduces raw SQL access in the hot recall path.

All 8 brainstorm-4 §10 invariants have regression tests. End-to-end correctness proof: reembed_post_swap_recall_returns_rows_under_new_embedder plants rows under E0 (sentinel 0.42), reembeds to E1 (sentinel 0.99), recalls under E1 vector space, all rows return.

v28 schema migration

Two new surfaces:

• memories.embedding_generation INTEGER — per-row stamp of which SearchState generation the row was encoded under. NULL on pre-v28 rows (treated as generation 0). Reembed Phase-2's swap transaction writes the new generation atomically with promoting embedding_new into embedding.
• meta.active_generation row — durable record of the current active SearchState generation. Read by open() and used to rebuild the SearchState across restarts.
• idx_memories_embedding_generation — supports the post-swap materializer's "scan rows under stale generation" query.

Migration is additive + idempotent. No data mutation on existing rows.

Limitations / deferred (documented)

• Cross-dim reembed is not supported in v1. The engine's standalone embedding_dim field still gates ~7 sites (engine/conflict.rs, engine/indices.rs, engine/record.rs, distributed/replication.rs); retiring it is a separate structural increment. Cross-dim returns a typed error pointing at the escape hatch ("open new DB at new dim, copy memories"). Same-dim covers the primary upgrade use case (model version bump within vector space).
• HTTP backend — this release is the engine primitive. The corresponding yantrikdb-server REST endpoint + cluster-mode reembed (openraft applier semantics) is a downstream PR. yantrikdb-hermes-plugin embedded mode is unblocked now; HTTP-mode plugin users wait for the server PR.

Test counts

| Build | Tests passing |
|---|---|
| cargo test --lib --release (default features) | 1492 |
| cargo test --lib --no-default-features --release (slim) | 1481 |
| cargo test --lib --features embedder-download --release | 1499 |
| cargo fmt --check --all | ✅ clean |
| cargo clippy | ✅ clean |

Compatibility

db.reembed() is purely additive. All pre-v0.7.17 APIs work unchanged. The v27 schema (shipped in v0.7.16) and the new v28 schema are both forward-only — opening a v0.7.16 DB with v0.7.17 binary triggers idempotent additive migration; opening a v0.7.17 DB with v0.7.16 binary stays at v27 and ignores the new columns.

Acknowledgements

Designed in 4 brainstorm rounds with gpt-5.5; each round caught real correctness bugs (not gold-plating). hermes-agent shaped the ergonomics asks (implicit auto-load, Probing-emits-total). The substrate-paper experiment (agent-in-loop catching design-level + structural-fit bugs before they ship) continues to validate.

— closes #41