YantrikDB

Three substantive engine improvements bundled, plus perf regression coverage to prevent future versions of the v0.7.0 → v0.7.1 incident.

What's in

🎯 Saga task 18 Option 4 — event-driven compactor wake

Pre-v0.7.2 the compactor unconditionally slept 250ms between cycles. At sustained 1000 wps, the delta refilled inside one tick and saturated before the compactor woke. yantrikdb-server's 90s bench (msg b9c98a4d) identified the racing-tick mechanism — three-phase regime change (saturation → recovery at sec 71) confirmed the compactor reaches an amortization equilibrium but takes too long to get there at default cadence.

Fix: Condvar pair on DeltaIndex. append() and tombstone() signal when delta crosses 80% of delta_max. Compactor's loop replaces thread::sleep(250ms) with wait_for_compaction_signal(250ms) — wakes within microseconds of pressure, falls back to the 250ms timeout for the age-trigger path and graceful shutdown.

| Metric (60s 8w/4r/dim=384) | v0.7.1 | v0.7.2 |
|---|---|---|
| Recovery onset (regime change) | sec 71 | sec 58 |
| Steady-state p99 (sustained) | ~90ms | ~90ms ✓ |
| Summary p99 60s | 1586ms | 953ms (-40%) |

Honest caveat. Transient growth-phase spikes during the saturation phase (sec 1-58) are unchanged. Investigation via compactor instrumentation showed they are HnswIndex::insert apply cost dominating each compaction cycle (clone_ms=7-29, apply_ms=219-401 at cold=10-20k entries). This is HNSW-irreducible cost during continuous-write on unbounded datasets — not a v0.7.x regression. Deeper fix is saga task 18 Option 3 (LSM CoW with bounded cold tier), gated on Phase 7 soak data.

🎯 Saga task 20 Slice C — db.set_embedder_named() runtime download

Companion to v0.7.0's bundled potion-base-2M default. Downloads larger model variants from yantrikos/yantrikdb-models on first call.

let mut db = YantrikDB::new("memory.db", 256)?;
db.set_embedder_named("potion-base-8M")?;
// → fetches .tar.gz from GitHub Releases
// → verifies SHA-256 against compile-pinned constant
// → caches under dirs::cache_dir() / "yantrikdb" / "models"
// → loads via model2vec-rs

| Name | Tarball size | Dim | MTEB vs MiniLM |
|---|---|---|---|
| potion-base-8M | 28 MB | 256 | ~92% |
| potion-base-32M | 121 MB | 512 | ~95% |

Behind a new embedder-download Cargo feature (off by default). Pure Rust deps: ureq (sync HTTP, no async runtime), sha2, dirs, tar, flate2. No HuggingFace dependency. No ONNX. No candle. Self-hosted model artifacts with SHA-256 pinning.

🎯 Perf regression tests

Three new tests in engine::stats::pending_ops_tests capture the v0.7.0 → v0.7.1 SELECT COUNT bug class structurally:

• pending_op_count_atomic_matches_sql_after_workload — drift detection
• log_op_pending_is_o1_under_pending_load — 5000 ops + assert <10ms
• pending_op_count_under_concurrent_mark — atomic stays consistent under racing workers

If a future contributor (or future-us) re-introduces a SELECT-on-hot-path, the regression fires immediately.

Other

• count_pending_ops_sql() exposed as #[doc(hidden)] reconciliation oracle (debug-only)
• README updated with with_default(), set_embedder_named(), slim build instructions, and a quality-vs-size table for picking an embedder tier
• compactor_drains_delta_periodically test updated — Option 4 drains the delta mid-burst, racing the prior intermediate delta_len >= 128 proxy assertion. The contract being tested (entries land in cold) is unchanged.

Tests

1419 lib tests pass (was 1416 + 3 new perf regression tests). 1423 with embedder-download feature on (+ 4 Slice C registry/url tests). Both feature configurations + slim build all clean.

Migration

Zero migration. No schema change, no API break. Just bump:

yantrikdb = "0.7.2"
# yantrikdb = { version = "0.7.2", features = ["embedder-download"] }  # opt-in for set_embedder_named

Still pending

• LSM CoW with bounded cold tier (saga task 18 Option 3) — for transient-phase p99 spike remediation
• Phase 7 long-duration soak — runs on Linux homelab when hardware time available; validates steady-state methodology
• Multi-language embedder variants — potion-multilingual-128M is 488MB, defer to future tier

Diagnosis credit

yantrikdb-server msg b9c98a4d (90s bench) — surfacing the three-phase regime change that motivated Option 4. Plus msg b951a2de from the v0.7.1 cycle that diagnosed the previous SELECT COUNT bottleneck. Without empirical cross-validation neither of these would have shipped before getting to production.