YantrikDB

Headline

spawn_all_workers — engine-side fix for the fresh-install delta_max wedge surfaced 2026-05-20 in yantrikdb-server CT 132 benching. Adds a bundled spawn API that wires up BOTH the materializer + the compactor in one call, eliminating the "easy-to-forget the compactor" footgun that wedged servers at exactly 256 writes.

Closes (informally) the CT 132 wedge investigation. Shipped as PR #43.

The bug

yantrikdb-server v0.8.16 startup called engine::materializer::spawn_materializers(&db_arc, 6) but did NOT call engine::materializer::spawn_compactor(&db_arc). Without the compactor thread, the in-memory DeltaIndex.delta Vec (write-behind buffer for the HNSW cold tier) filled to delta_max (default 256). Every subsequent write then returned synchronously:

503 "ingest queue full (256 pending ops, max=256); retry after 50ms"

Operators read the error and looked at oplog tuning — but the oplog drained fine; the materializer was healthy. The actual stuck queue was the in-memory delta tier (no thread was calling vec_index.compact()).

The smoking gun (CT 132 wedge, pid 8173): gdb --batch -ex "thread apply all bt 20" showed zero threads named yantrikdb-compa (Linux truncates the yantrikdb-compactor thread name to 15 chars). The compactor simply wasn't spawned.

The fix

New public function spawn_all_workers returns AllWorkerGuards { materializers, compactor }. Production callers (yantrikdb-server, plugin embedded mode, CLI applications) switch from spawn_materializers to spawn_all_workers and get both worker pools spawned in one call.

// Before (v0.7.17 and earlier — bug):
let _materializers = yantrikdb::engine::materializer::spawn_materializers(&db_arc, 6);
// MISSED: spawn_compactor → wedge at write #257

// After (v0.7.18+):
let _workers = yantrikdb::engine::materializer::spawn_all_workers(
    &db_arc,
    yantrikdb::engine::materializer::recommended_worker_count(),
);

spawn_materializers + spawn_compactor remain available as primitives for tests and advanced callers. Docs on the new function call out the failure mode so future readers understand the bundle's rationale.

Regression tests

Two new tests, both green:

1. spawn_all_workers_bundles_materializer_and_compactor — drives 350 writes (past delta_max=256) via the new bundled API; verifies all 350 land in memories without Backpressure. This is the fix verification.

2. spawn_materializers_without_compactor_wedges_at_delta_max — reproduces the CT 132 bug: spawns ONLY materializers (mirroring the server-side bug), drives writes past delta_max, asserts Backpressure { pending: >=max } fires. This locks the failure mode against future regression — a refactor that accidentally re-orders spawn or makes the compactor opt-in fails this test.

Plus a sibling fix: engine::write_router::tests had 3 timing-sensitive tests using thread::sleep(20ms) then asserting r.inflight() == N. The macOS-14 aarch64 CI runner sometimes hasn't scheduled the spawned writer thread within 20ms. Replaced fixed sleeps with polling loops (up to 500ms). Pre-existing issue from issue #41 cp 2; just made the tests robust on slow scheduler-warm CI.

Test counts

| Build | Tests passing |
|---|---|
| cargo test --lib --release (default) | 1494 (1492 + 2 new) |
| cargo test --workspace --no-default-features (slim) | 1483 |
| Engine-direct bench at writers=32, readers=4, 30s, dim=384 | 16,078 writes, no Backpressure, delta=0 cold=17915 at end |

cargo fmt --check --all: clean. cargo clippy: no new errors.

Migration

For yantrikdb-server (and any downstream that wires up the engine workers manually):

use yantrikdb::engine::materializer::{recommended_worker_count, spawn_all_workers};

// Replace your existing spawn_materializers call with:
let _workers = spawn_all_workers(&db_arc, recommended_worker_count());
// Store _workers wherever you previously stored the materializer guards;
// the bundle includes both materializer + compactor and shuts both down
// on Drop.

recommended_worker_count() is cores/2, clamped to [2, 16]. On 4-core hardware that's 2 workers; on 16-core it's 8.

Postmortem item (not in this release)

Trader's default DB has 23,043 memories but ZERO record/record_with_rid ops applied=1 in oplog — surfaced by yantrikdb-server's SQL probe during this investigation. Trader's trader_ledger DB shows the same magic-number wedge at exactly 256 record_with_rid ops. The "fresh-install" framing of this bug undersells it: the silent-write-loss pattern has been present for 39 days across both production DBs.

v0.7.18 fixes the spawn issue going forward; the historical silent-shed pattern + how memories end up in the table without an oplog row (alt-write path?) are separate postmortem items tracked for a future release.

v0.7.19 follow-up

The combined-log_op-INSERTs optimization yantrikdb-server greenlit alongside this fix is deferred to v0.7.19. Shipping the compactor fix alone in v0.7.18 because trader was wedging in production; risk-management favored focused release > bundled release.

Closes

References swarmcode threads with yantrikdb-server: messages 537ac7be, 36dcfaeb, 25192f6d, 278ab045, f2476f62, 45fdc641, 8bd002da, f6b5e15d, dd74a55b, 820a0aea, 449373ff, 3d1b84f1.

🤖 Generated with Claude Code