YantrikDB

Headline

Two postmortem-driven fixes from the 2026-05-20 CT 132 bench + trader-default SQL probe:

1. Compensating DELETE on Backpressure — closes the orphan-on-Backpressure pattern that left 1203 orphan rows in CT 132's bench DB and 23k orphans on trader's default accumulating over 39 days.
2. replication_apply_log audit table (v29 migration) — replication-apply paths now write an audit row. Operators can run a three-population query to distinguish "locally originated" / "received via replication" / "true orphan" — silent-shed becomes queryable.

Builds on v0.7.18 which fixed the compactor-spawn wedge. v0.7.18 made backpressure transient; v0.7.19 prevents leak on each transient event.

Empirically validated on yantrikdb-server's CT 132 LXC: post-v0.7.19 bench shows memories=5889, oplog=5889, true_orphan=0 (vs v0.7.18's memories=6937 oplog=5734 = 1203 orphans).

Problem 1 — Orphan-on-Backpressure

yantrikdb-server's v0.7.18 bench surfaced the smoking gun: memories(6937) − record_with_rid(5734) = 1203 = EXACTLY the 503 count.

Critical path that leaked:

1. INSERT INTO memories ... — row committed
2. vec_index.append(...) returns Err(Backpressure) (delta full)
3. Error propagates up; handler returns 503
4. log_op(record, ...) NEVER reached
5. Memory row stays in SQL with no oplog provenance

Same pattern accumulated to 23,043 orphans on trader's default over 39 days. v0.7.18's wedge fix made the Backpressure transient (compactor drains), but each transient event still leaked a row.

Fix

Wrap vec_index.append in if let Err(e), run a compensating DELETE FROM memories WHERE rid = ? on the failure path, propagate the error. Only fires on rare failures. Caller sees a clean "write rejected" outcome; SQL state matches what the caller observes.

Applied to all three write paths:

• record() — single rid, single DELETE
• record_batch() — N rids, DELETE all on first failure (atomic batch contract)
• record_with_rid() — only DELETE when was_new_row=true (replay path's INSERT OR IGNORE didn't create the row; we shouldn't yank a pre-existing one)

Problem 2 — Replication apply is invisible to oplog audit

Trader's default DB probe showed 23,043 memories with zero local oplog rows for record/record_with_rid/consolidate/correct. Root cause: replication-apply paths in distributed/replication.rs (materialize_record, materialize_consolidate, materialize_correct) INSERT into memories but deliberately do NOT log a local oplog row — that would create a replication loop.

Without a sibling audit table, "received via replication" is indistinguishable at SQL level from "true orphan (write-path bug)."

Fix

New v29 schema migration:

CREATE TABLE replication_apply_log (
    rid           TEXT PRIMARY KEY,
    op_type       TEXT NOT NULL,
    source_actor  TEXT NOT NULL,
    applied_at    REAL NOT NULL
);
CREATE INDEX idx_replication_apply_log_source_actor
    ON replication_apply_log(source_actor, applied_at);

Each replication-apply site writes a row at apply time:

• materialize_record → op_type='record'
• materialize_consolidate → op_type='consolidate'
• materialize_correct → op_type='correct'

Three-population audit query

-- Locally originated
SELECT COUNT(*) FROM memories
WHERE rid IN (
  SELECT target_rid FROM oplog
  WHERE origin_actor = '<self_actor_id>'
    AND op_type IN ('record', 'record_with_rid', 'consolidate', 'correct')
);

-- Received via replication
SELECT COUNT(*) FROM memories WHERE rid IN (SELECT rid FROM replication_apply_log);

-- True orphan (post-v0.7.19, expected: 0 on healthy DBs)
SELECT COUNT(*) FROM memories
WHERE rid NOT IN (SELECT target_rid FROM oplog WHERE target_rid IS NOT NULL)
  AND rid NOT IN (SELECT rid FROM replication_apply_log);

For existing deployments (trader's 23k pre-v0.7.19 rows): historical rows do NOT auto-populate the audit log. Operators can run a one-off SQL to backfill suspected replication apply:

INSERT INTO replication_apply_log
SELECT rid, 'record', origin_actor, updated_at
FROM memories
WHERE rid NOT IN (SELECT target_rid FROM oplog WHERE target_rid IS NOT NULL);

A note on trader's mystery: sync_peers probe confirmed trader is NOT a replication follower (sync_peers=0 on both DBs). So those 23k pre-v0.7.19 orphans came from some non-oplog write path (suspected: pre-oplog migration data OR yantrikdb import CLI subcommand OR pyo3-direct pre-HTTP path). Investigation is queued for v0.7.20+; not blocking this release.

Validation

Empirically validated by yantrikdb-server on CT 132 LXC (Debian 13, 2-core, 4GB):

| Metric | v0.7.18 | v0.7.19 |
|---|---|---|
| ok writes (15s bench) | 5734 | 5889 |
| 503 responses | 1203 | 724 |
| memories — oplog rwr | 1203 (orphans) | 0 |
| true_orphan from audit query | n/a | 0 |
| yantrikdb-compa thread present | ✓ | ✓ |
| replication_apply_log table | n/a | ✓ |
| meta.schema_version | 28 | 29 |

503s during the bench are transient backpressure (compactor draining), not the wedge. Every accepted write has its oplog entry; every rejected write leaves no trace.

Test counts

| Build | Tests |
|---|---|
| cargo test --lib --release (default) | 1497 (1494 + 3 new) |
| cargo fmt --check --all | ✅ clean |

3 new regression tests:

• record_with_rid_backpressure_does_not_leak_orphan_memories_row — pump past delta_max, catch Backpressure, assert no orphan row
• schema_v29_fresh_install_has_replication_apply_log_table
• test_materialize_record_writes_replication_apply_log (in replication.rs tests module — engine A records, engine B applies via apply_ops, B's audit log records source_actor='actor-A')

Migration

Backwards-compatible. CREATE TABLE IF NOT EXISTS is idempotent. No data mutation on existing rows. v28 → v29 migration runs once on next open; subsequent opens are no-op.

For downstream consumers (yantrikdb-server, plugin embedded mode, CLI):

# Bump dep
yantrikdb = "0.7.19"

No code change required — both fixes activate automatically.

Closes

References swarmcode threads with yantrikdb-server: 537ac7be, 36dcfaeb, 25192f6d, 278ab045, f2476f62, 45fdc641, 8bd002da, f6b5e15d, dd74a55b, 820a0aea, 449373ff, 3d1b84f1, 6103fbd7, 96a55603, 95ffb7ca, d43f9108, ac732cb1 (this PR locks the discussion).

Engine-side fix completes the v0.7.x reliability arc started by v0.7.18. yantrikdb-server v0.8.17 will bundle both engine fixes.

🤖 Generated with Claude Code