YantrikDB

TL;DR

Multilingual embedder works now. Issue #15 closed.

set_embedder_named("potion-multilingual-128M") was silently producing empty cache dirs and failing — broken since v0.7.9 shipped 2026-05-12, fixed today.

The bug

The tarball extractor in crates/yantrikdb-core/src/embedder/downloaded.rs unconditionally stripped the leading path component of every tarball entry. This was correct for the v0.1.0 model artifacts (potion-base-8M.tar.gz / potion-base-32M.tar.gz) which nest files under a top-level directory matching the model name. But the v0.2.0 multilingual artifact (potion-multilingual-128M.tar.gz) packages files at the archive root — so the unconditional strip produced an empty path for every entry, the loop's continue skipped them all, and no files made it into the cache dir. The post-extract sanity check then failed with "after extract, expected files missing."

The fix

Per-entry component-count check. If the path has 2+ components, strip the leading one (v0.1.0 layout). If 1, write as-is (v0.2.0 layout). Refactored into a dedicated extract_tarball_to(bytes, dest_dir) helper so it's unit-testable without network.

Tests

Three new tests using an in-memory tar.gz builder:

• extract_tarball_files_at_root_layout — exercises the v0.2.0 case. Pre-fix would have produced an empty dir; post-fix asserts all four expected files land at dest_dir root.
• extract_tarball_files_under_prefix_layout — symmetric guard for v0.1.0. Ensures the fix doesn't regress the existing case.
• registry_includes_potion_multilingual_128m — locks the registry entry's properties (dim=256, release_tag=v0.2.0, sha256 length).

Upgrade path

pip install --upgrade yantrikdb

For the multilingual path:

from yantrikdb import YantrikDB
db = YantrikDB("./mem.db", embedding_dim=256)
db.set_embedder_named("potion-multilingual-128M")
db.record_text("Bonjour le monde", "semantic")
hits = db.recall_text("hello world", top_k=5)

First call downloads ~460 MB from yantrikos/yantrikdb-models, SHA-256-verified, caches under dirs::cache_dir(). Subsequent calls (any process) hit the cache.

Also fixed

• Stale error message: unknown embedder name no longer lists only potion-base-8M, potion-base-32M — now includes potion-multilingual-128M (was added to the registry in v0.7.9 but the help text never updated).

Discipline note

This bug shipped in v0.7.9 because I never ran set_embedder_named("potion-multilingual-128M") end-to-end after uploading the v0.2.0 tarball + adding the registry entry. Same user-side-smoke discipline failure as v0.7.11. Going forward: every new entry in the downloadable embedder registry gets an end-to-end fetch+extract+load smoke (gated on env var so CI doesn't depend on network) before claiming the entry works.

Known issues (carried forward)

| Issue | Status |
|-------|--------|
| #16 — clippy cleanup (~73 warnings) | open, follow-up cleanup |
| #17 — pylint W cleanup (~73 codes) | open, follow-up cleanup |
| #18 — examples + benches drift | open |
| #19 — macOS native linker flags | open |
| #20 — pyo3 0.28 cleanup | open |
| #22 — compactor flake on Windows/macOS-14 | open, still flaky (macOS-14 failed on this PR's CI run, confirming it's real) |