lyonzin/knowledge-rag

Search Performance & Output Quality

128× Faster BM25 Search

Custom inverted-index BM25 replaces rank-bm25 full-corpus scan. Only documents containing query terms are scored via posting lists. numpy.argpartition provides O(n) top-k selection instead of O(n log n) sort.

• Batched adjacent chunk fetch — single ChromaDB collection.get() call replaces N round-trips per result
• O(1) reverse lookup via _source_to_docid dict eliminates linear scans across search, update, and remove operations

Smarter Output

Two new parameters on search_knowledge:

| Parameter | Default | Description |
|-----------|---------|-------------|
| snippet_mode | true | Truncates content to ~500 chars at natural break points. Reduces token consumption by ~72%. Adds content_length field with original size |
| min_score | 0.0 | Filters results below normalized relevance threshold (0.0-1.0). Response includes filtered_by_score count |

Both parameters are fully backwards-compatible — existing callers see improved output by default.

Changes

• PERF: Inverted-index BM25 with numpy top-k (128× speedup on 50K+ chunk corpora)
• PERF: Batch adjacent chunk fetch (single ChromaDB call)
• PERF: O(1) source→doc_id reverse lookup
• NEW: snippet_mode parameter (default: true)
• NEW: min_score parameter (default: 0.0)
• NEW: filtered_by_score + content_length response fields
• DEPS: rank-bm25 replaced by numpy (direct dependency)
• TEST: 6 new tests + updated backwards-compat baseline
• DOCS: Updated architecture flowcharts, API reference, changelog

CI Status

✅ Quality Gate — 16/16 checks passed (7 pillars)
✅ CI — 9/9 matrix cells passed (Linux + Windows + macOS × Python 3.11/3.12/3.13)
✅ Security — CodeQL passed
✅ 226 tests passed, 0 failed

Install / Upgrade

pip install --upgrade knowledge-rag
# or
npx -y [email protected]