pomazanbohdan/memory-mcp-1file

🚀 Memory MCP v0.5.8: Robust Indexing & Rate Limiting Pipeline

This release completely overhauls the background indexing mechanism to be perfectly resilient to mass file changes (like git pull, checkout, or mass deletion) without causing CPU/Memory spikes or full redundant re-indexes.

🌟 Key Features

• File Manifest Diffing: We now store a file_manifest (mtime, size, blake3) in the database. When the MCP server starts, it compares the actual disk state against the DB manifest. You will never have a stale index after pulling new code before starting the server.
• Index Worker Queue: All file events (startup validation, real-time file watcher, and periodic catch-ups) now flow through a unified IndexWorker.
• Smart Rate Limiting & Batching: The IndexWorker batches file processing (default 20 files) and introduces adaptive throttling (default 2s) to prevent any CPU/RAM spikes even if 50,000 files are changed at once.
• Debounced BM25 Rebuilds: The search engine index rebuild is now debounced. No more expensive O(N) rebuilds for every incremental change. It now intelligently waits until the processing queue empties.
• Optimized Embedding Pipeline: Embedding queue capacity increased (from 64 to 256) and processing batch size increased (from 2 to 8) to get maximum throughput from your local/remote LLM models.
• Directory-level MTime scanning: Periodic background checks now check directory mtime rather than performing expensive full-disk tree scans.

⚙️ New Configuration Environment Variables

Added several new environment variables to tune the pipeline for your hardware:

• EMBEDDING_QUEUE_CAPACITY (Default: 256)
• EMBEDDING_BATCH_SIZE (Default: 8)
• INDEX_BATCH_SIZE (Default: 20)
• INDEX_DEBOUNCE_MS (Default: 2000)
• MANIFEST_DIFF_INTERVAL_MINS (Default: 10)

Enjoy the most stable memory system yet!