webpeel/webpeel

What's New in v0.12.0

🔬 Deep Research Agent

• webpeel research "<query>" — autonomous 5-phase pipeline: Search → Fetch → BM25 Extract → Follow Links → Synthesize
• CLI + MCP tool (webpeel_research) for Claude Desktop / Cursor
• 85% average relevance, 5/5 source success rate, ~5s per query
• Graceful degradation: no LLM key = ranked sources with BM25-filtered findings

🧹 Content Pruner v2 (Complete Rewrite)

• Two-pass architecture: semantic removal + density scoring
• Real-world token savings: Wikipedia 22%, BBC 21%, MDN 20%, StackOverflow 33%
• Enabled by default — opt out with --full-content

📊 BM25 Relevance Scoring

• computeRelevanceScore() for document-level relevance (0-1)
• Calibrated: relevant 70-94%, irrelevant 0%

🐍 Python SDK

• pip install webpeel — full client: scrape, search, batch, crawl, map, extract, screenshot, research
• Sync + async support, type hints, httpx-based

📜 Smart Infinite Scroll

• --scroll-extract now auto-detects content loading (height stabilization)
• Stops after stable height, max scrolls, or timeout

🧠 Token Efficiency Pipeline

• BM25 Query Filtering (--focus "query") — 54% savings on targeted queries
• Smart Chunking (--chunk N) — semantic/fixed/paragraph strategies
• Combined pipeline: Prune → Focus → Budget = up to 77% savings

📈 Stats

• 654 Node.js + 39 Python = 693 tests, 0 failures
• 13 MCP tools, 7 CSS schemas
• Zero TypeScript errors