GoldenMatch

What's New

Iterative LLM Calibration

• Replaces brute-force LLM scoring with stratified sampling + grid-search threshold learning
• Samples ~100 pairs per round, converges in 2-3 rounds (~200 pairs, ~$0.01)
• Previously: scored 37,500 pairs (~$0.50, ~25 minutes, rate limit issues)
• Concurrent LLM requests via ThreadPoolExecutor (configurable max_workers)
• Thread-safe BudgetTracker with threading.RLock
• Default batch_size bumped from 20 to 75
• "Never demote" behavior: LLM-rejected pairs keep original fuzzy score

Auto-Config Classification Fixes

• ID patterns checked before phone/zip — SalesID no longer misclassified as "phone"
• Price/cost/amount patterns prevent zip misclassification — SalePrice no longer "zip"
• Identifier, date, geo, numeric classifications are authoritative over data profiling
• Utility-based fuzzy field ranking (cardinality × completeness × string length) replaces column-order truncation
• LLM-assisted column classification corrects ambiguous types when LLM boost is enabled

ANN Hybrid Blocking

• Oversized blocks that would be skipped now fall back to ANN sub-blocking via embeddings
• Embeds only unique text values (e.g., 61K records → 187 unique texts = seconds)
• 10x max_block_size cap prevents ANN on truly massive blocks
• Multi-pass blocking passes ANN config to static builder
• get_embedder() respects GPU routing (returns VertexEmbedder when mode=vertex)

Robust Error Handling

• Concurrent fut.result() guarded — thread exceptions no longer crash the pipeline
• Network errors (URLError, timeout, ConnectionError) retried with backoff
• API failures return empty results instead of false-negative labels
• ANN sub-block failures caught gracefully — skip block, don't crash pipeline
• VertexEmbedder import/init failures fall back to local embedder

Benchmark (Bulldozer 401K rows)

| Metric | v1.2.5 | v1.2.6 |
|---|---|---|
| Clusters | 28,703 | 27,937 |
| Records matched | 383,701 | 384,650 |
| LLM pairs scored | 37,500 | 200 |
| LLM cost | ~$0.50 | ~$0.01 |
| Confidence >= 0.4 | 77.1% | 87.7% |