GoldenMatch

Ray Backend for Large-Scale Entity Resolution

GoldenMatch now scales to 10M+ records with an optional Ray backend.

pip install goldenmatch[ray]
goldenmatch dedupe huge_dataset.parquet --backend ray

Or in config:

backend: ray

Zero configuration needed. Ray auto-starts locally and uses all CPU cores. For datasets over 50M records, point at a Ray cluster.

| Records | Backend | User Setup |
|---------|---------|------------|
| < 500K | Polars (default) | Nothing |
| 500K - 2M | DuckDB or Ray local | pip install goldenmatch[ray] |
| 2M - 50M | Ray local | Same |
| 50M+ | Ray cluster | User provides infra |

How it works

Block scoring (the bottleneck) is distributed as independent Ray tasks. Each block is scored on a separate CPU core. Results are collected and merged into the pipeline seamlessly.

For small workloads (<= 4 blocks), falls back to the existing ThreadPoolExecutor automatically.

Stats

• 909 tests passing (6 new)
• CI green on Python 3.11/3.12/3.13

pip install --upgrade goldenmatch