This release adds 2 notable features for engineering teams evaluating rollout.
✓ No known CVEs patched in this version
Topics
+14 more
Summary
AI summaryCross‑partition cluster‑id collision fixed, unlocking driver‑collect‑free distributed deduplication of 100 M rows.
Changes in this release
| Type | Severity | Summary | CVE |
|---|---|---|---|
| Feature | Medium |
Adds driver-collect-free end-to-end distributed deduplication pipeline. Adds driver-collect-free end-to-end distributed deduplication pipeline. Source: llm_adapter@2026-06-04 Confidence: high |
— |
| Feature | Medium |
Adds local_cc_assignments using per-partition Union-Find for connected components. Adds local_cc_assignments using per-partition Union-Find for connected components. Source: llm_adapter@2026-06-04 Confidence: high |
— |
| Feature | Medium |
Adds distributed join and write for golden records without driver materialization. Adds distributed join and write for golden records without driver materialization. Source: llm_adapter@2026-06-04 Confidence: high |
— |
| Bugfix | High |
Fixes cross-partition __row_id__ collision causing incorrect cluster merging. Fixes cross-partition __row_id__ collision causing incorrect cluster merging. Source: llm_adapter@2026-06-04 Confidence: high |
— |
Full changelog
GoldenMatch 1.26.0 - distributed pipeline scales to 100M
Verified on a real 4-worker GCP Ray cluster (e2-standard-16, 64 worker CPU):
a full 100,000,000-row dedupe in 213 s wall, 20,000,000 golden records,
driver process peak 0.30 GB RSS.
The distributed pipeline (GOLDENMATCH_DISTRIBUTED_PIPELINE=2) is now
driver-collect-free end to end:
score -> per-partition local connected-components -> distributed join
-> distributed golden build -> distributed write
Nothing funnels back to a single node, which is what unlocks 100M.
Added
- local_cc_assignments: connected components via a single per-partition
local Union-Find map_batches. Distributed scoring is per-partition, so a
component's edges are always co-located in one block; a local Union-Find
yields the global components with no cross-node merge, no driver collect,
no iterative graph algorithm. - End-to-end distributed pipeline: rows annotated with cluster_id via a
distributed Dataset.join (not a broadcast dict); golden records built AND
written distributed (build_golden_records_distributed + write_parquet),
never materialized on the driver.
Fixed
- Cross-partition cluster-id collision: the pipeline synthesized row_id
per-partition, so ids collided across partitions and connected-components
silently merged unrelated clusters (a 50M run reported ~156K clusters vs
the true ~10M). The generator now carries a global row_id, respected
by the pipeline.
See CHANGELOG.md and scripts/bench_phase5_explicit.py.
Weekly OSS security release digest.
The CVE patches and breaking changes that affected production tools this week. One email, every Sunday.
No spam, unsubscribe anytime.
Share this release
About GoldenMatch
All releases →Related context
Related tools
Earlier breaking changes
- v3.1.0 `GOLDENMATCH_FRAME=polars` now requires the `[polars]` extra; raises error without it.
- vgoldencheck-v3.0.0 `inferred_type` emits neutral dtype vocabulary (str/int/uint/float/date/datetime/bool/other) instead of raw Polars dtypes.
- vgoldencheck-v3.0.0 'inferred_type' now emits a neutral dtype vocabulary instead of raw Polars dtype strings.
- vgoldencheck-v3.0.0 `scan_file`, `scan_dataframe`, and CLI `check` now run without Polars, using Arrow-native pyarrow.Table.
- v3.0.0 Result frames now return pyarrow.Table instead of Polars DataFrame.
Beta — feedback welcome: [email protected]