yakew7/Fair-Code

Fair Code v2.0.0 — Cross-Domain Fairness Benchmark Harness

Fair Code moves from seven bespoke bias audits to a benchmark harness that runs one uniform, reproducible pipeline over all of them — the core of what a research paper built on this repo would cite. Nothing about the existing audits (unfair.py / fair.py, the website, the explainers) changed or broke; this release is additive.

🎯 Cross-Domain Benchmark Harness

• audit.yaml manifests — every audit folder now carries a declarative manifest naming its label column, protected attributes, proxy features, and core (fair) feature set. Schema: faircode/MANIFEST_SPEC.md.
• Five mitigation strategies (S0–S4), run uniformly across all seven domains:
  • baseline → unawareness → unawareness_proxy_removal (the existing fair.py method) → in_processing (Fairlearn ExponentiatedGradient) → post_processing (Fairlearn ThresholdOptimizer)
• Three model families (logistic regression, random forest, gradient boosting) × six fairness metrics (demographic parity, disparate impact ratio, equal opportunity, equalized odds, predictive parity, accuracy equality) — each with a bootstrap CI and permutation-test p-value — plus accuracy/AUC/F1 and intersectional gaps.
• New CLI: faircode benchmark (optional faircode[benchmark] extra). Writes results_fairness.csv, results_performance.csv, summary.csv, and 300-dpi figures to results/.

🔁 Reproducibility & Paper Freeze

• All randomness pinned to random_state: 42 across every model, split, bootstrap, and permutation — verified, not assumed.
• requirements-lock.txt — exact environment (Python 3.13.2, scikit-learn 1.8.0, fairlearn 0.14.0, pandas 3.0.2).
• scripts/freeze_paper_results.py — snapshots results/ into paper/results-frozen/ with a MANIFEST.md recording the git commit, package versions, and the exact set of audits included, so a citation points at a defined, reproducible snapshot instead of "whatever was in the repo that week."

✅ Tests & CI

• 50 new tests (108 total, ~10s) covering all six fairness metrics against hand-computed examples, manifest validation, S0–S4 strategy correctness, and a genuine end-to-end harness run.
• New benchmark-harness CI job runs the suite plus a real CLI smoke test on one small audit. The full seven-domain sweep stays out of CI by design — it's run locally and the output committed.

🐛 Fixed

• ThresholdOptimizer (S4) was calibrating decision thresholds on the same rows its base model was trained on, producing a near-zero gap in-sample but +0.22 out-of-sample on German Credit Lending. Fixed with a proper fit/calibration split — residual gap now a defensible +0.07.
• All 14 fair.png / unfair.png screenshots regenerated from fresh runs — the old images predated the bootstrap CI / permutation-test output and only showed the bare headline gap. Every underlying number was independently re-verified against the live site (all matched exactly).

📈 The headline result (COMPAS)

Demographic parity gap across the five strategies: 86% → 18% → 12% → 1.4% → 2.4%. The two Fairlearn-based strategies land within a couple points of simple proxy removal — evidence for a residual fairness floor, not a limitation of any one method.

Full changelog: CHANGELOG.md § 2.0.0