Skip to content

Release history

Bencher releases

Suite of continuous benchmarking tools designed to catch performance regressions in CI.

Back to tool Latest release

All releases

16 shown

Review required
v0.6.6 Breaking risk
Auth

Rename usage to metrics

Open
v0.6.5 Bug fix
Notable features
  • Project scoped API keys
  • Build time and file size tracking for bare metal runs
Full changelog
  • Add project scoped API keys
  • Add build time and file size tracking support for bare metal runs
  • Fix bencher run hanging on stdin when no command is provided, including for --image remote bare metal runs (Thank you @sophie-h)
View release on GitHub
v0.6.4 Breaking risk
Breaking changes
  • Switched runner key format from hex encoding (79 characters) to alphanumeric encoding (45 characters); existing keys must be rotated.
Full changelog
  • Allow a minimum sample size of 1 for the Percentage (percentage) Threshold Test (Thank you @teofr)
  • BREAKING CHANGE Switch runner key format from hex to alphanumeric encoding, reducing key length from 79 to 45 characters; existing runner keys must be rotated
View release on GitHub
v0.6.3 Breaking risk
Breaking changes
  • Rename `runner token` to `runner key` across the API, CLI, and environment variables
Notable features
  • Add API token revocation with `bencher token revoke` and Revoke button in the Console
Full changelog
  • Fix sample standard deviation for Z-score, t-test, and Log Normal Thresholds to use Bessel's correction (divide by n − 1) (Thank you @ggirol-rc)
  • Fix t-test Threshold to use the prediction interval (scale the standard deviation by √(1 + 1/n)) so the boundary tests whether a new Metric is consistent with the historical distribution rather than whether it contains the historical mean (Thank you @ggirol-rc)
  • Existing Thresholds will produce slightly wider (statistically correct) boundaries; no configuration changes required
  • Fix Alert JSON to return the Spec for the Report in which the Alert was generated instead of the Testbed's current Spec
  • BREAKING CHANGE Rename runner token to runner key across the API, CLI, and environment variables
  • Fix race condition where concurrent branch creation could cause a "branch head is null" 500 error (Thank you @vmelamed)
  • Self-heal legacy Branches whose head_id is still NULL from the race condition above so subsequent bencher run submissions succeed instead of 500ing (Thank you @vmelamed)
  • Add API token revocation with bencher token revoke and Revoke button in the Console
View release on GitHub
v0.6.2 Bug fix

Fix URL fragment ordering per RFC 3986 and fallback for missing --start-point-hash.

Full changelog
  • Allow unauthenticated docker push to unclaimed projects
  • Fix URL fragment placed before query string per RFC 3986 (Thank you @ricardoboss)
  • Fix --start-point-hash failure on non-existent hash by falling back to the start point branch's latest version (Thank you @teofr)
View release on GitHub
v0.6.1 Breaking risk
⚠ Upgrade required
  • --poll-interval is deprecated as an alias for --job-poll-interval in bencher run
Notable features
  • Added `idempotency_key` to `bencher run` for deduplicating retried submissions
  • Added `--detach` flag to `bencher run` to skip waiting for remote job completion
  • Configurable S3 chunk size via `registry.data_store.chunk_size` (default 5 MB) for OCI uploads
Full changelog
  • Buffer OCI upload network frames into configurable S3 chunks (registry.data_store.chunk_size, default 5 MB) to reduce S3 operations during Docker push
  • Add idempotency_key to bencher run to deduplicate retried submissions
  • Rename --poll-interval to --job-poll-interval for bencher run (--poll-interval is still supported as a deprecated alias)
  • Add --detach flag to bencher run to skip waiting for remote job completion
View release on GitHub
v0.6.0 Breaking risk
⚠ Upgrade required
  • Use positional project argument instead of deprecated flag/env var for `bencher threshold create`
  • Specify only a single start point in `bencher run`
  • Replace removed language-level adapter options with specific tool adapters (`rust_bench`, `cpp_google`, etc.) or `magic`
Breaking changes
  • Update BMF to accept UUID, slug, or name for Benchmark identifier
  • Move Organizations and Projects to soft deletion with admin-only hard delete
  • Remove `POST /v0/server/restart` endpoint and CLI `bencher server restart` command
Notable features
  • Add OCI compliant image registry API
  • Add bare metal `runner`
  • [MemoryDiagnoser] support to C# DotNet Adapter
Full changelog
  • BREAKING CHANGE Update Bencher Metric Format (BMF) to accept UUID, slug, or name for the Benchmark identifier
  • Move to soft deletion for Organizations and Projects with admin-only hard deletion
  • Add an OCI compliant image registry API
  • Add a bare metal runner
  • BREAKING CHANGE Remove the POST /v0/server/restart endpoint and CLI bencher server restart command
  • BREAKING CHANGE Remove the PUT /v0/server/config endpoint and CLI bencher server config update command
  • BREAKING CHANGE Remove the deprecated --project flag and BENCHER_PROJECT environment variable from CLI bencher threshold create command (use the positional project argument instead)
  • BREAKING CHANGE Remove deprecated support for specifying multiple --start-point values for CLI bencher run (only a single start point is now accepted)
  • BREAKING CHANGE Remove language-level adapter options (rust, cpp, go, java, c_sharp, js, python, ruby, shell, dart) from the CLI and API; use the specific tool adapter (e.g. rust_bench, cpp_google) or magic instead
  • Improve server concurrency with database connection pooling
  • Add [MemoryDiagnoser] support to the C# DotNet Adapter (Thank you @ricardoboss)
  • Add Dart benchmark_harness adapter (dart_benchmark_harness) (Thank you @Louisvranderick)
View release on GitHub
v0.5.10 Bug fix

Fixed GitHub PR comment truncation exceeding maximum length.

Full changelog
  • Fix GitHub PR comment exceeding maximum length (Thank you @alexeyr-ci2)
  • Improve GitHub API error messages
  • Improve developer onboarding docs (Thank you @Xenira)
  • Update GitHub Actions docs (Thank you @pgoslatara)
View release on GitHub
v0.5.9 Feature
Notable features
  • Add Single Sign-On (SSO) API (Bencher Plus Enterprise feature)
Changelog
  • Add Single Sign-On (SSO) API (Bencher Plus Enterprise feature)
View release on GitHub
v0.5.8 Bugfix

Fixed Bencher Console for non‑Bencher Plus builds.

Full changelog
  • Allow the BenchmarkDotNet namespace to be optional (Thank you @ntransw)
  • Fix Bencher Console for non-Bencher Plus builds (Thank you @flokli)
View release on GitHub
v0.5.7 Bugfix

Fixed workspace-level feature unification for non‑Bencher Plus builds.

Full changelog
  • Fix workspace level feature unification for non-Bencher Plus builds (Thank you @flokli)
View release on GitHub
v0.5.6 Breaking risk
Breaking changes
  • Renamed configuration key/flag `rust_iai_callgrind` to `rust_gungraun`
Full changelog
  • Fix Iai-Callgrind adapter parsing bug (Thank you @gamma0987)
  • Add alias for Iai-Callgrind to Gungraun rename, (rust_iai_callgrind -> rust_gungraun)
View release on GitHub
v0.5.5 Feature
Notable features
  • Login with Google authentication option added
  • --file flag can be specified multiple times to track several result files
Full changelog
  • Add Login with Google as an authentication option
  • Allow the --file option to be specified multiple times to track multiple result files
View release on GitHub
v0.5.4 Breaking risk
Breaking changes
  • Creating an on-the-fly project when CI=true now requires the --ci-on-the-fly flag.
Full changelog
  • Fix fork pull request example (Thank you @ruyadorno)
  • Improve database backup to run in the background
  • Improve server stats to run in the background
  • Add branch head version to the Console UI
  • BREAKING CHANGE Creating an on-the-fly project when the CI environment variable is set to true will now require the --ci-on-the-fly flag (Thank you @jaqx0r)
View release on GitHub

Beta — feedback welcome: [email protected]