zenml

Known Issues

If the Docker credential helper contains invalid credentials for a registry that ZenML is trying to use, those will have precendence over the credentials set by the service connector and will cause the following error: RuntimeError: Docker error: denied: Your authorization token has expired. Reauthenticate and try again... This will be fixed in the next release, in the meantime docker logout <REGISTRY> should fix it.

Breaking Changes

• The minimum supported transformers version has been raised. If you use ZenML with Hugging Face/transformers, update your environment and dependency pins to a newer compatible transformers release before upgrading ZenML. PR #4976
• The Azure integration now requires newer Azure dependency versions, and support for the deprecated azureml-core library has been fully removed. If you use ZenML on Azure, update your Azure-related dependency pins and migrate any remaining azureml-core usage to the currently supported Azure SDK packages before upgrading. PR #4987
• In open-source ZenML server deployments without RBAC enabled, service account and API key management is now restricted to admins only. Non-admin users will no longer be able to manage service accounts or API keys they previously created, so move any required credentials and automation to admin-managed accounts as part of your upgrade. PR #5007

New integrations and execution backends

• Trackio experiment tracking: ZenML now includes a Trackio experiment tracker integration, allowing pipelines to log experiment data through Trackio’s public API. This makes it easier to manage trial results and connect ZenML runs with Hugging Face-backed Trackio workflows such as datasets, spaces, and buckets. PR #4841
• Backblaze B2 artifact store: You can now configure Backblaze B2 as a ZenML artifact store. This adds another S3-compatible storage option for teams that want to store pipeline artifacts in Backblaze infrastructure. PR #4791
• Baseten step operator: ZenML now supports a baseten step operator flavor for running GPU workloads as Baseten Training jobs. It supports regular single-node steps with ZenML artifacts and logs, as well as multi-node distributed training through command steps that can consume Baseten’s distributed training environment variables. PR #4973
• Generic OAuth2 service connector: A new OAuth2 service connector lets you authenticate external services using a static token, client credentials, or a client ID with refresh token. This provides a reusable connector option for services that expose OAuth2-based authentication. PR #4992

Workflow controls and platform operations

• Replay input overrides by step name: When replaying a run, you can now use step_default_input_overrides to override a step input for every invocation of a step with the same name. Per-invocation step_input_overrides still take precedence, giving you both broad and targeted control during replay. PR #4978
• Trigger cycle protection: ZenML now detects execution loops in Platform Event Trigger chains at the pipeline level. Cyclic trigger dispatches are skipped with the new SKIPPED_TRIGGER_CYCLE status, while unrelated downstream dispatches can continue normally and the affected cycle can be inspected through the SDK. PR #4971
• Optional sandbox cleanup on exit: Sandbox sessions can now be configured to destroy the sandbox automatically when the session exits. The option defaults to False, preserving the behavior from previous releases unless you opt in. PR #4986
• Configurable Kubernetes API retries: Kubernetes-based deployments can now configure retry behavior for Kubernetes API calls. This gives operators more control over resilience in clusters where transient API failures or throttling can occur. PR #5004
• Dashboard filtering and connector selection improvements: The dashboard Timeline View now has additional filtering options, including more status filters. Component creation also gets a more efficient connector selector, making setup flows smoother in larger workspaces. PR #1084

Performance and scalability

• Faster pipeline sorting by latest run: Listing pipelines sorted by latest run is now more efficient on large deployments. ZenML changed the query shape and supporting database indexing so the server no longer has to scan all runs for all pipelines just to compute the latest run timestamp. PR #4969
• More efficient run and artifact queries: Several common server queries now load only the data they need and fetch related metadata more efficiently. This improves performance for DAG, pipeline run, step run, artifact version, and model version views, especially in workspaces with many entities. PR #4994
• Catch-up cleanup for expired API transactions: Expired API transaction cleanup now works through bounded catch-up passes instead of a single fixed delete per interval. This helps servers recover from cleanup backlogs while keeping each database operation bounded, and also allows completed expired idempotency transactions to be safely reclaimed. PR #4943

• Artifact store caching prevents server OOMs: The ZenML server now caches artifact store instances used for operations such as reading logs and visualizations. This avoids repeatedly rebuilding heavy storage clients and helps prevent memory growth that could previously lead to OOM kills on busy servers. PR #4974
• Docker credentials for image builds and pushes: ZenML now configures Docker credentials correctly when building and pushing container images. This fixes cases where username/password credentials were not applied to the expected registry URI, which could cause authentication failures in build and push workflows. PR #5005
• Secret backup and restore authorization: Secret backup and restore endpoints now enforce an explicit admin check when RBAC is disabled. This closes an authorization gap where an authenticated non-admin user could access admin-only secret operations in the default non-RBAC setup. PR #5009
• Safer custom flavor loading: Custom flavor sources are now validated before server-side hydration. ZenML ensures the configured source resolves to a Flavor subclass before instantiation, preventing arbitrary zero-argument callables from being invoked during flavor loading. PR #5008

What's Changed

• Add version 0.95.0 to legacy docs by @github-actions[bot] in https://github.com/zenml-io/zenml/pull/4967
• Improve Modal orchestrator docs by @strickvl in https://github.com/zenml-io/zenml/pull/4970
• Bump the minor-and-patch group with 2 updates by @dependabot[bot] in https://github.com/zenml-io/zenml/pull/4962
• Improve listing pipelines sorted by latest run by @schustmi in https://github.com/zenml-io/zenml/pull/4969
• Fix CI-blocking dead docs links and transformers version floor by @htahir1 in https://github.com/zenml-io/zenml/pull/4976
• Bump anthropics/claude-code-action from 1.0.143 to 1.0.150 in the minor-and-patch group by @dependabot[bot] in https://github.com/zenml-io/zenml/pull/4981
• Add Trackio Integration for ZenML by @ParagEkbote in https://github.com/zenml-io/zenml/pull/4841
• Document multi-node distributed training via CommandStep by @htahir1 in https://github.com/zenml-io/zenml/pull/4980
• Fix databricks test leaking the global custom source root by @htahir1 in https://github.com/zenml-io/zenml/pull/4979
• Cache artifact store in server to avoid OOM kills by @schustmi in https://github.com/zenml-io/zenml/pull/4974
• Make the docs agent-aware: Agents guide in Learn, Kitaru routing across spaces by @htahir1 in https://github.com/zenml-io/zenml/pull/4897
• Extract shared Modal settings/credentials mixins by @htahir1 in https://github.com/zenml-io/zenml/pull/4954
• Add Backblaze B2 artifact store flavor by @goanpeca in https://github.com/zenml-io/zenml/pull/4791
• Restore the source-root env var in the databricks test isolation fixture by @htahir1 in https://github.com/zenml-io/zenml/pull/4985
• Add Missing Doc Links for Trackio by @ParagEkbote in https://github.com/zenml-io/zenml/pull/4989
• Honesty pass on Replay and improve docs (drop non-existent cohort/Recipe/diff APIs) by @htahir1 in https://github.com/zenml-io/zenml/pull/4984
• Misc DB improvements by @schustmi in https://github.com/zenml-io/zenml/pull/4982
• Add option to destroy sandbox on session exit by @schustmi in https://github.com/zenml-io/zenml/pull/4986
• Add Baseten step operator (single-node + multi-node distributed training) by @htahir1 in https://github.com/zenml-io/zenml/pull/4973
• Improved snapshot fetching by @schustmi in https://github.com/zenml-io/zenml/pull/4991
• Allow overwriting step inputs for all invocations when replaying by @schustmi in https://github.com/zenml-io/zenml/pull/4978
• Add isolated step cleanup hooks by @schustmi in https://github.com/zenml-io/zenml/pull/4990
• Update azure dependencies by @Json-Andriopoulos in https://github.com/zenml-io/zenml/pull/4987
• Feature/prevent trigger execution loops by @Json-Andriopoulos in https://github.com/zenml-io/zenml/pull/4971
• More query improvements by @schustmi in https://github.com/zenml-io/zenml/pull/4994
• Configurable Kubernetes API retries by @schustmi in https://github.com/zenml-io/zenml/pull/5004
• Async snapshot execution by @Json-Andriopoulos in https://github.com/zenml-io/zenml/pull/5003
• Fix configuration of docker credentials for building/pushing container images by @stefannica in https://github.com/zenml-io/zenml/pull/5005
• Bump actions/checkout from 6.0.3 to 7.0.0 by @dependabot[bot] in https://github.com/zenml-io/zenml/pull/5002
• Bump the minor-and-patch group across 1 directory with 3 updates by @dependabot[bot] in https://github.com/zenml-io/zenml/pull/5000
• Bump actions/cache from 5.0.5 to 6.1.0 by @dependabot[bot] in https://github.com/zenml-io/zenml/pull/5001
• OAuth2 service connector by @schustmi in https://github.com/zenml-io/zenml/pull/4992
• Changed default otel-service-name as workspace name for zenml-pro workspaces by @amitvikramraj in https://github.com/zenml-io/zenml/pull/5006
• Add catch-up cleanup for expired API transactions by @safoinme in https://github.com/zenml-io/zenml/pull/4943
• Add admin permissions check to the secret backup/restore endpoints by @stefannica in https://github.com/zenml-io/zenml/pull/5009
• Validate custom flavors at loading time by @stefannica in https://github.com/zenml-io/zenml/pull/5008
• Enforce admin-only permissions for service account management in non-RBAC deployments by @stefannica in https://github.com/zenml-io/zenml/pull/5007
• Prepare release 0.96.0 by @github-actions[bot] in https://github.com/zenml-io/zenml/pull/5010

New Contributors

• @ParagEkbote made their first contribution in https://github.com/zenml-io/zenml/pull/4841
• @goanpeca made their first contribution in https://github.com/zenml-io/zenml/pull/4791

Full Changelog: https://github.com/zenml-io/zenml/compare/0.95.1...0.96.0