What's new in 0.2.0

Dataset

• CSV support via Dataset.from_csv() — default column names input and ideal, with input_field/output_field overrides for custom schemas
• Arbitrary JSONL field mapping via --input-field / --output-field CLI flags and Python API
• Label-free evaluation — datasets without reference answers work end-to-end; reference-based metrics raise a clear error upfront

Metrics

• Multi-dimensional LLM-as-judge via the dimensions parameter — score multiple criteria (e.g. fluency, accuracy, safety) in a single judge call

v0.1.0 — Initial release

aevyra-verdict is a framework for evaluating and comparing LLM outputs across models and providers.

What's included

Core evaluation engine

• Run completions concurrently across any combination of models and providers
• Configurable concurrency, retries, and exponential backoff for rate limit handling
• Structured results with per-model scores, latency, and token usage

Providers

• Built-in support for OpenAI, Anthropic, Google Gemini, Mistral, Cohere, and OpenRouter
• Local model support via any OpenAI-compatible API (Ollama, vLLM)

Metrics

• Reference-based: ROUGE, BLEU, exact match
• LLM-as-judge with configurable criteria and custom prompt templates
• Custom scoring functions via Python callables

Dataset formats

• Auto-detection of OpenAI, ShareGPT, and Alpaca formats
• Filtering by metadata fields

CLI

• aevyra-verdict run — run evals and print comparison table
• aevyra-verdict inspect — preview a dataset
• aevyra-verdict providers — check which API keys are configured

Docs

• Full documentation at aevyra.mintlify.app/verdict/introduction

Install

pip install aevyra-verdict