Skip to content
EleutherAI / Lm-Evaluation-Harness
AI Coding Tools
A unified framework for evaluating generative language models on over 60 academic benchmarks and custom tasks
Python
·
Latest v0.4.12 · 2mo ago
Security brief →
Features
-
Supports >60 standard LLM evaluation benchmarks with hundreds of subtasks
-
Works with major model backends: HuggingFace transformers, vLLM, GPT‑NeoX, Megatron‑DeepSpeed, and commercial APIs (OpenAI, TextSynth)
-
Allows quantization, adapters (LoRA), prompt customization via Jinja2, and flexible config‑based task creation
No immediate action
v0.4.12
Breaking risk
·
SteeredModel rename + vLLM bump + thinking flag change
Review required
v0.4.11
Maintenance
·
RCE / SSRF
Dependencies
Routine maintenance and dependency updates.
Review required
v0.4.10
Breaking risk
·
Breaking upgrade
Optional backend installation
Review required
v0.4.9.2
Breaking risk
·
Breaking upgrade
Python 3.10 minimum
No immediate action
v0.4.9.1
Breaking risk
·
New benchmarks + TruthfulQA
Weekly OSS security release digest.
The CVE patches and breaking changes that affected production tools this week. One email, every Sunday.
No spam, unsubscribe anytime.
About
Languages
Python
·
Shell
·
C++
Search tools, categories, lists, and users
Use ↑↓ to navigate, Enter to open, Esc to close
No results for ""
⌘K to open
↑↓ navigate
⏎ open
