This release adds 3 notable features for engineering teams evaluating rollout.
✓ No known CVEs patched in this version
Topics
+1 more
Summary
AI summaryUpdates What's New, SSE, and safetensors across a mixed release.
Full changelog
What's New
thaw serve — Pre-warmed Engine Pool
PgBouncer for GPU inference. Keep vLLM engines pre-initialized with dummy weights, then DMA-swap model
snapshots on demand (~1s instead of 20s cold start).
- OpenAI-compatible API (
/v1/completions,/v1/chat/completions) - Model affinity — zero swap cost when the requested model is already loaded
- Hot model registration via admin API (
/admin/pool,/admin/snapshots) - Streaming support (SSE)
Pre-built native wheels
thaw-native is now published to PyPI with CUDA 12.4 baked in. No more Rust toolchain on your GPU box:
pip install thaw-vllm[all]
Pure-Python restore fallback
restore_model_from_ram now copies region-by-region into existing GPU tensors when the Rust module
isn't available. No extra GPU memory allocation (fixes OOM on the previous fallback path).
Benchmarks (Llama-3.1-8B, A40)
| Metric | Value |
|--------|-------|
| Full cold start (safetensors) | 45.7s |
| thaw serve ready | 14.2s |
| DMA restore throughput | 11.6 GB/s |
| Weight restore time | 7.7s |
Weekly OSS security release digest.
The CVE patches and breaking changes that affected production tools this week. One email, every Sunday.
No spam, unsubscribe anytime.
Share this release
About Thaw
All releases →Related context
Related tools
Beta — feedback welcome: [email protected]