Hollow

v1.3.3 — Agent Checkpoints and Replay

Agents fail mid-task. SIGPAUSE suspends but doesn't capture full state for restoration on a different worker. This release adds checkpoint primitives: save everything, restore everything, resume exactly where you left off — including across process restarts.

What was added

Checkpoint system (agents/checkpoint.py)

AgentCheckpoint captures:

• Memory heap (all keys, content, priority, TTL, compression state)
• Unread inbox messages at checkpoint time
• Current task snapshot (task_id, status, partial output if any)
• Agent state (usage, metadata, locks)
• context_window_hash — SHA-256 of the heap snapshot for fast equality checks

CheckpointManager methods:

• save(agent_id, label) → checkpoint_id — persists to /agentOS/memory/checkpoints/{agent_id}/{id}.json
• restore(agent_id, checkpoint_id) → bool — overwrites heap and agent metadata
• list_checkpoints(agent_id) → newest-first list with metadata
• diff(chk_a, chk_b) → new/removed/changed memory keys, new inbox messages, agent state changes
• replay(checkpoint_id, task_description, n_runs) → ReplayResult with consistency_score and divergence_points

Auto-checkpointing (three triggers)

• Before transaction commit: every agent in the transaction is checkpointed before ops are applied. If rollback is needed, each agent can be independently restored to its pre-commit state
• On SIGPAUSE: checkpoint fires before status flip to suspended; checkpoint_id stored in agent metadata and included in the SIGPAUSE inbox message
• After tasks >30s: long-running tasks auto-checkpoint the submitting agent on completion

API (5 new endpoints)

• POST /agents/{id}/checkpoint — save, returns checkpoint_id
• POST /agents/{id}/restore/{checkpoint_id} — restore
• GET /agents/{id}/checkpoints — list (newest first)
• GET /checkpoints/{a}/diff/{b} — diff
• POST /checkpoints/{id}/replay — replay task N times, measure consistency

MCP tools (4 new)

• agent_checkpoint — save checkpoint
• agent_restore — restore from checkpoint
• checkpoint_diff — diff two checkpoints
• checkpoint_replay — replay task from checkpoint, returns consistency_score

New integration tests (tests/integration/test_checkpoint.py)

1. Save and restore — alloc 5 objects, checkpoint, clear, restore, assert heap identical
2. SIGPAUSE auto-checkpoint — SIGPAUSE fires checkpoint with label 'sigpause'
3. Checkpoint diff — 3 objects between A and B, diff shows exactly 3 new keys; self-diff is empty
4. Replay consistency — factual question 5×, consistency_score > 0.80 (Ollama required, auto-skip)
5. Replay divergence — ambiguous task, verify infrastructure reports score and divergence_points
6. Checkpoint persistence — save, list, assert metadata accurate, restore succeeds

Built on

• Working Memory Heap (v1.0.0): heap snapshot and restore
• Multi-Agent Transactions (v1.2.0): pre-commit auto-checkpoint for all transaction participants

Stats

• MCP tools: 71 → 75
• Integration tests: 82 → 88