olgasafonova/productplan-mcp-server

v4.3.0 Feature

This release adds 2 notable features for engineering teams evaluating rollout.

Published 5mo MCP Developer Tools

View tool

✓ No known CVEs patched

Read the diff → Tool health → What is this tool? →

✓ No known CVEs patched in this version

Topics

anthropic claude go mcp mcp-server model-context-protocol

+4 more

okr product-management productplan roadmap

Summary

AI summary

Added an evaluation framework for testing LLM tool selection accuracy in ProductPlan MCP.

Full changelog

What's New

MCP Evals Framework for Tool Selection Testing

Added a comprehensive evaluation framework to validate LLM tool selection accuracy with ProductPlan MCP tools.

Test Suites

| Suite | Tests | Description |
|-------|-------|-------------|
| Tool Selection | 69 | Tests correct tool selection for natural language prompts |
| Confusion Pairs | 40 | Tests disambiguation between commonly confused tools |
| Argument Correctness | 25 | Tests argument extraction from prompts |
| Total | 134 | |

Usage

import "github.com/olgasafonova/productplan-mcp-server/evals"

// Load all test suites
toolSel, confPairs, args, _ := evals.LoadAllEvals("evals/")

// Run with your LLM selector
metrics, _ := evals.EvaluateToolSelection(toolSel, yourSelector)
fmt.Println(evals.FormatMetrics(metrics, "Tool Selection"))

Files Added

evals/runner.go - Evaluation framework
evals/runner_test.go - Unit tests
evals/tool_selection.json - 69 tool selection tests
evals/confusion_pairs.json - 10 pairs with 40 disambiguation tests
evals/argument_correctness.json - 25 argument extraction tests
evals/README.md - Documentation

Full Changelog: https://github.com/olgasafonova/productplan-mcp-server/compare/v4.2.0...v4.3.0

View diff on GitHub

Weekly OSS security release digest.

The CVE patches and breaking changes that affected production tools this week. One email, every Sunday.

No spam, unsubscribe anytime.

Share this release

Share on X Share on Bluesky

Track olgasafonova/productplan-mcp-server

Get notified when new releases ship.

About olgasafonova/productplan-mcp-server

Query ProductPlan roadmaps. Access OKRs, ideas, launches, and timeline data.

All releases →