Skip to content

UQLM

v0.5.4 Feature

This release adds 2 notable features for engineering teams evaluating rollout.

Published 4mo AI Agents & Assistants
✓ No known CVEs patched
Read the diff → Tool health → What is this tool? →

✓ No known CVEs patched in this version

Topics

ai-evaluation ai-safety confidence-estimation confidence-score hallucination hallucination-detection
+8 more
hallucination-evaluation hallucination-mitigation llm llm-evaluation llm-hallucination llm-safety uncertainty-estimation uncertainty-quantification

Summary

AI summary

Adds multiple new white‑box uncertainty scorers to UQEnsemble and corrects embedding model specification.

Full changelog

Highlights

1. Add new white-box scorers to UQEnsemble accepted scorers list:

Top-logprobs scorers (3):

  • min_token_negentropy - Minimum negentropy across tokens
  • mean_token_negentropy - Average negentropy across tokens
  • probability_margin - Mean difference between top-2 token probabilities

Sampled-logprobs scorers (4):

  • semantic_negentropy - Entropy based on semantic clustering
  • semantic_density - Density-based confidence measure
  • monte_carlo_probability - Average sequence probability across samples
  • consistency_and_confidence - Cosine similarity × response probability

P(True) scorer (1):

  • p_true - LLM's estimate of P(response is true)

2. Fix embeddings model specification for cosine_sim and consistency_and_confidence, enable with WhiteBoxUQ

Corrects a string error in embedding model specification with sentence_transformer parameter of BlackBoxUQ. Previously, the string was forced to begin with "sentence_transformers" but now the full string is specified with the parameter.

Previous: sentence_transformer=all-MiniLM-L12-v2 was specified and then "sentence-transformers/" was prepended to the string when storing the class attribute.

Now: sentence_transformer=sentence-transformers/all-MiniLM-L12-v2 is specified. This allows other embeddings models that don't start with "sentence_transformers/", such as jinaai/jina-embeddings-v2-base-code to be specified.

Also adds missing sentence_transformer parameter for WhiteBoxUQ

What's Changed

  • v0.5.3 updates by @dylanbouchard in https://github.com/cvs-health/uqlm/pull/327
  • Fix embedding model specification by @dylanbouchard in https://github.com/cvs-health/uqlm/pull/332
  • Enable use of new white-box scorers in UQEnsemble by @dylanbouchard in https://github.com/cvs-health/uqlm/pull/333
  • Feature/enable all white box scorers by @kaushik-42 in https://github.com/cvs-health/uqlm/pull/328
  • Patch release: v0.5.4 by @dylanbouchard in https://github.com/cvs-health/uqlm/pull/334

Full Changelog: https://github.com/cvs-health/uqlm/compare/v0.5.3...v0.5.4

View diff on GitHub

Weekly OSS security release digest.

The CVE patches and breaking changes that affected production tools this week. One email, every Sunday.

No spam, unsubscribe anytime.

Share this release

Share on X Share on Bluesky

Track UQLM

Get notified when new releases ship.

Sign up free

About UQLM

All releases →

Related context

Beta — feedback welcome: [email protected]