Local Deep Research

New search engine, settings-API secret leak fixes, and a batch of security hardening

Security fixes

Several defense-in-depth gaps were closed this release — all scoped to a user's own data, but worth upgrading for:

• GET /settings/api/bulk could return your configured API keys in plaintext: namespace requests (e.g. keys[]=llm) skipped redaction of nested *.api_key values, and keys[]=% could dump the whole settings table via an unescaped LIKE (#5028). Non-classic secret names like client_secret, bearer_token, and api_secret are now also redacted by the bulk endpoint (#5038).
• User-supplied strings reaching SQL LIKE queries (library search/domain filters, news subscription filters, and more) now escape % and _, so searches for values like 100% or a_b match literally instead of acting as wildcards (#3094).
• Exception text returned by /api/v1/quick_summary and /api/v1/analyze_documents is now scrubbed at the HTTP boundary, so error payloads can no longer echo credentials or stack-trace fragments to API clients (#5032).
• All LLM provider, embedding provider, and search engine modules now log through the diagnose-gated secure-logging wrapper — production logs no longer receive full tracebacks (which can echo API keys) from provider/engine failures unless diagnose mode is explicitly enabled (#4976).
• Registration/login no longer leave you silently half-logged-in when post-authentication setup fails partway; the partial session is rolled back so the reported failure matches reality (#5006).

Bug fixes

• SearXNG titles no longer lose their spaces — multi-word result titles were rendering as WordOneWordTwoWordThree in the ## Sources block of saved reports; snippets had the same issue. This also improves relevance filtering and candidate extraction on SearXNG-sourced collections, which previously saw single-token titles (#5022).
• Citation labels for library/RAG sources no longer render as empty [] links — relative URLs like /library/document/<uuid> now fall back to a slugified document title so the citation carries the document name (#4880). Web citations are unchanged.
• The langgraph-agent strategy now shows 📖 Reading the page: "<url>" when the agent fetches a URL, instead of the misleading 🔍 Searching Fetch Content (#5011). It also caps subtopic research at the documented limit of 5 and, when the model requests more, names the dropped subtopics in logs, the UI, and the tool reply so uninvestigated topics don't get cited (#5013).
• The Research Logs panel now keeps old warnings and errors when its 500-entry cap is hit, dropping routine info entries first (#4900), and library sizes of 1 TB or more no longer display as "1 undefined" (#4982).
• The news subscription-history endpoint no longer errors for subscriptions with research runs (#3094).

Under the hood, importing the citation formatter and url_classifier no longer drags in the full fetch stack (requests, playwright, bs4, …) thanks to lazy ContentFetcher resolution (#5023, #5041), and the release pipeline now pins PyPI builds and git tags to the exact gated release commit so published wheels can never drift ahead of the tag (#5026).

🔒 Security

• User-supplied strings that reach SQL LIKE queries — library search and domain filters, domain-classifier sampling, and news subscription filters — now escape % and _ wildcards. Previously a value like 100% or a_b was interpreted as a wildcard and matched unrelated rows; searches and filters now match literally, closing a wildcard-injection/enumeration vector within a user's own database. (#3094)
• All LLM provider, embedding provider, and web search engine modules now log through the diagnose-gated security.secure_logging wrapper, so exception tracebacks in these paths only appear in logs when diagnose mode is explicitly enabled (LDR_APP_DEBUG + LDR_LOGURU_DIAGNOSE) — production logs no longer receive full tracebacks that could echo API keys or request internals from provider/engine failures. The check-sensitive-logging pre-commit hook now enforces this by construction: raw loguru imports and every known bypass route (logger.opt(), @logger.catch, private logger handles, traceback/sys.exc_info() interpolation, dynamic loguru/traceback imports, logger rebinding, and getattr dodges) are rejected in these directories, bind()/patch() chain messages get the same exception-scrubbing checks as direct logger calls, and exception detail reached through attribute or subscript access (e.args, e.response.text, e.strerror) is flagged like str(e) itself. (#4976)
• Fixed a plaintext-secret disclosure in GET /settings/api/bulk: a namespace request (e.g. keys[]=llm) returned nested *.api_key values unredacted because the redaction check looked only at the outer requested key, and keys[]=% could dump the whole settings table via an unescaped LIKE. Redaction now recurses into setting subtrees and the key lookup escapes LIKE wildcards. (#5028)
• GET /settings/api/bulk now redacts password-typed settings whose leaf name is a non-classic secret token (e.g. client_secret, secret_key, bearer_token, api_secret, app_secret). The bulk endpoint redacts by key name only, so these were previously shipped in the clear while the singular GET masked them. (#5038)
• Exception-derived strings flowing through the /api/v1/quick_summary and /api/v1/analyze_documents JSON responses are now scrubbed at the HTTP boundary (and at the strategy layer in SourceBasedSearchStrategy) via sanitize_error_for_client, closing a CWE-209 information-exposure vector flagged by CodeQL (#8019): a caught exception interpolated into a strategy's error fields could otherwise reach the API client — as summary (the key quick_summary()/analyze_documents() actually return), formatted_findings, or per-finding content — with credentials or stack-trace text intact. Credentials are scrubbed and length is capped; the "Error: " prefix is preserved so error payloads stay recognizable to clients, and the strategy-layer scrub independently covers the web-UI SSE path (research_service.py → ErrorReportGenerator), which consumes strategy output directly. /api/v1/generate_report gets the same boundary scrub purely as a precaution — its current payload (content/metadata) carries no error fields.

✨ New Features

• Sofya search engine — Added Sofya as an optional hosted web-search engine. Sofya's /v1/search returns ranked results with SERP snippets and extracted page content (markdown) in a single call, fitting LDR's two-phase retrieval model without a separate fetch round trip. Supports basic/snippets depth, general/news topics, recency filtering, and domain include/exclude. Configure your API key under Settings → Search → Sofya (search.engine.web.sofya.api_key) or the LDR_SEARCH_ENGINE_WEB_SOFYA_API_KEY environment variable; GitHub sign-up grants 1000 free credits/month.

🐛 Bug Fixes

• Fixed the news subscription-history endpoint (/news/api/subscriptions/<id>/history) raising an internal error for any subscription that had research runs: get_subscription_history called .isoformat() on the created_at/completed_at values, which are stored as isoformat strings, so it now returns them directly. (#3094)
• Fix collapsed-whitespace bug in SearXNG-sourced titles. The HTML parser was calling BeautifulSoup.Tag.get_text(strip=True) to read result titles, which strips leading/trailing whitespace and collapses every internal whitespace run to nothing — so multi-word titles like "Word One Word Two Word Three" rendered as "WordOneWordTwoWordThree" in the ## Sources block. Swapped to get_text(" ", strip=True) (the documented BeautifulSoup idiom for "strip edges, replace internal runs with a single space") so word boundaries survive. The same fix was applied to result snippets. Title preservation flows through to the LLM relevance filter and BaseCandidateExplorer / ProgressiveExplorer candidate-phrase extraction, both of which previously saw single-token titles from SearXNG-sourced collections. (#4970)
• Registration/login no longer leave a user silently logged in when post-authentication setup fails partway: the partial session is now rolled back, so a reported failure matches reality (you are not logged in). (#5006)
• Fix citation labels rendering as empty [] for library/RAG sources. The DOMAIN_HYPERLINKS, DOMAIN_ID_HYPERLINKS, and DOMAIN_ID_ALWAYS_HYPERLINKS modes previously fed urlparse(url).netloc straight into the citation label, so any relative URL (most notably RAG / library hits that look like /library/document/<uuid>) produced an empty string. With a single relative-URL citation the label collapsed to [[]](url) and rendered as a [] link with no anchor text; with multiple relative-URL citations the suffix leaked out as [[-1]](url), [[-2]](url). The formatter now falls back to a slugified (lowercased, alphanumeric-only) version of the document title, and as a final fallback to the citation number itself, so the label is always non-empty and the link carries the document name. Real web URLs (arxiv.org, github.com, etc.) are unchanged — the domain still wins when netloc is non-empty.
• Fixed formatBytes rendering sizes of 1 TB or larger as "N undefined" (e.g. a library blob total of 1 TB showed as 1 undefined). The shared byte formatter only defined units up to GB; it now covers TB through EB and clamps the unit index at both ends, so very large values no longer index past the end of the unit list and sub-1-byte values no longer produce a negative index.
• When the Research Logs panel hits its 500-entry DOM cap, the oldest entries are flushed first — which previously meant old warnings and errors got dropped even when the cap was blown by a flood of routine info entries. The prune now walks the cap-excess slots in priority order (info first, then milestones, then warnings, then errors), so the panel keeps its most diagnostic entries even on long research runs.

📝 Other Changes

• Removed the importlib.util disk-load workaround for url_classifier in citation_formatter.py (introduced in #4880). Now that content_fetcher no longer eagerly imports its heavy dependency tree (#5023), the formatter uses a normal from ..content_fetcher.url_classifier import URLClassifier, URLType. Also fixed two nits in the #5023 test file: an off-by-one project-root path in the subprocess tests and check=True calls that hid the subprocess stderr from failure reports. (#4992)
• Fixed an order-dependent test flake: the security import-fallback tests replaced local_deep_research.security* modules with fresh copies, recreating enums like Sensitivity and breaking identity checks in tests that ran later on the same worker.
• Source-tagged citation formatting no longer re-runs the URL-classifier label lookup for every citation occurrence — the pre-computed label cache is consulted lazily, so each source's label resolves once. (The disk-loading import this originally worked around was removed entirely in #4992.)
• The Sofya egress-labels test now compares enum members by .value instead of by identity. Under the Docker test setup the package is simultaneously available from /app/src (PYTHONPATH) and the installed wheel, so SofyaSearchEngine's class attribute and the test's Sensitivity can resolve to distinct class objects with identical members. Python's Enum.__eq__ returns NotImplemented across distinct classes (so == fails just like is); only .value (or .name) survives the dual-class scenario.
• The check-pathlib-usage pre-commit hook now catches import os as <alias> followed by <alias>.path.*. Previously the AST matcher only flagged the literal name os, so an aliased import silently bypassed the check — which is how a _os.path.join(...) slipped into #4880 and had to be caught by human review instead.
• content_fetcher.ContentFetcher is now resolved lazily on first attribute access (PEP 562 module __getattr__) instead of being eagerly imported in the package __init__. url_classifier (and any other import that goes through the content_fetcher package) no longer drags in the fetcher's transitive deps (requests, playwright, bs4, lxml, etc.), so minimal test setups and the citation formatter can stop using the disk-load workaround introduced in #4880. Public API is unchanged.

What's Changed

🔒 Security Updates

• feat(security): enforce diagnose-gated logging wrapper in provider/engine paths (#4183) by @Donny-Guo in https://github.com/LearningCircuit/local-deep-research/pull/4976
• security: escape LIKE wildcards across library, domain classifier, news, and journal_quality by @LearningCircuit in https://github.com/LearningCircuit/local-deep-research/pull/3094
• fix(auth): roll back partial session so a post-creation failure isn't a silent login by @LearningCircuit in https://github.com/LearningCircuit/local-deep-research/pull/5006
• security: recognize client_secret/bearer_token/api_secret/… across settings + error-log redaction by @LearningCircuit in https://github.com/LearningCircuit/local-deep-research/pull/5038
• fix(security): scrub exception data at API boundary + close pending CVEs by @LearningCircuit in https://github.com/LearningCircuit/local-deep-research/pull/5032
• chore(dev): consolidate server scripts into restart_server.sh flags by @LearningCircuit in https://github.com/LearningCircuit/local-deep-research/pull/4953

✨ New Features

• feat(search): add Sofya search engine by @yusufgurdogan in https://github.com/LearningCircuit/local-deep-research/pull/4434
• feat(release): complete signal-ordered diff input + hallucination guard for AI release notes by @LearningCircuit in https://github.com/LearningCircuit/local-deep-research/pull/5027

🐛 Bug Fixes

• fix(hooks): catch aliased os imports in check-pathlib-usage by @LearningCircuit in https://github.com/LearningCircuit/local-deep-research/pull/4994
• fix: formatBytes shows "undefined" for sizes >= 1 TB by @Osamaali313 in https://github.com/LearningCircuit/local-deep-research/pull/4982
• fix(logpanel): preserve warnings/errors when cap prunes the log by @skbs-eng in https://github.com/LearningCircuit/local-deep-research/pull/4900
• fix(citations): fall back to title slug when citation URL has no domain by @skbs-eng in https://github.com/LearningCircuit/local-deep-research/pull/4880
• fix(langgraph-agent): route fetch_content calls through the "📖 Reading" branch by @skbs-eng in https://github.com/LearningCircuit/local-deep-research/pull/5011
• fix(release): keep release-artifacts 7 days so approval delay cannot expire them by @LearningCircuit in https://github.com/LearningCircuit/local-deep-research/pull/5016
• perf(citations): resolve source-tag labels from cache lazily by @LearningCircuit in https://github.com/LearningCircuit/local-deep-research/pull/5007
• fix(langgraph-agent): align MAX_SUBTOPICS with 'pass 2-5' contract and make truncation observable (#5012) by @skbs-eng in https://github.com/LearningCircuit/local-deep-research/pull/5013
• fix(test): restore original security modules after import-fallback tests by @LearningCircuit in https://github.com/LearningCircuit/local-deep-research/pull/5029
• fix(search-engine-searxng): preserve word boundaries in parsed titles (#4970) by @skbs-eng in https://github.com/LearningCircuit/local-deep-research/pull/5022
• test(sofya): compare egress labels by value, not identity by @LearningCircuit in https://github.com/LearningCircuit/local-deep-research/pull/5031

⚡ Performance Improvements

• refactor(content_fetcher): defer ContentFetcher import via PEP 562 getattr (#4992) by @skbs-eng in https://github.com/LearningCircuit/local-deep-research/pull/5023
• test(citations): fresh-interpreter guard that citation_formatter import stays light by @LearningCircuit in https://github.com/LearningCircuit/local-deep-research/pull/5036

📚 Documentation

• docs(release): correct retention comment — effective approval window is 5 days, bounded by SLSA provenance by @LearningCircuit in https://github.com/LearningCircuit/local-deep-research/pull/5035

🔧 CI/CD & Maintenance

• chore: bump patch version to 1.9.1 by @github-actions[bot] in https://github.com/LearningCircuit/local-deep-research/pull/4993
• chore: clear changelog fragments for 1.9.0 by @LearningCircuit in https://github.com/LearningCircuit/local-deep-research/pull/5017
• 🤖 Update dependencies by @github-actions[bot] in https://github.com/LearningCircuit/local-deep-research/pull/5009
• 🤖 Update dependencies by @github-actions[bot] in https://github.com/LearningCircuit/local-deep-research/pull/5020
• ci(changelog): fail loud on undeclared towncrier fragment categories by @LearningCircuit in https://github.com/LearningCircuit/local-deep-research/pull/5045

Other Changes

• security: stop /settings/api/bulk leaking nested plaintext secrets by @LearningCircuit in https://github.com/LearningCircuit/local-deep-research/pull/5028
• refactor(security): add shared scrub_error() dual-scrub helper (#4183 follow-up) by @LearningCircuit in https://github.com/LearningCircuit/local-deep-research/pull/5019
• refactor(citations): replace url_classifier disk-loader with a normal import (#4992) by @LearningCircuit in https://github.com/LearningCircuit/local-deep-research/pull/5041
• refactor(strategies): remove dead error-handler helpers with raw exception interpolation by @LearningCircuit in https://github.com/LearningCircuit/local-deep-research/pull/5042
• fix(security): explicit grype ignore for CVE-2026-4360 (NVD-CPE match has fix-state unknown) by @LearningCircuit in https://github.com/LearningCircuit/local-deep-research/pull/5044
• fix(test): restore parent-package security attribute when snapshot is empty by @LearningCircuit in https://github.com/LearningCircuit/local-deep-research/pull/5037
• fix(security): grype ignore for CVE-2026-15308 (CPython html.parser DoS) by @LearningCircuit in https://github.com/LearningCircuit/local-deep-research/pull/5046
• fix(release): pin PyPI build to the release commit, not dispatch-time HEAD by @LearningCircuit in https://github.com/LearningCircuit/local-deep-research/pull/5026

New Contributors

• @Osamaali313 made their first contribution in https://github.com/LearningCircuit/local-deep-research/pull/4982
• @yusufgurdogan made their first contribution in https://github.com/LearningCircuit/local-deep-research/pull/4434

Full Changelog: https://github.com/LearningCircuit/local-deep-research/compare/v1.9.0...v1.9.1