Web Researcher MCP

Brand Research Pipeline Overhaul & Scraper Improvements

Branding Capabilities

Authoritative Brand-Portal Pipeline (new Tier 4)

• Replaced third-party BrandFetch dependency and CSS color extraction with a direct brand-portal probe pipeline that discovers official brand portals
• Probes 26 URL patterns across 7 categories (brand.domain, *.domain/brand, styleguide.*, etc.) to locate authoritative brand pages
• Tiers 2 (homepage meta) + 4 (brand-portal probe) run concurrently via goroutines; Tier 5 (web search) runs after portal probe when depth=full
• New brand_portal_resource field returns a research://artifact/{id} URI when a portal is found, storing the rendered brand-portal text for AI agent analysis (24h TTL)

New Fields & Behavior

• suggestion field: populated when no portal was found (guides AI to retry with depth=full), or when a portal URL was discovered but its content was unextractable (includes the URL for manual review)
• NXDOMAIN rejection: subdomain probes (e.g. brand.example.com) that resolve to non-existent DNS are rejected early, preventing false-positive portal matches
• Source fields deduplication: order-preserving dedup applied at fetchHomepageMeta return, fixing duplicate field names (e.g. identity.name appearing twice) when multiple extraction passes touch the same field

Scraping Capabilities

SPA Fast-Path Extension

• isSPADomain() extended to cover app., portal., design., and style. subdomains — these skip markdown/stealth/HTML tiers and go directly to browser tier, improving extraction quality for JS-heavy brand portals

Browser Tier Fixes

• #app, #root selector ordering improved for more reliable SPA content capture
• Relative href resolution fixed — paths like /brand/guidelines now resolve correctly against the page base URL
• Frontify bot-wall markers added ("please enter your viewer credentials", "request access to the brand owner") so Frontify-hosted portals are correctly classified as bot-walled rather than returning empty content

Exported Constant

• scraper.ChromeDisabled exported (was unexported) — allows callers and test code to disable the browser tier without a magic string. Fixes a flaky CI test caused by the Azure IMDS metadata endpoint (169.254.169.254) being reachable from Chromium on GH Actions Azure runners, bypassing Go's DialContext SSRF guard

Tests & Docs

• TestDeduplicateFields added (5 subtests) for the new dedup helper
• docs/TOOLS.md: suggestion field description expanded to cover both cases; Tier concurrency corrected (Tiers 2+4 concurrent, Tier 5 sequential)
• ARCHITECTURE.md: scraper annotation updated to reflect SPA fast-path and optional Exa 5th tier

What's Changed

Full changelog: https://github.com/zoharbabin/web-researcher-mcp/compare/v1.36.2...v1.36.3

Docker Images

# GitHub Container Registry
docker pull ghcr.io/zoharbabin/web-researcher-mcp:1.36.3

# Docker Hub
docker pull docker.io/zoharbabin/web-researcher-mcp:1.36.3

Homebrew

brew install zoharbabin/tap/web-researcher-mcp

Verify Checksums

sha256sum -c checksums.txt