Skip to content

AIMLPM/markcrawl

RAG & Retrieval

Crawl websites into clean Markdown, search pages, and extract structured data with LLMs. Built-in MCP server for web research and RAG pipelines.

Track releases GitHub
Python Latest v0.11.1 · 2mo ago Security brief →

Features

  • Crawl any webpage or entire website and convert content to clean Markdown
  • Generate a structured JSONL index with URLs, titles, crawl timestamps, and citations
  • Optional binary download support for PDFs, DOCX, etc.
  • Built‑in local embedder (torch + transformers) eliminates API key requirements
  • Tenacity‑backed exponential HTTP retry handling

Recent releases

View all 23 releases →
No immediate action
v0.11.1 Breaking risk

Reject print aggregator URLs

Open
Review required
v0.11.0 Breaking risk
Dependencies

Binary downloads + filters

Open
Config change
v0.10.6 Breaking risk
Auth

Respect robots flag and audit

Open
No immediate action
v0.10.5 Breaking risk

Auto‑scope broadening

Open
No immediate action
v0.10.4 Breaking risk

Idle‑timeout reset behavior

Open

Weekly OSS security release digest.

The CVE patches and breaking changes that affected production tools this week. One email, every Sunday.

No spam, unsubscribe anytime.

About

Stars
3
Forks
0
Languages
Python HTML Shell
View on GitHub

Install & Platforms

Install via
pip

Similar tools

ragflow
graphify
webscraping-ai/webscraping-ai-mcp-server
0xMassi/webclaw
pragmar/mcp-server-webcrawl

Beta — feedback welcome: [email protected]