0xMassi/webclaw

Added

• --output-dir: save each extracted page to its own file instead of printing to stdout
• Works with single URL, crawl, and batch modes
• Filenames derived from URL path: /docs/api → docs/api.md
• Root URLs use hostname/index.md to avoid collisions
• Subdirectories created automatically
• CSV input with custom filenames: url,filename format in --urls-file

Examples

# Crawl a docs site, one file per page
webclaw https://docs.example.com --crawl --depth 2 --output-dir ./docs

# Batch with custom filenames
echo "https://example.com,home.md
https://example.com/about,about.md" > urls.csv
webclaw --urls-file urls.csv --output-dir ./output

# JSON format
webclaw https://example.com --crawl -f json --output-dir ./data

Full changelog: https://github.com/0xMassi/webclaw/blob/main/CHANGELOG.md

Full Changelog: https://github.com/0xMassi/webclaw/compare/v0.1.4...v0.1.5