polite-crawler
A polite web crawler for research and AI agents. It fetches a source once, keeps it, and reads it many times: robots.txt obeyed, one request per site, full-text search. The MCP server is the package polite-crawler-mcp.
as observed 2026-11-03T12:42:38.775Z- Identifier
polite-crawler- Source
- npm
- Version observed
- 0.3.0
- Source repository
- https://github.com/mark1russell7/crawler
- Repository observation
- Source repository not yet checked
- First observed here
- 2026-11-03T12:42:38.775Z
- Observations recorded
- 1
- Weekly downloads (upstream)
- 257
- Publish rights held by
- mark1russell7
- Declared license
- MIT
Observation history
One observation recorded so far. Differences appear here once a second observation differs from the first.
Correction
If you maintain this extension and believe anything above is inaccurate, request a correction. Corrections are published, and disputed entries are marked as disputed while under review.