Robots Parser

lastmod 2022-01-17created 2022-01-171 edits

Robots.txt parser for NodeJS

NodeJS robots.txt parser with support for wildcard (*) matching.

Features

• Developed with the goal of compatibility with the IETF Robots Exclusion Protocol draft
• Supported rules
  • User-agent, Allow, Disallow, Sitemap, Crawl-delay, Host
  • Path : Wildcard(*) and EOL($) matching

See also

• Robots.txt

Favorite site

• Github - robots-parser