grobotstxt is a native Go port of Google's robots.txt parser and matcher library.

Generate an XML sitemap for a GitHub Pages site using GitHub Actions

Manage the robots.txt from the Kirby config file

Alternative robots parser module for Python

Lightweight R wrapper around rep-cpp for robot.txt (Robots Exclusion Protocol) parsing and path testing in R

Robots Exclusion Standard/Protocol Parser for Web Crawling/Scraping

Robots for Kirby CMS

RFC 9309 spec compliant robots.txt builder and parser. 🦾 No dependencies, fully typed.

User-Agent parser for robots.txt, X-Robots-tag and Robots-meta-tag rule sets

PowerShell module for reading robots.txt files

The repository contains Google-based robots.txt parser and matcher as a C++ library (compliant to C++17).

Parse robots.txt and sitemaps using dotnet

python binding for Google robots.txt parser C++ library

Typescript robots.txt parser with support for wildcard (*) matching.

Parsers for robots.txt (aka Robots Exclusion Standard / Robots Exclusion Protocol), Robots Meta Tag, and X-Robots-Tag