Go language library for parsing Sitemaps
Alternative robots parser module for Python
R 📦 for parsing and checking robots.txt files 🤖
A collection of Docker images: robotstxt, linuxbrew, gcloud, and psql
Provides python access to Googles parser for robot.txt files as used by their GoogleBot webscraper.
The right robots.txt file for your project
Parsers for robots.txt (aka Robots Exclusion Standard / Robots Exclusion Protocol), Robots Meta Tag, and X-Robots-Tag
GitHub action to rebuild CDN cache according to sitemaps
The Blogger Robots.txt Generatois a tool designed to simplify the process of creating a robots.txt file for websites hosted on the Blogger platform. A robots.txt file is crucial for controlling how search engines index your site. This generator allows users to customize and generate a robots.txt file tailored to their specific needs.
🌐 Displays the contents of robots.txt and sitemap.xml files of a website google extension
Makes it easy to add robots.txt, sitemap and web app manifest during build to your Astro app.
Parse robots.txt and traverse sitemaps.
⚙️ A quality `robots.txt` ruleset parser to ensure your application follows the standard specification for the file.
Be nice on the web
A Simple Robots.txt Manager Plugin For Wordpress.
A webpack plugin to generate a robots.txt file
Add a description, image, and links to the robotstxt topic page so that developers can more easily learn about it.
To associate your repository with the robotstxt topic, visit your repo's landing page and select "manage topics."