RobotSeeker

Features • Installation • Usage • Status • Flags • Output • CHANGELOG

RobotSeeker is fast and reliable python tool that grabs robots.txt files from a bunch of subdomains asynchronously

Grabbing robots.txt is useful because it can include sensitive information and endpoints intended to be hidden

Features

• Write all found robots.txt URLs into one file
• Generate a wordlist from the words in all the robots.txt files. Helpful for fuzzing the same target
• Map the endpoints from each robots.txt with its subdomain

Installation

git clone https://github.com/iis2h/RobotSeeker.git &&
cd RobotSeeker &&
pip3 install -r requirements.txt

Usage

Flags

Flag	Description
-h	Help menu
-f	Path to the input file
-m	Map the content of robots.txt to its corresponding URL
-g	Generate a wordlist
-r	Requests per second (Default is 3) Increasing the rate might affect the result
-v	Enable verbose output
-q	Quite Mode (no banner)
--version	Display version

---

Status

Status	Description
[Satus Code]	HTTP status codes (200, 404, 502 ...)
BLANK	Either an empty page or a Soft 404 page
ERROR	Error when trying to connect

normal mode

python3 robotseeker.py -f subdomains.txt -m -g

  ___  ___  ___  ___ _____ ___ ___ ___ _  _____ ___
 | _ \/ _ \| _ )/ _ |_   _/ __| __| __| |/ | __| _ \
 |   | (_) | _ | (_) || | \__ | _|| _|| ' <| _||   /
 |_|_\\___/|___/\___/ |_| |___|___|___|_|\_|___|_|_\

 Crafted with Passion by iis2h aka Frenzy

[200]: http://sub1.example.com/robots.txt
[200]: http://sub2.example.com/robots.txt
[200]: http://sub3.example.com/robots.txt
[200]: http://sub4.example.com/robots.txt
[200]: http://sub5.example.com/robots.txt

verbos mode

python3 robotseeker.py -f subdomains.txt -m -g -v

  ___  ___  ___  ___ _____ ___ ___ ___ _  _____ ___
 | _ \/ _ \| _ )/ _ |_   _/ __| __| __| |/ | __| _ \
 |   | (_) | _ | (_) || | \__ | _|| _|| ' <| _||   /
 |_|_\\___/|___/\___/ |_| |___|___|___|_|\_|___|_|_\

 Crafted with Passion by iis2h aka Frenzy

[404]: https://sub6.example.com/robots.txt
[200]: https://sub7.example.com/robots.txt
[ERROR]: Cannot connect to https://sub8.example.com
[502]: https://sub9.example.com/robots.txt
[BLANK]: https://sub10.example.com/robots.txt

---

Output

File	Description	Created
valid	Valid URLs	Automatically
wordlist	Generated Wordlist	When using -g flag
mapped	Endpoints Mapped to its Subdomain	When using -m flag

valid.txt

wordlist.txt

mapped.txt

---

Your stars greatly support our project ⭐