Skip to content

Commit

Permalink
Block common AI bots
Browse files Browse the repository at this point in the history
  • Loading branch information
@paulmwatson
paulmwatson committed Oct 4, 2024
1 parent 959ef95 commit 23d7193
Showing 1 changed file with 13 additions and 1 deletion.
14 changes: 13 additions & 1 deletion robots.txt
View file
Original file line number Diff line number Diff line change
@@ -1,4 +1,16 @@
User-agent: *
Crawl-delay: 120
Disallow: /user/
Disallow: /mailreporter/
Crawl-delay: 120
User-agent: Amazonbot
Disallow: /
User-agent: Bytespider
Disallow: /
User-Agent: PerplexityBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: OAI-SearchBot
Disallow: /
User-agent: GPTBot
Disallow: /

0 comments on commit 23d7193

Please sign in to comment.