Update robots.txt (#300)

* Update robots.txt Block CCBot, OpenAI, ChatGPT and GoogleBot other from crawling our site. * Update robots.txt Remove GoogleBot other from our block list because unclear if it's actually related to Bard and don't want to rock the indexing boat right now.
nypublicradio · Aug 2, 2023 · 8523e21 · 8523e21
1 parent bbb4329
commit 8523e21
Show file tree

Hide file tree

Showing 2 changed files with 10 additions and 1 deletion.
diff --git a/public/robots-prod.txt b/public/robots-prod.txt
@@ -2,5 +2,14 @@ User-agent: *
 Disallow: /mt/
 Disallow: /profile/
 
+User-agent: CCBot
+Disallow: /
+
+User-agent: ChatGPT
+Disallow: /
+
+User-agent: OpenAI
+Disallow: /
+
 Sitemap: https://gothamist.com/sitemap.xml
 Sitemap: https://gothamist.com/sitemap-news.xml
diff --git a/public/robots.txt b/public/robots.txt
@@ -3,4 +3,4 @@ Disallow: /mt/
 Disallow: /profile/
 
 Sitemap: https://gothamist.com/sitemap.xml
-Sitemap: https://gothamist.com/sitemap-news.xml
+Sitemap: https://gothamist.com/sitemap-news.xml