Inclusion
Up to now, the content was only scraped from pages which returned with a success code (200's and similar), it is now scraped from every page, including error pages, 404's and redirects.
Up to now, the content was only scraped from pages which returned with a success code (200's and similar), it is now scraped from every page, including error pages, 404's and redirects.