Sitemap crawling should respect robots.txt

[benjaminestes]

If for some reason a site blocks its own sitemap with a robots.txt file, the crawler should respect that and not request the sitemaps in sitemap mode.