Saturday, April 4, 2020

ahrefs can't crawl site hosted on Bluehost despite being allowed in robots.txt

Hello,

Trying to use ahrefs against a site. The error returned is:

The HTTP server returned error 406: "Not Acceptable". This request was likely filtered by a firewall.
This website is protected by the ModSecurity firewall

Robots.txt is configured to permit ahrefs to crawl the site:

User-agent: AhrefsSiteAudit Allow: / User-agent: AhrefsBot Allow: / User-agent: * Disallow: / 

Any idea how to fix this?

Thanks

submitted by /u/FubarOtter
[link] [comments]

from Search Engine Optimization: The Latest SEO News https://www.reddit.com/r/SEO/comments/fuyiks/ahrefs_cant_crawl_site_hosted_on_bluehost_despite/>

No comments:

Post a Comment