Version: 6.1
Bot Type: Good (Identifies itself, has an official moniker)
Category: Marketing
Obeys Robots.txt: Yes (only website owners can request to disobey robots.txt on their sites)
User-Agent String: Mozilla/5.0 (compatible; AhrefsSiteAudit/6.1; +http://ahrefs.com/robot/)
IP address range
Check the status of your robots.txt in our crawler:
AhrefsSiteAudit is a Web Crawler that powers Ahrefs' Site Audit tool. This tool is used by Ahrefs' users to analyze websites in search for issues with technical SEO and on-page SEO.
AhrefsSiteAudit bot can crawl any website unless disallowed. To prevent any excessive load on a website server, crawling is limited to 1 request per 2 seconds by default. Website owners can crawl their sites at higher speeds and can allow AhrefsSiteAudit crawler to ignore robots.txt. To do that, they need to verify ownership in Site Audit tool.
If you noticed AhrefsSiteAudit in your server log, this means that someone had performed an analysis of your website using Ahrefs' Site Audit tool. Our bot crawls a website, following its internal links from one URL to another. It also checks the website resources such as CSS files, JS files, and images.
AhrefsSiteAudit crawler will not add numbers to your Google Analytics traffic.
Yes. By default.
Only verified owners can allow AhrefsSiteAudit crawler to disobey robots.txt on their site
AhrefsSiteAudit follows the “allow” and “disallow” directives in the robots.txt files. So you can control it on your website if you need.
If for some reason you want to prevent AhrefsSiteAudit from crawling your site, put the two following lines into the robots.txt file of your site:
User-agent: AhrefsSiteAudit
Disallow: /
You can also allow or disallow any part of your website for crawling in robots.txt file. Read more about robots.txt at www.robotstxt.org.
AhrefsSiteAudit bot will check your robots.txt file prior to a new crawl.
Please also note that if your robots.txt contains errors and AhrefsSiteAudit bot won’t be able to recognize your commands, it will continue crawling your website the way it did before.
If you think that AhrefsSiteAudit bot is somehow misbehaving on your website or if you have any questions about it, please don’t hesitate to contact our support team support@ahrefs.com.