Bot Type: Good (Identifies itself, has an official moniker)
Obeys Robots.txt: Yes
Obeys Crawl Delay: Yes
User-Agent String: Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)
Reverse DNS suffix: ahrefs.com
IP address range
Check the status of your robots.txt in our crawler:
AhrefsBot is a Web Crawler that powers the 12 trillion link database for Ahrefs online marketing toolset. It constantly crawls web to fill our database with new links and check the status of the previously found ones to provide the most comprehensive and up-to-the-minute data to our users.
Link data collected by Ahrefs Bot from the web is used by thousands of digital marketers around the world to plan, execute, and monitor their online marketing campaigns.
This data has immense value for SEO community as it helps marketing professionals to better understand the fundamental algorithms of the world's largest search engines so that they could optimize websites accordingly. Every 24 hours our crawler visits over 6 billion web pages and updates Ahrefs’ index every 15-30 minutes.
Additionally data collected by AhrefsBot is used to power Ahrefs services and is used to provide search service at Yep. Beyond this, it is also used to train and evaluate different AI algorithms.
AhrefsBot is the most active web crawler in the “Search Engine Optimization” category on Cloudflare Radar. It’s also the 6th most active web crawler in the world.
We strictly respect robots.txt, both disallow and allow rules.
As mentioned above, AhrefsBot strictly follows the robots.txt file on your website. So you can fully control it on your website if you need.
To change the frequency of AhrefsBot visiting your site, you can specify the minimum acceptable delay between two consecutive requests from our bot in your robots.txt file:
Where Crawl-Delay value is time in seconds.
If for some reason you want to prevent AhrefsBot from visiting your site, put the two following lines into the robots.txt file on your server:
Please note that AhrefsBot may need some time to pick the changes in your robots.txt file. This will be made prior to each next scheduled crawl.
Please also note that if your robots.txt contains errors and AhrefsBot won’t be able to recognize your commands it will continue crawling your website the way it did before.
Read more about robots.txt at www.robotstxt.org.
If you think that AhrefsBot is someway misbehaving on your website or if you have any questions about it, please don’t hesitate to contact our support team [email protected]