Bot Type: Good (Identifies itself, has an official moniker)
Obeys Robots.txt: Yes
Obeys Crawl Delay: Yes
User-Agent String: Mozilla/5.0 (compatible; AhrefsBot/6.1; +http://ahrefs.com/robot/) IP address range
Check the status of your robots.txt in our crawler:
What is AhrefsBot?
AhrefsBot is a Web Crawler that powers the 12 trillion link database for Ahrefs online marketing toolset. It constantly crawls web to fill our database with new links and check the status of the previously found ones to provide the most comprehensive and up-to-the-minute data to our users.
Link data collected by Ahrefs Bot from the web is used by thousands of digital marketers around the world to plan, execute, and monitor their online marketing campaigns.
This data has immense value for SEO community as it helps marketing professionals to better understand the fundamental algorithms of the world's largest search engines so that they could optimize websites accordingly. Every 24 hours our crawler visits over 6 billion web pages and updates Ahrefs’ index every 15-30 minutes.