Ahrefs
Tools
Site explorer Keywords explorer Site audit Rank tracker Content explorer SEO toolbar
Pricing Help Blog Academy Our data
Deutsch English Español Français Italiano 한국어 日本語 Nederlands Polski Português Русский Svenska Türkçe 中文
  • Sign in
  • Start a 7-day trial for $7
Deutsch English Español Français Italiano 한국어 日本語 Nederlands Polski Português Русский Svenska Türkçe 中文
AhrefsBot AhrefsSiteAudit bot

AhrefsSiteAudit bot

Version: 6.1
Bot Type: Good (Identifies itself, has an official moniker)
Category: SEO
Obeys Robots.txt: Yes by default (website owners can request to disobey robots.txt on their sites)
Obeys Crawl Delay: Yes by default (website owners can request to disobey crawl delay on their sites)
User-Agent String: Mozilla/5.0 (compatible; AhrefsSiteAudit/6.1; +http://ahrefs.com/robot/)
Reverse DNS suffix: ahrefs.com
IP address range

Check the status of your robots.txt in our crawler:

What is AhrefsSiteAudit crawler?

AhrefsSiteAudit is a Web Crawler that powers Ahrefs' Site Audit tool. This tool is used by Ahrefs' users to analyze websites in search for issues with technical SEO and on-page SEO.

AhrefsSiteAudit bot can crawl any website unless disallowed. To prevent any excessive load on a website server, crawling is limited to 1 request per 2 seconds by default. Website owners can crawl their sites at higher speeds and can allow AhrefsSiteAudit crawler to ignore robots.txt. To do that, they need to verify ownership in Site Audit tool.

What is AhrefsSiteAudit bot doing on your website?

If you noticed AhrefsSiteAudit in your server log, this means that someone had performed an analysis of your website using Ahrefs' Site Audit tool. Our bot crawls a website, following its internal links from one URL to another. It also checks the website resources such as CSS files, JS files, and images.

AhrefsSiteAudit crawler will not add numbers to your Google Analytics traffic.

Does it respect robots.txt file?

Yes. By default.

Only verified owners can allow AhrefsSiteAudit crawler to disobey robots.txt on their site

How to control AhrefsSiteAudit crawler on your website?

AhrefsSiteAudit follows the “allow” and “disallow” directives in the robots.txt files. So you can control it on your website if you need.

If for some reason you want to prevent AhrefsSiteAudit from crawling your site, put the two following lines into the robots.txt file of your site:

User-agent: AhrefsSiteAudit
Disallow: /

You can also allow or disallow any part of your website for crawling in robots.txt file. Read more about robots.txt at www.robotstxt.org.

AhrefsSiteAudit bot will check your robots.txt file prior to a new crawl.

Please also note that if your robots.txt contains errors and AhrefsSiteAudit bot won’t be able to recognize your commands, it will continue crawling your website the way it did before.

If you think that AhrefsSiteAudit bot is somehow misbehaving on your website or if you have any questions about it, please don’t hesitate to contact our support team support@ahrefs.com.

  • Main tools

    • Site explorer
    • Keywords explorer
    • Site audit
    • Rank tracker
    • Content explorer
    • SEO toolbar
  • Extra tools

    • Domain comparison
    • Batch analysis
    • Link intersect
    • Content gap
    • Email alerts
  • Features

    • Backlink checkerTry for free
    • Broken link checkerTry for free
    • Website authority checkerTry for free
    • Keyword generator
    • YouTube keyword tool
    • Amazon keyword tool
  •  

    • Bing keyword tool
    • SERP checker
    • SEO analyzer
    • Keyword rank checker
    • Keyword difficulty
About Team Jobs Our data SEO blog Tech blog Robot Pricing API Help Contact us
© 2019 Ahrefs Pte. Ltd. (201227417H) 16 Raffles Quay, #33-03 Hong Leong Building, Singapore 048581
Terms & conditions  |  Privacy policy

7 days. 7 dollars. Full access.

Our 7-day trial gives you full access to all tools and features of your chosen plan.

  • Lite

    7 days for $7, then $99/mo

    Suits freelance marketers and solopreneurs. Get full access to Ahrefs' core tools and features with enough data to do SEO for your personal projects.

  • Standard

    7 days for $7, then $179/mo

    Perfect for SEO consultants and in-house marketers. Get everything in Lite with more features and increased data limits to research a large number of websites.

Email already registered. Please, Sign in

I accept Ahrefs' Terms of use

You can upgrade, downgrade, or cancel your subscription anytime.
No contracts, no hidden charges.