play watch

Ahrefs’ Crawler and Index

Crawler FAQs

Video for this tutorial is coming soon.

These are some frequently asked questions on our support chat that might help with your final exam.

Question: Why hasn’t Ahrefs found my link? I see it in Google Search Console.
Answer: Our scheduler prioritizes the most authoritative pages that have a higher quality of backlinks pointing to them.

It’s important to also understand the difference between discovery vs. crawling/parsing.

Crawler faqs-1

One of the times our crawler discovers new pages is when our parser discovers a link on a page. So the higher the authority of a page, the easier it is for a link on it to get discovered.

In general, Ahrefs will likely crawl your page if it comes from a high-authority domain.

If you want to get it crawled faster, one way to do it is to raise crawl demand by getting high-quality backlinks to the page.

Question: Why has my link been reported as lost?
Answer: Lost links occur on two occasions:

  1. When something happens to the referring page (like a 404)
  2. When something happens with the link, while the referring page remains live and unchanged (like a swap/removed link)

Redirects from unsecured (HTTP) to secure protocols (HTTPS) count as one lost link and one new link.

Question: Why is Ahrefs’ link data different from other tools?
Answer: Our method of counting links is different from other tools.

  • We trust all redirects and consolidate to the canonical URL.
  • Links we try not to store:
    • Links with URL parameters
    • Links from pages in infinite crawl paths
  • Links we don’t store:
    • Links in PDFs or other docs
    • Links in iframes
    • Links from pages not indexed
    • Same links from multiple IPs
    • Multiple links to the same page from a single page

Next lesson