Ahrefs’ Crawler and Index
Crawler FAQs
These are some frequently asked questions on our support chat that might help with your final exam.
Question: Why hasn’t Ahrefs found my link? I see it in Google Search Console.
Answer: Our scheduler prioritizes the most authoritative pages that have a higher quality of backlinks pointing to them.
It’s important to also understand the difference between discovery vs. crawling/parsing.
One of the times our crawler discovers new pages is when our parser discovers a link on a page. So the higher the authority of a page, the easier it is for a link on it to get discovered.
In general, Ahrefs will likely crawl your page if it comes from a high-authority domain.
If you want to get it crawled faster, one way to do it is to raise crawl demand by getting high-quality backlinks to the page.
Answer: Lost links occur on two occasions:
- When something happens to the referring page (like a 404)
- When something happens with the link, while the referring page remains live and unchanged (like a swap/removed link)
Redirects from unsecured (HTTP) to secure protocols (HTTPS) count as one lost link and one new link.
Answer: Our method of counting links is different from other tools.
- We trust all redirects and consolidate to the canonical URL.
-
Links we try not to store:
- Links with URL parameters
- Links from pages in infinite crawl paths
-
Links we don’t store:
- Links in PDFs or other docs
- Links in iframes
- Links from pages not indexed
- Same links from multiple IPs
- Multiple links to the same page from a single page