Skip to content

How the Top 1,000 Sites Steer Crawlers: 70% Declare a Sitemap, Half Block Parameter URLs and Internal Search, and One in Eight Still Uses Crawl-delay

Sohail Akhtar
Researched by Sohail AkhtarTheToolsVerse
November 15, 20267 min read
How the top 1,000 sites steer crawlers in robots.txt: of 510 readable files, 70.2% declare a sitemap, 49.4% block parameter URLs, 48.4% block internal search, 26.1% have more than 100 rules, 12.2% use Crawl-delay and 7.8% have no rules.

Disclosure

This article is promotional content produced as part of the Semrush Creator Accelerator programme. It contains affiliate links to Semrush — if you buy through them we may earn a commission at no extra cost to you — and we may also receive a reward from Semrush for publishing it. All measurements in this article are our own. Semrush did not supply, review or approve the data, and the conclusions we draw are ours.

Advice about crawl budget says that big sites should keep crawlers away from low-value pages, now that AI-generated pages and a growing list of AI crawlers are both adding load. The file where that policy lives is robots.txt. So we asked a simple question: what do the biggest sites actually write in theirs?

This is the second look we have taken at the same files. The first one was about which AI bots they turn away. This one is about everything else.

Find the crawl problems on your own site

Crawl your site and list crawlability and indexability issues with Site Audit in the SEO Toolkit. The AI Visibility Toolkit comes in the same subscription: try Semrush One free for 7 days.

The test

The sample is the same as before: the 1,000 most popular websites by the Tranco ranking, with /robots.txt fetched from each on 13 November 2026. 510 files were readable. On those we counted sitemaps, the number of Allow and Disallow rules, rules that block URLs with parameters, rules that block internal search results, and Crawl-delay lines.

What the files contain

How the biggest sites steer crawlers in robots.txt. Share of the 510 readable robots.txt files among the top 1,000 sites that: declare a sitemap 70.2% (358); block parameter URLs 49.4% (252); block internal search results 48.4% (247); more than 100 rules 26.1% (133); use Crawl-delay 12.2% (62); have no rules at all 7.8% (40).
How the biggest sites steer crawlers in robots.txt. Share of the 510 readable robots.txt files among the top 1,000 sites that: declare a sitemap 70.2% (358); block parameter URLs 49.4% (252); block internal search results 48.4% (247); more than 100 rules 26.1% (133); use Crawl-delay 12.2% (62); have no rules at all 7.8% (40).
  • 358 files (70.2%) declare at least one sitemap. Most of them list one; 56 list ten or more, and one lists 443.
  • 252 (49.4%) block URLs with parameters, the kind of rule that keeps filters, sort orders and tracking variants out of the crawl.
  • 247 (48.4%) block internal search results. Search pages create unlimited URLs.
  • The median file has 28 allow and disallow rules. 133 files (26.1%) have more than 100, and 9 have more than 1,000. The longest, figma.com's, has 4,187.
  • 62 (12.2%) use Crawl-delay, a directive that some crawlers read and that Googlebot does not.
  • 40 (7.8%) have no rules at all.

The size of a file is not the size of a policy. Half of all files are under 1.4 KB, and only one is over 100 KB.

Rank does not change the habits

The most popular sites are not noticeably more elaborate than the rest. A sitemap was declared by 70.6% of the sites in ranks 1 to 100, 68.8% in ranks 101 to 500 and 71.1% in ranks 501 to 1,000. Parameter rules were found in 51.0%, 45.7% and 51.6%. The median number of rules was 54 in the first band, 25.5 in the second and 28 in the third. Only the top 100 write noticeably longer files.

What this cannot tell you

  • That a rule saves crawl budget. We counted what is written, not what a crawler does with it.
  • That the files are current. A rule list can be years old.
  • How the other 490 sites manage crawling. They returned no readable robots.txt to our request, or none at all.
  • That absence means neglect. A site with no rules may control crawling in other ways, such as at the CDN or with noindex.
  • That the sample describes your site. These are the largest, most visible sites. A site with a few thousand pages has a smaller problem.

What to do with it

  1. Declare your sitemap in robots.txt. More than two thirds of the biggest sites do.
  2. Block the URL families that multiply: parameters, internal search and sort orders. About half of the biggest sites do both.
  3. Do not rely on Crawl-delay for Google. It is a request that Googlebot does not read, so check what your crawl settings say instead.
  4. Keep the file short enough to review. A file with thousands of rules is hard to audit.
  5. Review it when your templates change, and when you add a new family of pages.

Find the waste, watch the rankings, see what AI cites

Find crawl waste. Site Audit lists errors, warnings and notices by category, including Crawlability, with an explanation and a fix for each.

Semrush's own illustration of Site Audit's issues list, from its Site Audit feature page (retrieved 29 October 2026): errors, warnings and notices filtered into categories including Crawlability and AI Search, with an "About the issue / How to fix" panel. Semrush's illustration, not our site.
Semrush's own illustration of Site Audit's issues list, from its Site Audit feature page (retrieved 29 October 2026): errors, warnings and notices filtered into categories including Crawlability and AI Search, with an "About the issue / How to fix" panel. Semrush's illustration, not our site.

Catch visibility drops early. Position Tracking in the SEO Toolkit follows daily keyword positions.

Semrush's own illustration of the SERP features menu in Position Tracking, from its Knowledge Base (retrieved 29 October 2026): options to track Local pack, Hotels, AI Overview and Jobs for a tracked domain. Semrush's illustration, not our account.
Semrush's own illustration of the SERP features menu in Position Tracking, from its Knowledge Base (retrieved 29 October 2026): options to track Local pack, Hotels, AI Overview and Jobs for a tracked domain. Semrush's illustration, not our account.

See which of your pages AI platforms cite. Visibility Overview in the AI Visibility Toolkit shows mentions, citations and cited pages across AI platforms.

Semrush's own illustration of the Visibility Overview report in the AI Visibility Toolkit, from its Knowledge Base (retrieved 9 October 2026): an AI Visibility score of 36/100, an "All AI platforms" selector, and mentions, citations and cited pages over six months. Semrush's illustration, not our account.
Semrush's own illustration of the Visibility Overview report in the AI Visibility Toolkit, from its Knowledge Base (retrieved 9 October 2026): an AI Visibility score of 36/100, an "All AI platforms" selector, and mentions, citations and cited pages over six months. Semrush's illustration, not our account.

Semrush One carries the 7-day trial; the AI Visibility Toolkit has no trial of its own, and Semrush One includes it with the SEO Toolkit. To audit a site and watch your rankings without paying first, try Semrush One free for 7 days. Semrush's product page says "Unlimited access to all Semrush One tools" (read on 16 October 2026). That is an offer Semrush publishes and we have not tried, and no figure here came from a Semrush account.

Semrush One's own product page: "Try free for 7 days" button with "Unlimited access to all Semrush One tools" beneath it. Captured from Semrush's page on 26 September 2026; the same wording was live on 16 October 2026.
Semrush One's own product page: "Try free for 7 days" button with "Unlimited access to all Semrush One tools" beneath it. Captured from Semrush's page on 26 September 2026; the same wording was live on 16 October 2026.
Semrush's official Semrush One graphic, from the Semrush Creator Accelerator affiliate resources: "Semrush One" at the centre of ten capabilities, including AI + SEO Visibility, Site Health + AI Search Auditing, AI Brand Narrative and Share of Voice. Semrush's own image.
Semrush's official Semrush One graphic, from the Semrush Creator Accelerator affiliate resources: "Semrush One" at the centre of ten capabilities, including AI + SEO Visibility, Site Health + AI Search Auditing, AI Brand Narrative and Share of Voice. Semrush's own image.

A short file, kept on purpose

Most of the biggest sites keep crawlers out of the same few places and say where their sitemap is. Do the same, and put a date in your calendar to read the file again.

Try Semrush One free for 7 days

Site Audit and Position Tracking in the SEO Toolkit, with Visibility Overview in the AI Visibility Toolkit, together in Semrush One. Free for 7 days for new users.

Try Semrush One free for 7 days

How we counted

  • Files: the 510 plain-text robots.txt files read on 13 November 2026 from the Tranco top 1,000 (list saved 5 November 2026). Comments were removed before counting.
  • Sitemaps: lines starting with "Sitemap:". Rules: lines starting with "Allow:" or "Disallow:" that have a value.
  • Parameter rules: a Disallow whose path contains "?" or "=". Internal search: a Disallow containing "search", "q=" or "s=". Crawl-delay: any Crawl-delay line.
  • Bands: ranks 1 to 100 (51 readable files), 101 to 500 (186) and 501 to 1,000 (273).
  • Left out: what crawlers do with the rules, other ways of controlling crawling, and the 490 sites with no readable file.
  • Reproduce it: the script and the summary are in research/r2-e5-crawl-budget-robots/.

Explore More AI Tools

Browse our curated directory of 855+ verified AI tools.

Browse Directory

Some links may be affiliate links. We may earn a small commission at no extra cost to you.