How the Top 1,000 Sites Steer Crawlers: 70% Declare a Sitemap, Half Block Parameter URLs and Internal Search, and One in Eight Still Uses Crawl-delay

Disclosure
This article is promotional content produced as part of the Semrush Creator Accelerator programme. It contains affiliate links to Semrush — if you buy through them we may earn a commission at no extra cost to you — and we may also receive a reward from Semrush for publishing it. All measurements in this article are our own. Semrush did not supply, review or approve the data, and the conclusions we draw are ours.
Advice about crawl budget says that big sites should keep crawlers away from low-value pages, now that AI-generated pages and a growing list of AI crawlers are both adding load. The file where that policy lives is robots.txt. So we asked a simple question: what do the biggest sites actually write in theirs?
This is the second look we have taken at the same files. The first one was about which AI bots they turn away. This one is about everything else.
Find the crawl problems on your own site
Crawl your site and list crawlability and indexability issues with Site Audit in the SEO Toolkit. The AI Visibility Toolkit comes in the same subscription: try Semrush One free for 7 days.
The test
The sample is the same as before: the 1,000 most popular websites by the Tranco ranking, with /robots.txt fetched from each on 13 November 2026. 510 files were readable. On those we counted sitemaps, the number of Allow and Disallow rules, rules that block URLs with parameters, rules that block internal search results, and Crawl-delay lines.
What the files contain
- 358 files (70.2%) declare at least one sitemap. Most of them list one; 56 list ten or more, and one lists 443.
- 252 (49.4%) block URLs with parameters, the kind of rule that keeps filters, sort orders and tracking variants out of the crawl.
- 247 (48.4%) block internal search results. Search pages create unlimited URLs.
- The median file has 28 allow and disallow rules. 133 files (26.1%) have more than 100, and 9 have more than 1,000. The longest, figma.com's, has 4,187.
- 62 (12.2%) use
Crawl-delay, a directive that some crawlers read and that Googlebot does not. - 40 (7.8%) have no rules at all.
The size of a file is not the size of a policy. Half of all files are under 1.4 KB, and only one is over 100 KB.
Rank does not change the habits
The most popular sites are not noticeably more elaborate than the rest. A sitemap was declared by 70.6% of the sites in ranks 1 to 100, 68.8% in ranks 101 to 500 and 71.1% in ranks 501 to 1,000. Parameter rules were found in 51.0%, 45.7% and 51.6%. The median number of rules was 54 in the first band, 25.5 in the second and 28 in the third. Only the top 100 write noticeably longer files.
What this cannot tell you
- That a rule saves crawl budget. We counted what is written, not what a crawler does with it.
- That the files are current. A rule list can be years old.
- How the other 490 sites manage crawling. They returned no readable robots.txt to our request, or none at all.
- That absence means neglect. A site with no rules may control crawling in other ways, such as at the CDN or with
noindex. - That the sample describes your site. These are the largest, most visible sites. A site with a few thousand pages has a smaller problem.
What to do with it
- Declare your sitemap in robots.txt. More than two thirds of the biggest sites do.
- Block the URL families that multiply: parameters, internal search and sort orders. About half of the biggest sites do both.
- Do not rely on
Crawl-delayfor Google. It is a request that Googlebot does not read, so check what your crawl settings say instead. - Keep the file short enough to review. A file with thousands of rules is hard to audit.
- Review it when your templates change, and when you add a new family of pages.
Find the waste, watch the rankings, see what AI cites
Find crawl waste. Site Audit lists errors, warnings and notices by category, including Crawlability, with an explanation and a fix for each.

Catch visibility drops early. Position Tracking in the SEO Toolkit follows daily keyword positions.

See which of your pages AI platforms cite. Visibility Overview in the AI Visibility Toolkit shows mentions, citations and cited pages across AI platforms.

Semrush One carries the 7-day trial; the AI Visibility Toolkit has no trial of its own, and Semrush One includes it with the SEO Toolkit. To audit a site and watch your rankings without paying first, try Semrush One free for 7 days. Semrush's product page says "Unlimited access to all Semrush One tools" (read on 16 October 2026). That is an offer Semrush publishes and we have not tried, and no figure here came from a Semrush account.


A short file, kept on purpose
Most of the biggest sites keep crawlers out of the same few places and say where their sitemap is. Do the same, and put a date in your calendar to read the file again.
Try Semrush One free for 7 days
Site Audit and Position Tracking in the SEO Toolkit, with Visibility Overview in the AI Visibility Toolkit, together in Semrush One. Free for 7 days for new users.
Try Semrush One free for 7 daysHow we counted
- Files: the 510 plain-text robots.txt files read on 13 November 2026 from the Tranco top 1,000 (list saved 5 November 2026). Comments were removed before counting.
- Sitemaps: lines starting with "Sitemap:". Rules: lines starting with "Allow:" or "Disallow:" that have a value.
- Parameter rules: a Disallow whose path contains "?" or "=". Internal search: a Disallow containing "search", "q=" or "s=". Crawl-delay: any Crawl-delay line.
- Bands: ranks 1 to 100 (51 readable files), 101 to 500 (186) and 501 to 1,000 (273).
- Left out: what crawlers do with the rules, other ways of controlling crawling, and the 490 sites with no readable file.
- Reproduce it: the script and the summary are in
research/r2-e5-crawl-budget-robots/.
Some links may be affiliate links. We may earn a small commission at no extra cost to you.
Related Articles
Explore more guides and reviews from our experts.

Top Sites With Organization Markup Block AI Crawlers Slightly Less, 18.9% Against 23.1%, but Plain JSON-LD Shows No Such Pattern: Joining Two of Our Studies

We Re-Checked 2,098 URLs That Five AI Engines Cited: 91.6% Still Load, 0.8% Are Gone, and 7.7% Cannot Be Verified by a Bot
