41% of News Sites Block Google-Extended. It Doesn't Touch AI Overviews. We Read 2,380 Publishers' AI Rules as Google Tests Paying for Answers

In September, Digiday reported that Google is testing a Search Console programme that pays publishers when their content contributes to AI answers. Google called it an "early-stage learning pilot". Digiday could confirm that "at least dozens of publishers" had been approached, and one executive with knowledge of it said: "It's quite black box."

Nobody outside the pilot can see the terms, so nobody outside it can judge the deal. What anyone can read is the position publishers have already taken in public, in the files they leave for machines. So we read those files for 2,380 news sites and asked a narrow question: before a licensing offer arrives, what has each publisher actually said to AI systems?

Saying no is the common position

We started from Wikidata's newspapers, online newspapers and news websites, kept those whose own domain appears in the Tranco top-million list of popular sites, and removed shared platforms. That left 2,380 sites from more than 100 countries, from nytimes.com to small regional titles. On 5 November 2026 we fetched each site's robots.txt, its home page and its TDMRep file. 1,921 returned a robots.txt we could read.

1,046 of the 1,921 (54.5%) fully block at least one AI crawler. Another way to read the same file: 817 (42.5%) do not name a single AI crawler at all, so their position is whatever their general rules happen to say.

Many of these are not separate decisions. Newspaper groups share one file across their titles, and the 785 sites that block Google-Extended use 468 distinct rule sets between them. Even so, refusal is a mainstream position in news, and one that took a few lines of text to take.

The switch that does not do what it looks like

The line that matters most for Google's pilot is the fourth bar. 785 sites (40.9%) block Google-Extended. Here is what Google says that token does: it manages whether crawled content may be used to train future Gemini models and for grounding in Gemini Apps and the Vertex AI API, and it "does not impact a site's inclusion in Google Search" (Google's crawler documentation).

AI Overviews and AI Mode are part of Search. For those, Google's guidance on AI features points to different controls. So a publisher that blocks Google-Extended has said something real about Gemini, and nothing about the AI summary sitting above its own headline.

The Search Console setting is the honest gap in our data: some of these publishers may already have used it, and no outside check can tell. But it comes with a stated price. Google's own words: "Sites that opt out will not receive traffic or impressions from our generative AI features." That is the same trade the pilot asks publishers to weigh from the other side, and it is why the decision needs a number behind it.

Where deals show up in the file

Robots.txt also records relationships, if you read it as a list of exceptions rather than a list of bans.

The New York Times, which is suing OpenAI, blocks GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended. The Financial Times blocks ClaudeBot, PerplexityBot and Google-Extended but leaves GPTBot alone, and it announced a content partnership with OpenAI in April 2024. The Guardian follows a similar pattern: ClaudeBot and PerplexityBot blocked, GPTBot not, after its own OpenAI partnership in February 2025.

We cannot prove why any file says what it says. The pattern is still useful to anyone preparing to negotiate: a blanket block is a position; an exception is what a position looks like after a deal. Across the sample, 750 sites block both GPTBot and Google-Extended, 181 block GPTBot but not Google-Extended, and only 35 do the reverse.

Naming terms is the rare position

A block says "no". It does not say "yes, on these terms", and licensing needs the second sentence. Machine-readable ways to say it now exist:

Together, 219 of the 2,380 sites (9.2%) carry at least one of these signals. And most of them still only reserve rights. We opened all five RSL files:

So two sites in 2,380 publish, in a form a machine can read, the terms on which AI may use their journalism. Content-Signal is much the same. Cloudflare's definitions separate ai-train from ai-input, the latter covering grounding and "generative AI search answers", which is the use the Google pilot pays for. 133 of the 135 files say ai-train=no, but only 15 say ai-input=no, and Cloudflare says it will not serve an ai-input signal for its managed customers because it does not know their preference.

Another 362 files (18.8%) carry a human-readable notice in their comments, such as the FT's "We expressly prohibit any use of our content or data for any machine learning or artificial intelligence purposes". That is a clear legal statement, written where no parser will act on it.

Bigger sites refuse more, except for Google

The largest sites block PerplexityBot, OAI-SearchBot, ClaudeBot and GPTBot more often than the smallest. On Google-Extended the gap disappears: 42.5% against 42.2%. A robots file can't tell us why. One reading is that blocking Google-Extended costs nothing in Search, so it is an easy decision at any size, while the other crawlers are weighed more selectively.

Two more differences are worth noting. 64% of the largest sites block at least one AI crawler, against 49% of the smallest. And the smallest sites are more than five times as likely to carry a Content-Signal line (11.8% against 2.1%), in 72 of 76 cases inside Cloudflare's managed block, while three of the five RSL licences sit on top-10,000 sites. So at the small end, a publisher's stated position is often an infrastructure setting. If those publishers are later offered a licensing rate, they will come to that conversation with little of their own on record.

Five things to do before accepting any AI terms

1. Audit what your robots.txt actually says. Check each AI crawler by name. Inherited rules and CDN settings may have taken a position for you.
2. Stop treating Google-Extended as an AI Overviews switch. Decide about Gemini and about Search separately, using the controls Google names for each.
3. Measure your AI visibility now. A baseline taken after you join a programme cannot show what the programme changed.
4. Write down your terms. Even a one-line licensing contact gives a counterparty somewhere to go. An RSL file or a Content-Signal line tells a machine as well.
5. Separate training from answers. They are different uses, with different effects on traffic, and Google's documentation says it intends to support a shared AI-training signal in early 2027. Decide on each one before someone else's default does.

Build the baseline a licensing decision needs

If you are offered payment for AI use, the useful comparison is before and after: citations, rankings and traffic. Participants in Google's pilot reportedly see an earnings figure in Search Console, and an earnings figure alone cannot tell you what you gave up.

Track citations and visibility over time. Visibility Overview, in the AI Visibility Toolkit, reports an AI Visibility score with mentions, citations and cited pages by month, so you can set a date and compare either side of it. It covers Google AI Mode and AI Overviews, along with ChatGPT and Gemini, for the question of whether a Google deal changes anything elsewhere.

See which articles earn the citations. The citations data in the same report, under Cited Pages, ranks your URLs by how many AI prompts cite them. Those are the pages a contribution payment would most plausibly attach to, and the ones to watch if you opt out.

Watch rankings and AI Overviews together. Position Tracking records whether your tracked keywords show an AI Overview and whether your page is cited in it, so a change after opting in or out can be separated from an ordinary ranking move.

Check what your own site blocks. Site Audit lists the pages closed to AI search bots such as ChatGPT-User and OAI-SearchBot, which is the fastest way to learn what your robots.txt and CDN rules are saying on your behalf.

Semrush One bundles the SEO Toolkit and the AI Visibility Toolkit, and the 7-day free trial is Semrush One's — the AI Visibility Toolkit has no free trial of its own. To take a first baseline before any decision, try Semrush One free for 7 days: its page offers "Unlimited access to all Semrush One tools" and notes "Offer available to new users only", checked on 5 November 2026. Those are Semrush's published terms, not something we tested; none of the data above came from a Semrush account.

A "no" is not a negotiating position

Half of the news sites we could read have told AI crawlers no, and four in ten have said it to a token that does not reach the AI answers above their own results. Only two have published terms for a yes. Google's pilot may or may not become a real market, but any market needs sellers who have stated what they are selling and can measure what it is worth. Most publishers have done neither yet.

Where every figure comes from

• Publishers (2,380): Wikidata items that are newspapers, daily newspapers, online newspapers or news websites, with an official website, at least three Wikipedia-family links and no dissolution date, whose exact host appears in the Tranco top-1M list. Search engines, social networks, blog platforms, universities and government sites were removed. Size bands use Tranco rank: top 10,000 (371 sites), 10,001 to 100,000 (1,130), beyond 100,000 (879).
• Fetches: one request each for /robots.txt, the home page and /.well-known/tdmrep.json, on 5 November 2026, with a user agent naming this study. Of 2,380 robots.txt requests, 1,921 returned a readable file, 207 refused us, 118 could not be reached, 97 returned something that was not a robots file and 37 had none. Percentages for robots.txt use 1,921; home-page tags use the 1,767 home pages that loaded.
• A block means a group naming that crawler with Disallow: / and no Allow: /. Partial blocks and rules addressed only to * are not counted as naming a crawler. A "written notice" is a comment mentioning AI, machine learning or data mining together with a rights word such as licence, terms, copyright or prohibit.
• What the files can't reveal: Search Console settings, private licensing deals, why any file says what it says, or whether any crawler obeys it. Robots.txt is a request, not a lock. Firewalls can block crawlers the file allows. Tags were read on home pages only. Wikidata and Tranco favour established titles, so this is not a census of all news publishers. New York Times litigation and the partnerships named are public record; the link between them and each file is our inference.

Written by Sohail Akhtar. The saved files and analysis script can be re-run.