The finding
Forty-nine of Australia's most visited websites block an AI answer bot while leaving Googlebot completely free to crawl. They are in Google's index and out of the AI answer.
Only 29 of them appear to have chosen that. The other 20 have an old catch-all rule in their robots.txt that blocks anything not on a named list, and Google is on the list while the AI crawlers are not. Nobody decided to leave ChatGPT and Perplexity. A rule written for a different era did it for them.
That is the short version. Here is the whole study, the method, and what it does and doesn't prove.
What we measured
The global picture has been measured several times this year. The Australian one hadn't, so we ran it.
We took the Tranco top-1M list dated 21 September 2026 and filtered it to commercial Australian domains, keeping .com.au, .net.au and direct .au registrations, and excluding .gov.au, .edu.au and .org.au. Government and university sites have different reasons to allow or refuse crawlers and would distort a commercial picture.
That produced 500 domains. We tagged each by sector and by whether the brand is Australian-owned or a multinational operating an Australian site, then removed infrastructure domains (nameservers, CDNs, ISP mail hosts, which rank highly because of DNS lookups rather than visits), adult sites and sites we couldn't identify. 353 commercial brands remained.
On 29 September we requested /robots.txt from each one, at one request per second, with a user agent that identified us and linked back to this site. We parsed each file into its user-agent groups and evaluated 19 crawlers against it:
Training crawlers, which collect content to build or fine-tune models: GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended, meta-externalagent, Bytespider, Amazonbot and anthropic-ai.
Search and answer bots, which fetch a page at the moment someone asks a question and decide whether you get cited in the reply: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, DuckAssistBot and MistralAI-User.
Controls: Googlebot and bingbot. If our parser were wrong, these two would show implausible block rates. They don't.
We counted a crawler as blocked only where the whole site is disallowed. Partial rules, the kind that keep bots out of a cart or an admin path, are not blocking.
316 of the 353 returned a usable result. The rest never answered and are excluded rather than counted as allowing, which would have flattered the numbers.
The headline numbers
Of those 316 Australian sites:
- 26% block at least one AI training crawler
- 17% block at least one AI search or answer bot
- 15.5% block an answer bot while still allowing Googlebot
- Googlebot is blocked by 1.6%, bingbot by 3.5%
The controls are the proof the method works. Almost nobody deliberately blocks Google, and our measurement says exactly that.
Australia against the world
An independent census of the Tranco top 5,000, run on 7 September 2026, measured the same crawlers the same way. Its author published the raw data.
| Measure | Australia (316 sites) | Global top 5,000 (2,771 sites) |
|---|---|---|
| Block at least one training crawler | 26% | 25.2% |
| Block at least one answer bot | 17% | 13.3% |
| Block an answer bot, allow Googlebot | 15.5% | 10.7% |
| Googlebot blocked | 1.6% | 2.8% |
On training crawlers we look like everyone else. On answer bots we block more. Treat the gap as a direction rather than a precise difference: the samples are different sizes, three weeks apart, and we can't fully reconcile the two denominators.
It splits hard by industry
Blocking is not a web-wide behaviour in Australia. It is a media behaviour.
| Sector | Sites | Blocks training crawlers | Blocks answer bots |
|---|---|---|---|
| Media and publishing | 69 | 51% | 48% |
| Property | 9 | 44% | 33% |
| Marketplaces and directories | 10 | 60% | 30% |
| Gambling and racing | 12 | 25% | 25% |
| Automotive | 9 | 56% | 22% |
| Sport and entertainment | 16 | 31% | 19% |
| Tech and software | 22 | 27% | 18% |
| Travel and hospitality | 12 | 58% | 17% |
| Retail and ecommerce | 76 | 9% | 1% |
| Telco and internet | 15 | 7% | 0% |
| Finance and insurance | 33 | 3% | 0% |
| Health | 8 | 0% | 0% |
For a publisher, blocking is a negotiating position, and a legitimate one. If your business is the content, you want to be paid when a model consumes it. Half the media sample has taken that position on training crawlers, and roughly the same share has extended it to answer bots.
Everyone else has effectively opted in to AI crawling, whether or not anyone discussed it.
One more cut worth noting: multinationals' Australian sites block training crawlers at 38%, against 23% for Australian-owned businesses. Those rules are set at head office and applied to the local domain, with nobody here in the conversation.
The blocks that robots.txt never shows you
Fifteen sites refused to hand our script the file at all, then served the identical public file to a browser a minute later.
That is bot management at the network edge, judging visitors by what they claim to be. A security layer that turns away anything which isn't a recognised browser or a known search engine treats AI crawlers the same way, and it does it before robots.txt is ever read. Your SEO team can audit the file all day and never see it.
This is about to become much more common. On 1 July 2026 Cloudflare, which sits in front of a substantial share of the web, replaced its single AI-bot toggle with three categories — Search, Agent and Training — and announced new defaults from 15 September 2026: for all newly onboarding domains, Training and Agent crawlers are blocked by default on pages that display ads, while Search remains allowed.
Read that carefully if you run ads on your site. Agent crawlers are the ones that fetch your page when a user asks their assistant a question. Blocked by default. Nobody at your company will receive a notification.
How to check your own site in ten minutes
- Open
yourdomain.com.au/robots.txtin a browser. If nothing loads, you have no file, and nothing is blocked. - Find any group beginning
User-agent: *. If it containsDisallow: /, you are blocking everything that isn't named elsewhere in the file. Check which crawlers are named. Googlebot usually is. The AI crawlers usually aren't. - Search the file for GPTBot, ClaudeBot and Google-Extended. Those are your training-crawler decisions.
- Search for OAI-SearchBot, Claude-SearchBot and PerplexityBot. Those decide whether you can be cited in an AI answer. Most Australian sites have never named them either way.
- If you use Cloudflare, open the security settings and look at the AI bot policies for Search, Agent and Training. The file in step 1 is not the whole story, and this is the part the marketing team can't see.
- Decide deliberately. Blocking training crawlers while allowing answer bots is a coherent position. Blocking both is a coherent position. Blocking answer bots by accident while trying to stop scrapers is not.
What this doesn't prove
We measured what sites say, not what crawlers do. Robots.txt is a request. Reputable operators honour it; others don't, and some user-triggered agents explicitly may not apply it.
We can't prove intent. What we can show is that 20 sites blocked answer bots through a catch-all rule while naming Googlebot as allowed. That pattern is consistent with an oversight, and inconsistent with a deliberate AI policy, but we are reading a file, not a meeting.
The 37 sites we never reached may well include more edge-level blocking. Excluding them is the conservative choice and probably understates the true blocking rate.
And a domain-ending filter misses Australian brands that trade on a .com. A future run using actual Australian visit data would catch them.
Method and reruns
Tranco top-1M list dated 21 September 2026, filtered to commercial Australian domains. Checked 29 September 2026. One request per second, identified user agent, raw files retained. Sites that failed on both a research user agent and a browser user agent are excluded from all percentages.
We'll rerun this every quarter on the same list and the same method, so the direction of travel becomes visible rather than guessed at.
Sources
- Tranco list, tranco-list.eu
- Global comparison: a September 2026 census of robots.txt on the Tranco top 5,000, published with its raw data, dev.to/reesecalder
- Cloudflare's July 2026 announcement of Search, Agent and Training categories and the September defaults, blog.cloudflare.com
- Longitudinal context on web consent protocols: Consent in Crisis: The Rapid Decline of the AI Data Commons, arXiv:2407.14933
If you want to know what your own setup is telling the crawlers, get in touch. It's a ten-minute check and we're happy to do it.