By Yoni Greenbaum, Backstory & Strategy
Quick answer: AI Readiness Check is a free diagnostic tool that scans any website and reports whether AI training crawlers, citation bots, and AI search tools can access, extract, and reuse its content — and whether the site’s existing policies (robots.txt, llms.txt, AI policy, TDM opt-out) actually protect it if they do. It has been run on nearly 5,000 sites to date.
A note on independence: Backstory & Strategy is a personal, independent publication written by Yoni Greenbaum. The views, analysis, and tools described here are his own and do not represent an official position, strategy, or endorsement of the American Press Institute, its leadership, or its board.
What Is AI Readiness Check?
AI Readiness Check is a web-based scanning tool built for publishers, editors, and press associations. It requires no developer and no technical background to read the resulting report. A user submits a URL, and within roughly one to two minutes the tool returns a report scoring the site on two dimensions — Visibility and Protection — along with a breakdown of specific crawlers, files, and policy gaps.
Unlike manual audits, such as FT Strategies’ recent robots.txt review of 70 major publishers, AI Readiness Check runs automatically, continuously, and across an entire site rather than a single file at a single point in time.
The Four Questions AI Readiness Check Answers
Most newsrooms treat “AI and my site” as a single question. It is actually four separate questions, and a site can score differently on each one:
Can training crawlers reach the site? These are the bots that scrape a site’s archive to train the next generation of AI models.
Can citation bots identify and credit the site accurately? These are the systems behind AI chatbots that answer user questions using reporting the newsroom published first.
Do AI search tools summarize the site in a way that still drives traffic to it? Some AI search tools answer the user’s question directly from scraped content without sending a click to the original source.
Do the site’s own policies provide any protection? This includes whether robots.txt, licensing terms, and any stated AI policy actually restrict unauthorized use — or whether the site has no stated position at all.
AI Readiness Check answers all four questions in a single report, written for a non-technical newsroom leader.
What the Report Measures: Visibility and Protection
Every report returns two separate scores rather than one composite number:
ScoreWhat It MeasuresVisibilityWhether AI tools can find, parse, and cite the site’s content accuratelyProtectionWhether the site’s technical setup and policies restrict scraping and reuse without consent or compensation
Most sites are strong in one dimension and weak in the other. Few sites score well on both.
How to Read Your Score Combination
PatternWhat It MeansHigh Protection, Low VisibilityThe site is locked down — possibly too tightly — and may be invisible to legitimate citation tools it would otherwise want crediting itLow Protection, High VisibilityThe site is easy to find and easy to scrape, with no policy or technical barrier in placeLow Protection, Low VisibilityNo one at the organization has addressed AI access yet — the most common pattern found across nearly 5,000 sites scannedHigh Protection, High VisibilityRare; indicates deliberate configuration on both fronts
Across almost 5,000 sites, composite scores average in the mid-30s (out of 100). This reflects an industry that has largely not addressed AI crawler access, not an unusually low score for any individual site.
What the Report Checks: Crawlers and Policy Signals
Crawler Categories
The report tracks 15 individual training crawlers separately rather than grouping them under a single “AI bot” category. For example, ClaudeBot and the older anthropic-ai user agent are counted as two distinct crawlers, since merging them undercounts actual crawler activity on a site. The same individual tracking applies to citation and AI-search crawlers — the systems behind tools people currently use to get answers to questions without visiting a news site directly.
Policy and Technical Signals
SignalWhat It Checksrobots.txtWhether the file exists and what it permits or blocksllms.txtA newer file format some sites use to set AI-specific access termsAI policyWhether the site has a publicly stated policy on AI use of its contentTDM opt-outThe formal Text and Data Mining opt-out mechanism recognized under EU copyright law and reinforced by the EU AI Act, used to signal “do not train on this content”Cloudflare AI-permission settingsAI-blocking settings some sites have enabled without realizing it; the report distinguishes a blanket site-wide setting from a crawler named and blocked explicitly
Key finding: Across nearly 5,000 sites scanned, formal TDM opt-out adoption is effectively zero. This is not a matter of sites choosing to leave content unprotected — most sites are unaware the mechanism exists.
Key finding: Protection level does not correlate with newsroom size or budget. Some small weekly papers have stronger AI protection configured than national chains, because the fix is typically a text file change, not a technical redesign.
Why the Tool Is Free
The tool’s builder, Yoni Greenbaum, states the reasoning directly: AI presents a specific risk to journalism that sits between two extremes — mass job displacement and total industry collapse. That risk is the harm caused by inaction: newsrooms defaulting into unprotected configurations simply because no one asked the question, with the default becoming permanent by the time anyone notices.
AI Readiness Check is free with no card required and full report access on registration. The tool’s creator has stated that a paid model may be introduced later, and any change will be announced through the Backstory & Strategy newsletter.
How to Run AI Readiness Check
Enter a site’s URL into the tool.
Wait approximately one to two minutes while the scan runs.
Receive a report with a Visibility score, a Protection score, a crawler-by-crawler breakdown, and specific policy or file gaps — written in plain language, not technical jargon.
The report identifies the specific missing file, specific policy gap, or specific crawler currently accessing the site — not generic guidance such as “improve AI governance.”
Background: Where This Data Came From
The tool was already in use before this public launch. It supplied the underlying data for two prior Backstory & Strategy pieces: one analyzing public media organizations’ cloud infrastructure exposure, and one examining who controls an independent journalist’s AI exposure when they publish on a third-party platform such as Substack, Ghost, or Beehiiv rather than an owned domain.
Frequently Asked Questions
What is AI Readiness Check? It is a free scanning tool that reports whether AI training crawlers, citation bots, and AI search tools can access and cite a website’s content, and whether the site’s own policies protect it.
Is AI Readiness Check free? Yes. It requires quick registration and no payment card.
How many sites has it been tested on? Nearly 5,000 sites as of this publication.
Do I need a developer to read the report? No. The report is written for non-technical newsroom leaders — editors, publishers, and press association staff.
What is the difference between the Visibility and Protection scores? Visibility measures whether AI tools can find and accurately cite a site’s content. Protection measures whether the site’s technical setup and policies restrict AI scraping and reuse without consent or compensation.
What is TDM opt-out? Text and Data Mining opt-out is a formal mechanism recognized under EU copyright law and reinforced by the EU AI Act that signals to AI companies not to train on a site’s content. Across nearly 5,000 sites scanned, adoption is effectively zero.
How long does the scan take? Roughly one to two minutes per site.
Is AI Readiness Check affiliated with the American Press Institute? No. It is an independent tool built and maintained by Yoni Greenbaum on Backstory & Strategy, separate from his role at the American Press Institute.
What is a typical score? Composite scores across nearly 5,000 sites average in the mid-30s out of 100, reflecting industry-wide underpreparedness rather than any individual site performing unusually poorly.