
A Quick Disclaimer: Backstory & Strategy is a personal, independent publication. The views, analysis, and commentary expressed here are strictly my own and do not represent the official position, strategy, or endorsement of the American Press Institute, its leadership, or its board. This is my personal space for analyzing the media landscape, testing new frameworks, and thinking out loud.
Correction and update, October 3, 2026
A reader pushed me on the AI readiness numbers in this piece, which sent me back to the underlying data. Something was off. Here’s what I found, and what it changes.
When NewsGrade originally scanned the 381 INN member sites on August 26, 118 of them returned nothing readable for robots.txt. The tool recorded those as 0 on criterion 11.1 (AI crawler management), which pulled the group average down to 0.02 out of 3 — a number I quoted here as evidence of how little the field has done.
The actual issue was on my end. Those 118 sites were behind Cloudflare, and the scan was being blocked at the fetch step. The sites weren’t scoring 0 because they had no AI policy. They were scoring 0 because I couldn’t read their robots.txt file to find out.
I rebuilt the fetch pipeline this week to escalate through my tools when direct requests fail, and re-ran those 118 sites. The results:
108 of 118 are now readable. 10 remain locked down enough that I still can’t fetch their robots.txt even through premium — not an answer, but a different kind of non-answer.
34 of the 108 have active AI crawler directives in their robots.txt. That’s 29% of the recovered group.
The 118-site group now averages 0.63 out of 3 on 11.1, up from 0.02.
Combined with the 265 sites that were readable the first time (which averaged 0.91), the full readable set of 373 INN members averages 0.83 out of 3.
The direction holds. Most INN members have not configured AI crawler policies. But the share who have is meaningfully higher than I reported, and some of those members have made deliberate, specific choices worth naming: The Ithaca Voice, The Oaklandside, Berkeleyside, MinnPost, VTDigger, The Colorado Sun, Charlottesville Tomorrow, Sahan Journal, and others are actively blocking GPTBot, CCBot, Google-Extended, and PerplexityBot in their robots.txt. That’s a decision, not a default.
The 118 affected reports on NewsGrade now display a disclosure noting the rescore. The underlying tool has been fixed. I’ll be more careful about fetch failures masquerading as findings in future analyses.
Thanks to the reader who caught the smell of this.
Public media has a mission problem hiding inside what looks like a technical one. I want to walk through both, because I don’t think you can fix the second without naming the first.
Here’s the assumption I keep running into when I talk to public media leaders about AI crawlers, archive protection, and data licensing: because their content is public, their infrastructure should be too. It feels almost like a matter of principle. The station exists because of a federal charter and a community’s trust. The mission is openness. So the instinct is to leave the technical doors and windows unlocked, because locking them would somehow contradict what public media is for.
I understand the instinct. I think it’s wrong, and I think it’s already costing people money, archives, and control over their own work.
The Pattern, Three Times Now
Back in July, I ran roughly 400 nonprofit newsrooms through a scanning tool I built that measures how exposed a news site is to AI crawlers scraping content for training data without permission or compensation. In practice that means checking for a specific set of technical signals: whether robots.txt names and disallows the known AI crawlers (OpenAI’s GPTBot, Common Crawl’s CCBot, Anthropic’s ClaudeBot, and a dozen others), whether the site publishes an llms.txt file or a public AI-use policy, and whether it has adopted a TDM, or Text and Data Mining, reservation signal, the closest thing the web currently has to a machine-readable “don’t train on this without a deal” flag. None of these are exotic. A competent web developer can implement any of them in an afternoon. The finding, in short: the journalism was fine. The websites worked. Nobody was cutting corners on the reporting. But almost nobody had done anything to raise any of these flags. Good journalism, unlocked front door.
I’ve since run the same check on public media, and it’s a starker version of the same story. I scored 143 public television stations and 209 public radio stations, 352 total. TDM reservation adoption came back at zero. Not low. Zero. Out of 352. robots.txt itself was present on the large majority of sites, which sounds like a good sign until you look at what those files actually say: almost none of them name a single AI crawler, meaning the file exists but was never updated for the thing it would now need to stop. llms.txt and any form of published AI policy were both in the single digits.
The Protection scores tell the same story from a different angle. TV stations averaged 6 out of 100 on protection, with a median of 0. Radio came in a little higher, average 8, median still 0. Meanwhile, the Visibility scores, meaning how easy these sites are to find and crawl in the first place, sat in the high 90s across the board. These are not hard-to-reach, poorly built websites. They are extremely easy to find and completely undefended.
So that’s the data. Newsrooms, then newspapers, now public media. Three samples, same shape. At some point, a pattern stops being a coincidence and starts being a default setting nobody chose on purpose.
What NewsGrade Actually Tells You
Every newsroom has a website. Almost none of them know what that website looks like to the systems now deciding whether it gets found at all.
Two Different Things Being Treated as One
Here’s where I think the public media version of this gets more interesting, and more urgent, than the nonprofit newsroom version. Because public media has a specific story it tells itself about why the doors are open, and that story is doing real damage.
Being publicly funded, having a civic mission, existing to serve a community rather than a shareholder: none of that has anything to do with whether your servers should be configured to let anyone or anything walk in and take what they want. Mission openness and infrastructure openness are two completely different decisions, made for two completely different reasons, and public media has been quietly letting one stand in for the other.
That confusion doesn’t just expose stations to AI training crawlers, which is the framing I usually lead with because it’s the one with the clearest financial angle. It exposes them to everyone. Once the doors are unlocked, they’re unlocked for LLM companies building training sets, and they’re unlocked for anyone else with less benign intentions: people looking to scrape and resell archival material, undermine a station’s reporting by lifting it out of context, or simply help themselves to decades of work with no accountability at all.
This isn’t hypothetical. Nine PBS in St. Louis sued Iron Mountain Data Centers this summer after roughly 50 terabytes of archival material, spanning 70 years, became inaccessible when the storage vendor that held it, Open Source Storage, went out of business and Iron Mountain, the data center actually holding the files, refused to release them. A judge has since ruled Nine PBS can recover the archive, but the station had to go to court to get back material it already owned. That’s not an AI licensing case. It’s not even about crawlers. But it’s the exact same underlying failure: treating the protection of irreplaceable institutional material as someone else’s job, or nobody’s job, because the mission was public service and the infrastructure got left to chance.
Curious where your own site lands? Grade it free at NewsGrade — takes about ten minutes.
The Part That Should Really Get People’s Attention
If the moral argument doesn’t move a station’s leadership, the financial one probably will. And for a sector currently cutting staff after Congress eliminated federal funding, forcing the CPB to dissolve, this isn’t an abstract opportunity cost; it’s money being left on the table at the worst possible time. The emerging thinking in AI licensing circles is that the real value isn’t in tomorrow’s headline, it’s in the archive. A publisher’s homepage was the asset that mattered for decades: first thing readers saw, the thing that drove traffic and ad revenue. AI is quietly changing that math. A single story ages out fast, but an archive compounds. Fifty years of reporting on a region’s politics, culture, and institutions isn’t fifty years of old news to a model being trained to understand how something became what it is today. It’s the connective tissue. That reframes decades of “legacy content” once treated as digital storage into a genuine licensing asset, and public media, with some of the deepest, most locally-specific archives in the country, is sitting on exactly that kind of asset while commercial publishers elsewhere are already getting paid for the same category of material.
That’s the value sitting behind the doors public media has been leaving open. Not a hypothetical future threat. An asset, already built, already valuable, currently unprotected and unmonetized, because the technical posture never caught up to what the archive is actually worth, at precisely the moment these organizations can least afford to be giving anything away for free.
The Policies Already Exist. They’re Just Aimed the Wrong Way.
Here’s what makes the “we’re public, so we stay open” explanation hard to take at face value: public media has not been careless about AI. It’s been careful about the wrong half of it.
WUNC in Chapel Hill added an AI section to its ethics statement this spring, built by an internal committee with staff from every department, governing when AI can assist with editing and verification and when it can’t. NC Local in Wilmington has a published AI policy, drafted by its editorial director after a training session from the Online News Association, that commits the newsroom to disclosing AI use to its audience. PBS North Carolina leans on PBS’s own standards guidance for generative AI and is in the middle of a major project digitizing seventy years of its own back catalog, all detailed in a recent Current piece profiling how three North Carolina outlets are approaching AI. Elsewhere in the system, KQED built a generative-AI tool to make its “Forum” archive searchable, and Chicago Public Media brought on a dedicated AI Fellow. None of that is a system asleep at the wheel.
It’s a system that’s been rigorous about one direction of the AI relationship and silent on the other. Every policy above governs what happens when a station’s own staff reaches for an AI tool. None of them touch what happens when someone else’s AI tool reaches for the station. The committees, the ethics language, the disclosure commitments, the archive investment: all of it assumes AI is something you opt into deliberately, on your own terms. Meanwhile the crawler sitting outside the firewall doesn’t need an invitation, and 352 stations haven’t given it a reason to ask for one.
What Your Tech Team Never Told You About AI
Let’s be honest about what happens in most media leadership meetings. Someone brings up generative AI, the conversation quickly slides into server logs and user-agents, and anyone without an engineering degree quietly nods along so they don’t have to ask a basic question out loud.
The Silence Isn’t Just Local
This isn’t a North Carolina problem, or even a U.S. problem. Zoom out to public broadcasters internationally and the same lopsidedness shows up at a much larger scale. Spain’s RTVE has published a formal set of responsible-AI principles. France Télévisions has written generative-AI use directly into its code of conduct, down to mandatory labeling and a ban on unsupervised AI-written content. Germany’s Bavarian public broadcaster has its own AI ethics guidelines. Researchers who went looking recently counted three dozen-plus formal AI guidelines from media organizations across more than a dozen countries. Every one of them, as far as I can find, governs use. None of them governs exposure.
And at the leadership level here at home, where you’d expect this conversation to surface first if it were happening at all, I haven’t found it. PBS and NPR’s public statements on AI-adjacent territory this year have been almost entirely about the fight to preserve funding after the dissolution of the CPB, not about crawler protection or archive licensing. The closest thing to a public national move on this is a June cease-and-desist letter that Digital Content Next sent to Common Crawlon behalf of member organizations including NPR, alongside AP, the New York Times, NBCUniversal, Bloomberg, and Fox, demanding Common Crawl stop scraping protected content. That’s a real, public, national-level action, and it’s worth noting. But it’s a policy fight over one nonprofit crawler at the top of the system. It says nothing about the 352 individual stations, and no public media CEO or system leader has stood up and connected the dots between “our mission is openness” and “our archives are unprotected” the way I’m doing here.
I want to be honest about what that silence does and doesn’t tell us. It’s entirely possible, maybe likely, that conversations about archive value and AI exposure are already happening inside PBS, NPR, or individual station groups, just not in public. Boards and general managers talk about licensing and vendor risk behind closed doors as a matter of course, and a topic this consequential wouldn’t be an exception. What I can say is that nothing has surfaced publicly yet, no memo, no framework, no leadership statement, that treats this as the systemic issue the data says it is. If that conversation is happening, it’s happening quietly. The data I’m publishing here is an argument for bringing it into the open.
Openness Was Never the Argument Against Protection
I want to be careful here, because I’m not arguing that public media should wall itself off from its audience, retreat behind paywalls, or start acting like it has something to hide. That’s not the ask. The ask is much narrower: separate the decision about who gets to read your journalism for free from the decision about who gets to take your journalism, wholesale, to train a commercial product or resell your archive out from under you.
Those are not the same door. Treating them as the same door is what’s led to a Protection score with a median of zero across 352 stations that most people would assume are being carefully stewarded, because they’re public, because they’re trusted, because they’ve been around for decades and have earned that trust the hard way.
This isn’t Uncharted Territory, and the Tools Already Exist
None of this requires public media to invent something new. The rest of the publishing industry has spent the past two years building exactly the infrastructure this problem calls for, and increasingly it’s a checkbox, not a project. Cloudflare, which a large share of public media sites already sit behind for basic security, now blocks known AI crawlers by default for new customers and offers a Pay Per Crawl program that lets a publisher charge AI companies per fetch instead of just blocking them outright, enforced through the revived HTTP 402 “Payment Required” status code. Cloudflare also runs Project Galileo, which extends its security and AI-crawler tooling free of charge to qualifying nonprofit and independent news organizations, a program built for exactly this kind of budget-constrained newsroom. TollBit has built a parallel marketplace doing much the same thing with deeper publisher-side tooling and curated AI-buyer participation. None of this is theoretical or bleeding-edge. It’s mature enough that the real decision left for most publishers isn’t whether to use it, but which flavor of it fits their situation.
Concretely, closing the gap the data shows means three things: updating robots.txt to explicitly name and disallow AI training crawlers rather than leaving it silent on them, adopting a TDM reservation signal so the “no permission granted” flag is machine-readable, and deciding, station by station or as a system, whether to block AI crawlers outright or route them through a paid-access layer like Cloudflare’s or TollBit’s. None of that requires new hires, a vendor contract nobody can afford, or a rebuilt website. For most stations it’s a configuration change, not a capital project.
Protecting that work isn’t a betrayal of the public-service mission. It’s what lets the mission keep existing long enough to matter. An archive that gets scraped for free, or lost to a vendor’s bankruptcy, or quietly folded into someone else’s AI model with no attribution and no compensation, doesn’t serve the public. It just disappears into someone else’s product.
Zero out of 352 isn’t a technical footnote. It’s a decision, made by default, that’s worth making on purpose instead.
None of this needs to stay a conversation for industry conferences, board executive sessions, or hushed hallway talk at station offices. If you’re trying to figure out what this means for your organization, I’m glad to help think it through. Reach out. This isn’t a moment that requires being first. But being last carries real business consequences.
If you work inside public media, I want to hear from you directly: is this landing as new information, or something people already say quietly in the break room? Drop a comment. I read every one, and I'll respond.
Know someone wrestling with their digital distribution?
Forward this to a colleague, or share it with an editor, GM, or product lead who is navigating the shift to generative search.



