A Quick Disclaimer: Backstory & Strategy is a personal, independent publication. The views, analysis, and commentary expressed here are strictly my own and do not represent the official position, strategy, or endorsement of the American Press Institute, its leadership, or its board. This is my personal space for analyzing the media landscape, testing new frameworks, and thinking out loud.
Let’s be honest about what happens in most media leadership meetings.
Someone brings up generative AI, the conversation quickly slides into server logs and user-agents, and anyone without an engineering degree quietly nods along so they don’t have to ask a basic question out loud.
We’ve let the entire conversation get swallowed by developer jargon, legal acronyms, and high-minded debates about whether training models is fair use. Meanwhile, editors, broadcast GMs, and independent creators are left trying to figure out what is actually happening to the work they publish every single morning.
Here is the plain-English version of what is happening under the hood of your site, why it matters to your bottom line, and why the real problem isn’t just about training an algorithm.
1. Tokens: How Machines Actually “Eat” Your Words
Tech folks love talking about token limits and context windows. It sounds like arcade currency, but it’s really just the basic unit of machine reading.
Large language models don’t read complete sentences the way you and I do. They chop your text into small character bundles called tokens, which work out to roughly three-quarters of a word. When an automated crawler hits your site and pulls in a story, it is turning your reporting into commercial compute units.
When an AI vendor charges an enterprise client to summarize your investigative piece, they bill them by the token. In plain terms, your editorial work is being converted into raw fuel for someone else’s balance sheet.
Who Actually Controls Your AI Exposure?
Over 95 percent of independent Substack writers have zero AI-crawler protection, mostly because the setting is buried where almost nobody thinks to look. Here is what an audit of 700+ newsletters on Substack, Ghost, and Beehiiv reveals about who actually owns your work.
2. robots.txt Is an Honor-System Doormat, Not a Deadbolt
When leadership asks how a site protects its work from scrapers, the standard answer from IT is usually, “Don’t worry, we updated our robots.txt file.”
Think of your website like your front porch, and think of robots.txt like a handwritten sign on the door that says: Please don’t come inside.
It’s just a text file sitting on your server that asks automated visitors to stay out of certain folders. There is zero security behind it. As formalized in the IETF Robots Exclusion Standard (RFC 9309), the entire protocol is purely advisory. Well-behaved corporate crawlers from companies like Google or OpenAI generally follow the rules because they don’t want regulatory trouble. But rogue scrapers, offshore data brokers, and brand-new AI startups ignore the sign completely and walk right in.
Our CMS came with a robots file” (or “We updated it last year”):
Relying on the default file that came with your CMS is like keeping the factory password on your home Wi-Fi. Those defaults were written a decade ago for traditional search engines and have no idea what modern AI scrapers look like. Crawler tech moves fast, so a file you set twelve months ago is blind to the newest scrapers hitting the web today. It also likely fails to separate bulk training scrapers like
GPTBotfrom live answer engines likeOAI-SearchBotorPerplexityBot, which means you might inadvertently block traffic-driving citations while leaving other doors wide open. Managing bots is an active operational process, not a box you check once and forget.
3. llms.txt Is the VIP Room Service Menu
If robots.txt is the sign asking people to stay away, llms.txt is the exact opposite: a clean cheat sheet made specifically for machines.
Websites are naturally messy. They have navigation bars, newsletter pop-ups, tracking pixels, and ads everywhere. The emerging llmstxt.org proposal cuts through all that clutter, giving an AI engine a clean, Markdown-formatted summary of who you are, what you cover, and your most important links.
If your strategy relies on having generative search engines cite your analysis accurately when people search your beat, handing them an llms.txt makes their job easy. But if your entire business model depends on keeping your analysis behind a hard paywall, serving up a pre-digested summary works directly against your own subscription gate.
“Nobody else I know has one, so why should I bother?”:
Most publishers don’t have one simply because it’s a newer standard, but early adopters are using it as an intentional discovery tool. When a researcher uses an AI tool to dig into your beat, that tool has to parse messy code to figure out who you are. An
llms.txtfile makes sure the machine gets the right story straight from you. If you choose not to create one, just make sure it’s an intentional strategic choice rather than an accident.
4. TDM Rights: The Legal Fence
You might have heard media lawyers or European publishers bring up Text and Data Mining (TDM) rules.
Under Article 4 of the EU Copyright Directive, companies are actually allowed to scrape the open web for data mining by default unless a publisher posts a specific, machine-readable opt-out. Tools like the W3C TDM Reservation Protocol (TDM-Rep) give publishers a standardized way to embed a line of code in site metadata that formally says: We reserve our rights, and you do not have permission to harvest this work for commercial AI use without a license.
While robots.txt is a polite technical request, a TDM tag is a legal stake in the ground. If you ever want to join a licensing collective down the road or challenge unauthorized scraping, having that tag in your code gives you a documented paper trail.
“If this is a European thing, why care in the US?”:
The internet doesn’t care about borders, and neither do the bots scraping your servers. The platforms crawling your site operate globally. When American publishers eventually sit down for licensing talks, tech companies will argue that any content published without an automated reservation of rights was fair game. Adding the tag costs next to nothing, and it makes your position clear from day one.
5. Edge Blockers: The Actual Bouncer
Because robots.txt is only a polite request, infrastructure providers have introduced Web Application Firewalls (WAFs) and managed crawler barriers, such as Cloudflare’s AI scraper blocking controls, that act like real bouncers at the door.
These systems sit in front of your server and inspect incoming traffic before it ever touches your site. If a visitor looks like an automated scraper pretending to be a regular reader on a laptop, the system can block their IP address right at the perimeter.
The tricky part for publishers is striking the right balance. If you lock down your site too aggressively, you might protect your content from scrapers, but you could also block the real-time search crawlers that drive citations and discovery. It’s a continuous choice between protecting your intellectual property and staying visible to new readers.
6. Why You Should Actually Be Upset: It’s Not Just “Training”
The media narrative is always focused on big tech companies harvesting archives to train future models. That makes it sound like something that happened in the past and is already over.
The real day-to-day extraction is happening right now in three distinct ways:
The Zero-Click Synthesizer: When someone asks an AI search engine a question today, a bot visits your site live, reads your newest piece, summarizes the takeaways, and answers the reader directly on its own screen. The reader gets your insight, and you get zero traffic, zero ad impressions, and zero newsletter sign-ups.
The Shadow Data Brokers: Third-party scraping outfits harvest paywalled and local reporting at scale, clean it up, package it into neat databases, and sell access to software companies that want the data without taking on the legal risk of scraping you directly.
Autonomous Task Agents: AI agents scrape pricing tables, how-to guides, and local reporting to run automated tasks inside other productivity tools, removing the need for a human to ever visit your publication.
7. The Blind Spots for Broadcast and Print Leaders
If you run a local TV station, radio group, or newspaper, your risk reaches well beyond your homepage.
Broadcast Audio and Video Leaks: Closed captioning feeds, live video streams, and unencrypted podcast RSS feeds are wide open. Automated transcription models can download raw audio enclosures directly from your podcast feed or listen to an on-air broadcast, turning a major scoop into a searchable text summary on an AI engine faster than your digital team can publish the web writeup.
Voice and Likeness in Talent Deals: Scrapers harvest broadcast audio and video every day to build synthetic voice and video clones. If your anchor and reporter contracts don’t explicitly cover synthetic replication and likeness protections, you have a real institutional gap.
The Syndication Backdoor: You can build great defenses on your own website, but if your enterprise reporting gets shared across a state wire, a parent company CMS, or an affiliate partner with weak protections, scrapers will simply pull your story from their open servers.
We Got Really Good at the Wrong Problem
I ran NewsGrade across 400+ nonprofit newsrooms, and the data is clear: our websites are solid for human readers, but completely broken for machine discovery. Here is what happens when an entire sector solves yesterday’s technology problems while ignoring the one that actually controls its future reach.
8. The Social-First Trap
A lot of creators and modern publishers think that building entirely on social platforms lets them skip the open-web mess. In reality, it does the exact opposite.
You Agreed to the Terms of Service: When you post native video, graphics, or threads, policies like Meta’s AI Privacy Terms and X’s Terms of Service explicitly grant those platforms broad rights to train internal AI models on your public submissions. You are freely feeding the exact tools competing for your audience’s attention.
You Have Zero Defensive Controls: You cannot install a firewall, set a
robots.txtrule, or post a TDM legal reservation on an Instagram post or a YouTube video.In-Feed Summaries Cut You Out: Platforms are rapidly rolling out native AI summaries right inside their feeds. When an app summarizes your reported video directly above the post, users get the takeaway without ever clicking your profile or joining your email list.
9. What About Licensing Deals?
Every time a major publisher announces an eight-figure deal with OpenAI or Anthropic, my inbox gets flooded with questions from local station GMs and independent publishers asking: “Should we be pitching a licensing deal? Who do we call?”
The honest answer is that the big AI labs aren’t doing individual deals with regional newsrooms, single broadcast stations, or independent newsletter writers.
Those headline deals happened because massive conglomerates have massive archives and globally recognized brand names that look good during congressional hearings. Unless you are sitting on petabytes of structured data, cold-pitching a custom licensing deal is not a realistic path.
For the rest of the industry, monetization is going to happen in two practical ways:
Automated Tollgates: Infrastructure platforms like TollBit, ProRata, and Cloudflare are building systems that let publishers meter AI traffic directly at the server level, charging small fees or requiring attribution whenever a bot reads an article.
Collective Bargaining: As outlined in the News/Media Alliance White Paper on Generative AI, trade organizations are actively working on collective frameworks so independent and mid-market publishers can pool their content assets together, much like how musicians collect royalties through ASCAP or BMI.
You can’t license work you’ve abandoned to the open web. Make sure you set your TDM opt-outs, clean up your structured metadata, and ensure your freelance and syndication agreements state clearly that you own all downstream AI licensing rights.
The Monday Morning Checklist
You don’t need to write code to set the strategy here. You just need to ask your technical and operations teams four direct questions:
“Are we relying on a basic
robots.txtfile, or do we have edge-level firewall rules actively managing automated bots?”“Do our settings separate regular search engine bots from real-time synthesis bots that take our content without sending traffic?”
“Have we added machine-readable TDM reservations to our site metadata?”
“Do our talent contracts and syndication agreements protect our audio, video, and shared wire stories from secondary scraping?”
When search engines and social platforms stop sending referral traffic, the only real protection you have is owning the direct relationship with your audience through your email list, your app, and your community.
A Quick Practical Note:
If you don’t know where your site currently stands on structured data, metadata hygiene, or crawler access, you don’t have to guess. We built NewsGrade to audit these exact technical and editorial perimeters for newsrooms—checking your structured metadata, evaluating how cleanly machines can parse your work, and generating tailored recommendations and starter files so you don’t have to start from scratch.
Hit reply and let me know what your team says when you ask them question #1.
A quick note on how this was made: In the spirit of transparency, I used generative AI as a thought partner to help structure and edit this piece. My goal was to make sure the technical explanations remained clear, grounded, and accessible to non-technical newsroom leaders without losing the practical nuance.
Enjoyed this breakdown?
If you found this helpful, consider subscribing to Backstory & Strategy to get practical, non-jargon analysis on the business and technology of media delivered straight to your inbox every week.
Know someone wrestling with their digital distribution?
Forward this to a colleague, or share it with an editor, GM, or product lead who is navigating the shift to generative search.




