Some links on this page are affiliate links: if you buy through them we may earn a commission, at no extra cost to you.
AI companies are not automatically required to pay every time an answer uses web content. But a new open standard called Really Simple Licensing (RSL) is designed to let publishers publish machine-readable rules for crawling, training, retrieval, attribution and payment. Cloudflare is separately testing a system that can require payment before an AI crawler retrieves a page.
The important distinction is between a licensing signal, an enforceable access system and a functioning payment market. RSL provides the first. Products such as Cloudflare Pay Per Crawl attempt to provide the second and third, but adoption, identity, measurement and legal enforceability remain unsettled.
What happened?
The RSL Collective launched Really Simple Licensing in September 2025. The open standard is intended to extend the web’s traditional robots.txt convention beyond simple “allow” or “disallow” instructions. Its launch coalition included organizations such as Reddit, Yahoo, Quora, Medium, The Daily Beast and Fastly, according to Ars Technica’s launch coverage.
Do these 3 things before closing this tab:
1Scan for outdated or missing drivers - takes under a minute2Repair Windows errors before they cause bigger problems3Fix the driver behind crashes, sound loss and screen glitchesThe headline claim that AI firms were “blindsided” needs qualification. The reporting indicates that major companies had not publicly prepared a response to the proposal; it does not prove that they were literally unaware of it or technically unable to respond.
The simple version
| Layer | What it does |
|---|---|
| Signal | robots.txt, HTTP headers or HTML metadata point crawlers to a publisher’s terms. |
| License | RSL describes permitted uses, such as search, training, retrieval, attribution or paid use. |
| Enforcement and payment | A CDN, WAF, authentication system, license server, payment processor or contract determines whether access actually occurs. |
That means RSL is not simply a new mandatory robots.txt command. It is an XML-based licensing format that can be discovered through several web mechanisms. The Robots Exclusion Protocol remains primarily an access-preference system; it is not a universal payment rail.
How RSL works
An RSL license can be linked from a site’s robots.txt file, an HTTP Link header or an HTML <link> element. A simplified discovery entry might look like this:
User-agent: *
License: https://example.com/rsl-license.xmlThe linked document can express different permissions and payment models. An example from the RSL 1.0 specification is:
<rsl xmlns="https://rslstandard.org/rsl">
<content url="/">
<license>
<payment type="crawl">
<amount currency="USD">0.015</amount>
<standard>https://example.com/licenses/pay-per-crawl</standard>
</payment>
</license>
</content>
</rsl>This is an RSL example, not ordinary standardized robots.txt syntax. RSL 1.0 is described by its maintainers as a Recommendation and supports licensing for web pages as well as assets such as books, videos, images and datasets.
What does “pay-per-output” mean?
“Pay-per-output” is a useful shorthand, but it can blur several very different events. RSL’s specification refers to the relevant model as use, covering payment when content contributes to an AI-generated output, including inference, grounding or generation. RSL’s explanatory material also uses the term pay-per-inference.
| Model | Payment trigger | Core difficulty |
|---|---|---|
| Pay-per-crawl | A crawler successfully retrieves a page. | Duplicate requests, caching and page-level pricing. |
| Pay-per-training | Content is used in model training. | Attributing information inside datasets and model weights. |
| Pay-per-inference or pay-per-use | Content contributes to a generated answer. | Proving which sources materially influenced the answer. |
| Subscription | A recurring license fee. | Defining volume, scope and downstream uses. |
| Attribution | No payment, but a credit or link is required. | Ensuring the credit is displayed and useful. |
Pay-per-crawl is easier to meter because the trigger is a network request. Pay-per-inference is harder: an AI system may retrieve a page, cache it, combine it with other sources and produce an answer without a simple one-to-one relationship between page and output.
Is RSL the same as Cloudflare Pay Per Crawl?
No. RSL is an open format for declaring rights and licensing terms. Cloudflare Pay Per Crawl is an infrastructure product designed to control crawler access and attach a payment workflow to successful retrievals.
They may work together, but they solve different problems:
Rank #3
- RSL tells a crawler what the publisher’s terms are.
- Cloudflare can place those terms at the CDN and security layer.
- Authentication helps establish that a crawler is genuine.
- Payment and settlement determine whether the transaction actually completes.
- Contracts and courts, where relevant, address legal disputes that software alone cannot resolve.
How Cloudflare’s system works
As of August 2026, Cloudflare’s documentation describes Pay Per Crawl as a closed or private beta rather than a universally available marketplace. Its documented workflow is more concrete than a static license file:
- A publisher enables Pay Per Crawl for a domain.
- The publisher sets a price per successful retrieval. The documented minimum is $0.01 per crawl.
- The publisher chooses whether particular crawlers are allowed free access, charged or blocked.
- An AI crawler requests a page.
- The site can respond with
HTTP 402 Payment Requiredand acrawler-priceheader. - The crawler can retry with
crawler-exact-priceorcrawler-max-priceheaders. - The crawler must meet Cloudflare’s Web Bot Auth requirements.
- Cloudflare says it acts as merchant of record and provides the underlying infrastructure.
Cloudflare’s FAQ says that repeated retrievals of the same page incur the configured cost. Discovery resources such as /robots.txt, /sitemap.xml, /security.txt, /.well-known/security.txt and /crawlers.json are documented as free to crawl.
The system also has important limits. Cloudflare’s documentation says one price currently applies to all crawlers placed in the “Charge” category. Later 2026 changelog entries describe more advanced URI-specific configuration and dynamic pricing through a response header or Cloudflare Worker, but those developments do not make the service a universal pay-per-answer market.
Recommended Free Tools
Can a publisher make every AI company pay?
Not reliably. Several conditions must align:
- The crawler must identify itself accurately.
- Its request must pass through the publisher’s enforcement layer.
- The crawler must understand the licensing format.
- It must accept the price and support the payment workflow.
- Authentication must distinguish a genuine crawler from a spoofed user agent.
- The publisher must have rights to license the material.
- A payment, audit and dispute process must exist.
A crawler that ignores robots.txt or an RSL file may be blocked by a WAF, rate-limited, denied behind authentication or pursued through legal remedies, depending on the facts and jurisdiction. Cloudflare also notes that WAF or Bot Management blocking rules override the Pay Per Crawl charge feature: a blocked crawler does not simply become a paying crawler.
Rank #4
Most importantly, publishing a licensing instruction does not automatically establish that every crawler has entered a binding contract. The legal effect of machine-readable terms varies by jurisdiction and circumstances. Payment also does not automatically settle copyright, fair-use, ownership or database-rights questions.
Why AI companies may resist
At web scale, even a small per-page price can become significant. AI companies would also face operational problems involving:
- Duplicate URLs, syndicated stories and cached pages.
- Different prices and license terms across thousands of publishers.
- Whether a retrieval payment also covers training or later reuse.
- Attribution when a model combines many sources.
- Fraud, spoofed crawlers and payment-related denial-of-service attacks.
- Incentives to favor free or bulk-licensed sources.
Paying to retrieve a page is therefore not the same as purchasing the right to incorporate all of its information into model weights or reproduce its expression in an answer.
Quick wins for a faster PC:
Scan for outdated or missing drivers - takes under a minuteDriver Scan →Clear out junk files and repair common Windows errorsFree Scan →Why publishers may adopt it anyway
For publishers, RSL could provide a common way to distinguish among search indexing, AI training, live retrieval and generated summaries. It may also reduce the need to negotiate a separate technical format with every AI company.
But monetization is not automatically better than visibility. Charging or blocking search crawlers can reduce indexing, referral traffic and discoverability; Cloudflare specifically warns publishers to consider the SEO effects. A site might instead choose to:
- Allow search crawlers for free.
- Block training crawlers.
- Charge selected retrieval systems.
- Offer premium material through an authenticated API.
- Use RSL to publish different terms for different content categories.
The right choice depends on whether the expected licensing income outweighs the value of search traffic, advertising, subscriptions, referrals and audience growth.
The hardest unresolved issue: measuring influence
Pay-per-crawl has a clear event: a successful retrieval. Pay-per-inference does not. A model may retrieve an article, store it in a cache, combine it with ten other sources and generate an answer later. Alternatively, it may have learned similar facts during training but be unable to identify the original source at inference time.
A workable pay-per-output market would need agreed definitions for contribution, source attribution, caching, syndicated material, pricing, audits, refunds and disputes. It would also need trusted crawler identity and a way to prevent publishers from claiming payment for content that did not materially affect an answer.
What to watch next
RSL’s significance will be determined less by the specification than by adoption. The most useful indicators are:
- Whether major AI and search crawlers implement RSL discovery and payment.
- How many publishers deploy machine-readable licenses.
- Whether Cloudflare’s payment feature leaves private beta.
- Actual transaction volume and average prices.
- Whether pay-per-inference systems become operational beyond a specification.
- How search engines distinguish free indexing from paid AI use.
- Whether courts or regulators give legal weight to machine-readable licensing signals.
Bottom line
RSL is an attempt to put a meter and licensing vocabulary around AI access to online content. It can express pay-per-crawl and pay-per-inference terms, but it cannot by itself force an AI company to comply. Cloudflare’s Pay Per Crawl makes paid retrieval more tangible through 402 responses, crawler authentication and a documented minimum price of $0.01, yet it remains a beta infrastructure product rather than proof of a universal pay-per-output economy.
The near-term practical value is likely to be better differentiation and control: allowing search, restricting training, charging selected crawlers or moving valuable data behind authenticated systems. Whether publishers ultimately earn meaningful per-output revenue will depend on identity, attribution, enforcement and broad participation—not merely on adding a line to robots.txt.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
