Recommended Free Tools
Cloudflare is testing a system that lets participating website owners charge AI crawlers for successfully retrieving their content. The feature, Pay Per Crawl, remains in closed beta as of August 18, 2026: it is not a universal web toll, and AI companies do not have to pay unless they choose to access a site whose owner has enabled the system.
Why publishers want more control over AI crawling
Search engines have traditionally crawled pages to build indexes that can send visitors back to publishers. AI training crawlers, AI search services, assistants and agents may also retrieve large amounts of material, but that access does not necessarily bring comparable referrals. Publishers therefore face a choice between allowing access for potential reach, blocking it, or seeking compensation.
Cloudflare has argued that AI crawling can produce many more requests than referrals and that AI training traffic has grown. Those are Cloudflare’s measurements and claims, not a neutral industry-wide accounting; its discussion is at Cloudflare’s post on AI training and content control. The underlying concern is broader than training: search indexing, retrieval for AI answers, and ordinary scraping are different uses, even if all involve automated requests.
What Cloudflare Pay Per Crawl does
Announced on July 1, 2025, Pay Per Crawl is part of Cloudflare’s AI Crawl Control tools. It gives an enrolled site owner a way to set a charge for a crawler’s successful content retrieval. Cloudflare says it acts as merchant of record and handles the payment infrastructure. The feature is still in closed beta, so site owners need access from Cloudflare and crawler operators need to support the integration. See the launch announcement and Pay Per Crawl overview.
PC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware match#1 Best Overall
- 【Powerful Load-bearing】12U Network Rack Open Frame is constructed from durable cold rolled steel; Rack shelf supports enhance stability, wall-mounted capacity of 130lbs, the ground-mounted up to 260lbs
- 【Considerate Designs】Open-frame layout, including a top panel adding space, anti-slip shelf stops fixing devices and compatible racks for stack and expansion to meet requirements of home server rack
- 【Complete Accessories】A 12U open frame server rack, two ventilated shelves, four shelf stops, four velcro straps and a set of equipment mounting screws
- 【Versatile Application】Ideal for space-efficient multi-device setups in warehouses, retail, classrooms, offices and more; Excellent choices as AV Rack/IT Rack
- 【Effortless Setup】 Network Rack includes hardware, a comprehensive manual, mounting hole drilling template and an online assembly video to simplify setup
This is an access-and-payment mechanism, not a universal HTTP payment standard or a license that automatically governs how content is used after retrieval. A publisher wanting terms for training, attribution, redistribution, or downstream commercial use may need a direct licensing agreement instead.
How the payment flow works
- An AI crawler requests a page from a site using Cloudflare.
- Cloudflare identifies the crawler through its supported bot-identification process and applies the site owner’s policy: allow, block, or charge.
- For a chargeable request, a crawler that supplies the required payment intent and pricing information can receive the content with HTTP 200. If it does not, Cloudflare can respond with HTTP 402, “Payment Required,” and pricing information.
- A successful retrieval creates a charge. Error responses are not billed; requesting the same page again can create another charge.
- Cloudflare manages payment processing and merchant-of-record duties; the site owner can monitor deliveries and payouts.
HTTP 402 is used in this Cloudflare implementation. Its presence does not mean every website or AI crawler on the internet follows a shared payment protocol. Details of the workflow and crawler requirements are in Cloudflare’s technical overview and FAQ.
What publishers can configure
Choose crawlers and set a price
AI Crawl Control lets site owners assign individual crawlers an action: allow, block, or charge. Cloudflare’s documented minimum is $0.001 per crawl. That is a floor, not a recommended price or evidence that the amount will cover a site’s costs. Under the standard configuration, one price applies to all crawlers put in the Charge category; the basic setup does not provide a separate rate for each AI company.
Rank #2
- Adjustable Depth: 23-40'' adjustable depth is used for servers and network equipment, ensuring enough space for AV equipment, components, and cabling, while allowing you to access ports and equipment from multiple sides.
- Strong Load Capacity: Ground-Mounted Load Capacity: 500 lbs, Wall-Mounted Load Capacity: 150 lbs. The av rack is made of carbon steel for better weldability performance and can help save space while meeting your need to place multiple devices.
- User-friendly Design: Ergonomic design makes the open frame av rack easier to use. The additional top panel is able to place other items with more available space. Roller design moves anywhere and anytime, is convenient, and is more energy-saving.
- Complete Accessories: We provide the accessories you need, including 2 x Pallets, 145 x M5*10 Cross Head Screws, 4 x Casters, 4 x M10*50 Expansion Screws,10 x M6*12 Cage Nuts, 1 x Grounding Wire, 1 x User Manual.
- Wide Application: The server rack wall mount maximizes the use of available space, suitable for retail venues, classrooms, offices, and other places where space is limited.
For an eligible account, the documented dashboard route is AI Crawl Control → Payments → Pay Per Crawl → Enable. Set the default per-crawl price, optionally enable dynamic pricing, then save. Cloudflare says AI Crawl Control’s monitoring and controls are available across its plans, but Pay Per Crawl is separately in private beta. See Cloudflare’s setup instructions and its AI Crawl Control overview for current access details.
Do these 3 things before closing this tab:
1Clear out junk files and repair common Windows errors2Scan for outdated or missing drivers - takes under a minute3Repair Windows errors before they cause bigger problemsUse dynamic pricing selectively
With dynamic pricing, an origin or Worker can return a crawler-price response header, for example crawler-price: USD 3.14. Cloudflare documents an origin-facing header that indicates the mode: cf-pay-per-crawl: protocol=cloudflare, pricing=in-band. This allows prices to vary by content or request conditions, but it adds implementation and testing risk: an application could return the wrong price, omit a header, or apply a charge to a navigation page or API endpoint unintentionally. See the advanced configuration guide.
Keep policy and discovery files free
Cloudflare lists these paths as free to crawl: /robots.txt, /sitemap.xml, /security.txt, /.well-known/security.txt, and /crawlers.json. Its advanced guidance also suggests considering free access for homepages, category and directory pages, navigation, functional pages, login and search endpoints, and APIs not meant to be monetized. Charging every request can obstruct discovery or charge for low-value traffic rather than the material a publisher intends to license.
Rank #3
- Adjustable Depth: Depth adjustable from 23" to 40", this open frame server rack accommodates servers and network equipment while providing ample space for A/V gears and cable management. Enjoy easy access to ports and devices from multiple angles.
- High Weight Capacity: Supports up to 300 lbs on the floor (200 lbs when adjusted to maximum depth) and 200 lbs when wall-mounted (depth cannot be adjusted in wall-mounted mode). Made from carbon steel for superior welding performance and durability, this open frame rack is designed to save space while accommodating multiple devices.
- User-Friendly Design: Designed with your convenience in mind, this open frame server rack features an top shelf for extra storage and improved space utilization. The rolling casters let you move it effortlessly wherever you need it, making setup and movement a breeze.
- Widely Applicable: Maximize your space with this adaptable open frame server rack, designed to make the most of every inch. Ideal for retail spots, classrooms, offices, and any area where space is at a premium, it delivers practical solutions for your storage needs.
- Everything You Need: Our open-frame rack comes with fully equipped accessory kit for easy setup and secure installation: 2 x Trays, 4 x Casters, 1 x set of Screws, 16 x M6*12 Cage Nuts, 1 x Grounding Wire, 1 x Internal & External Hex Wrenches, and 1 x User Manual.
What crawler operators need to do
- Use the crawler’s standard user agent and be onboarded and identified through Cloudflare’s Web Bot Auth process.
- Implement Cloudflare’s supported payment-intent and pricing headers; this is not simply a matter of seeing a 402 response and retrying blindly.
- Set spending limits and deduplication or caching controls. Repeat retrievals may each incur a charge, so crawler operators are responsible for managing their own budgets.
- Account for the fact that error responses are not billed, while a successful retrieval generally is.
These requirements are described in Cloudflare’s Pay Per Crawl FAQ. An AI company can also decide that the price or integration cost is not worthwhile and stop accessing the site.
Limits and risks to understand
Payment is not guaranteed revenue
A site earns only if an eligible crawler accesses content and pays. At the documented floor of $0.001 per successful retrieval, one million such retrievals would total $1,000 before any applicable commercial terms or adjustments. The actual result depends on price, paid participation, content value, repeated fetches, and whether crawlers choose to leave. Per-fetch payments are not a replacement by themselves for subscriptions, advertising, referrals, or negotiated licensing.
Charging does not guarantee continued visibility
Allowing content may support indexing or inclusion in an AI service; charging asks the crawler to pay for access, and blocking denies it. If a crawler declines the fee, the publisher receives no payment and may also lose exposure in that product. A request-based fee pays for retrieval, not for a particular AI answer, attribution, or value generated downstream.
Rank #4
- ADJUSTABLE DEPTH: 4-Post 42U open frame server rack with 4 vertical rails and adjustable mounting depth 22" to 40" (56,0cm to 101,7cm); Compatible with various servers / switches / data / AV and other IT equipment; EIA/ECA-310-E Compliant
- EASY ASSEMBLY: Mobile network rack with easy-to-follow assembly instructions and online video; Compact flat-pack shipping to avoid damage and facilitate installation; Total product height of 80.3in (204 cm) with casters, 78in (198cm) without casters
- COLD ROLLED STEEL: Durable 4 Post 19in open frame rack designed for ventilation with 42U mounting height and 1320lb (600kg) weight capacity (stationary); 3 install options included: casters, levelling feet, or base-plate to secure rack to the floor
- HARDWARE INCLUDED: Rolling computer/data rack includes cage nuts and screws to mount equipment, easy to read Units (U) and depth adjustment markings, cable management hooks for organization, and required assembly tools
- THE IT PRO'S CHOICE: Designed and built for IT Professionals, this 42U rack is backed for 2-years, including free lifetime 24/5 multi-lingual technical assistance
Identification and security rules matter
The system depends on recognizing the crawler. Cloudflare describes its detection as using machine learning, behavioral analysis, fingerprinting, and network visibility, but that is a product claim rather than an independently established accuracy guarantee. Misidentification, disguised bots, or third-party fetchers can complicate policy enforcement. Cloudflare also says applicable WAF or Bot Management blocks override a Charge action: a security rule can prevent delivery instead of allowing a paid retrieval. Review the interaction among AI Crawl Control, WAF, Bot Management, robots.txt, caching, and origin behavior, and test a limited set of paths before applying a policy broadly. Cloudflare’s product description is at AI Crawl Control.
Beta terms leave commercial questions open
As of August 18, 2026, Pay Per Crawl remains closed beta. Publicly established information does not settle typical publisher earnings, adoption by major AI services, dispute handling, payout terms, Cloudflare fees, or the long-term effect on referrals. Cloudflare’s documentation establishes a mechanism, not proof that it will produce meaningful revenue for a particular site.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Allow, block, or charge? A publisher decision guide
| Choice | Best fit | Main trade-off |
|---|---|---|
| Allow | Sites seeking search visibility, AI product inclusion, referrals, or broad access to public and promotional material. | Automated access may consume content or infrastructure without comparable traffic or payment. |
| Block | Confidential, premium, user-generated, or legally sensitive content; sites that do not want AI use; or unidentified and suspicious crawlers. | Blocking earns nothing and can reduce discovery or inclusion in AI services. |
| Charge | Sites with original, costly or frequently updated material, meaningful crawler volume, and capacity to monitor results. | Crawlers may decline to pay; repeat retrievals can bill again; pricing and policy need careful operation. |
For a small site seeking guaranteed near-term income, Pay Per Crawl is a weak fit while it remains a beta and crawler participation is uncertain. If the goal is simply to express preferences, robots.txt is a low-cost signal but does not itself enforce payment or guarantee compliance. For premium archives or structured data where permitted uses and attribution matter, direct licensing or a paid API can specify terms more clearly, though it requires more legal and technical work. A conventional paywall is aimed at human readership and solves a different problem.
The Tool Desk
Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Best Value
- Standard 1U Height: Get more space with our 1U server rack shelf—it comes in a set of 4! Ideal for 19-inch 4-post server racks, stacking routers, switches, firewalls, and other network gear. Easy storage and a neat setup in one simple solution
- Heavy-Duty Construction: Crafted from premium Q235 carbon steel with a robust 0.06 in (1.5 mm) thickness, our network rack shelf can handle up to 50 lbs (22.68 kg) with ease. Say goodbye to wobbles and tilts—keeping everything in its place
- Optimal Ventilation: Featuring a vented bottom design, our rack mount shelf effectively reduces equipment temperature, ensuring stable operation and lowering the risk of malfunctions. Keep your gear running smoothly for longer-lasting performance
- Flexible Partitioning: Each shelf features a depth of 10 in (254 mm). Our server rack shelf helps you organize and optimize your rack space efficiently. Keep your equipment neatly separated to reduce clutter and minimize interference or collisions
- Installation Made Easy: Everything you need for installation is included—screws and nuts are provided, making the process quick and hassle-free. Simply use a Phillips screwdriver, and you'll have your network rack shelf installed in no time
What may change next
Cloudflare’s July 2026 announcement described testing new AI-traffic classifications and defaults, with September 15, 2026 named as a deadline for finalizing those defaults and classifications. As of August 18, that date was still in the future; the announced changes should not be treated as already active. The announcement is available at Cloudflare’s “Your Content, Your Rules” release.
Cloudflare has also discussed experiments with AI search companies and payment models that could go beyond charging per retrieval. Those are developing ideas, not the current Pay Per Crawl contract; see its July 2026 AI search post. Whether access fees become a useful income stream will depend on publishers enrolling, crawlers implementing the payment flow, and the economics of paid access compared with simply crawling elsewhere.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
