Cloudflare Pay Per Crawl test guide: what publishers and AI crawlers can verify in private beta

Last checked: 2026-07-15. Cloudflare Pay Per Crawl is still private beta. It currently gives a publisher three crawler controls—Allow, Charge, or Block—and a flat site-wide per-request price. Cloudflare has not published a standard Pay Per Crawl list price: the $0.002/request used later is a hypothetical publisher-configured amount for arithmetic, not a Cloudflare price.

Current Pay Per Crawl protocol snapshot

The official announcement documents this flow. Web Bot Auth and RFC 9421 supply the signed-request identity mechanism. Without beta access and configured payment details, a team can study these protocol fields but cannot claim a live paid-crawl deployment.

Actors and responsibilities

Crawler identity matters because a user-agent string can be spoofed. Cloudflare links the flow to Web Bot Auth and HTTP Message Signatures in RFC 9421. The announcement describes an Ed25519 key pair, a public key directory, registration of crawler identity, and signed requests containing signature-agent, signature-input, and signature headers.

Eligibility checklist

The publisher should confirm:

• the test zone is accepted into Pay Per Crawl private beta;
• a payout/payment configuration is available in the account;
• the test uses a hostname or content set safe for controlled crawling;
• the current flat site-wide price is understood;
• free, charged and blocked crawler policies are documented;
• log retention does not expose private content.

The crawler should confirm:

• the crawler identity is accepted and tied to a published key directory;
• every test request is signed as required;
• payment details are configured;
• the client can handle 402 without treating it as a generic retryable server error;
• only one of crawler-exact-price or crawler-max-price is sent per request;
• maximum spend and request count are capped.

Without beta acceptance, this remains a protocol review. Do not simulate revenue by sending unsigned public requests to an ordinary site.

Reactive flow

The crawler first requests a paid resource without price intent:

GET /test/article-1 HTTP/1.1
Host: beta.example.com
signature-agent: <registered crawler identity>
signature-input: <signature metadata>
signature: <request signature>

Cloudflare can respond:

HTTP/1.1 402 Payment Required
crawler-price: USD 0.002

The crawler decides whether to retry:

GET /test/article-1 HTTP/1.1
Host: beta.example.com
crawler-exact-price: USD 0.002
signature-agent: <registered crawler identity>
signature-input: <signature metadata>
signature: <request signature>

On accepted payment intent, the response can include:

HTTP/1.1 200 OK
crawler-charged: USD 0.002
server: cloudflare

The test log must connect the original 402 and paid retry without counting them as two paid content deliveries.

Proactive flow

A crawler can send its maximum acceptable price on the first request:

GET /test/article-2 HTTP/1.1
Host: beta.example.com
crawler-max-price: USD 0.003
signature-agent: <registered crawler identity>
signature-input: <signature metadata>
signature: <request signature>

If the configured publisher price is at or below the maximum and the request is otherwise valid, the response can succeed and report the actual configured charge. If the configured price is above the crawler maximum, the response remains 402 and reports the required price. The maximum is not permission to charge more than the publisher's configured amount.

Worked test: reconcile 100 requests

A technical publisher and research crawler have beta access. For this hypothetical test—not an official Cloudflare price—the publisher creates a test hostname and sets its own flat price of $0.002 per request. The plan caps the run at 100 authenticated requests and $0.20 of expected successful charges.

The dataset contains:

• 20 proactive requests with a maximum below $0.002; expected 402, no content and no charge;
• 30 reactive requests that receive 402 and retry with the exact price; expected 30 paid 200-level responses;
• 50 proactive requests with a maximum at or above $0.002; expected 50 paid 200-level responses;
• a separate excluded crawler identity tested under the publisher's free or block policy, not mixed into the paid 100.

The crawler log records timestamp, URL, request ID, identity, price-intent header, response status, crawler-price, crawler-charged, response bytes and retry linkage. The publisher log records the same request ID and configured policy. Reconciliation expects 80 paid content deliveries at $0.002, or $0.16, before any account-specific settlement timing or adjustments.

This arithmetic is a test expectation, not a claim about Cloudflare fees or payout timing. The final evidence is the Cloudflare account record matched to the request log.

Failure samples

Failure one: a signed request receives 200 content but lacks crawler-charged. Do not assume it was billed. Mark it unresolved and compare the configured crawler exception policy.

Failure two: an unsigned client copies the crawler user agent and receives the same paid treatment. Stop the test and investigate identity configuration; user-agent matching alone is not the intended trust mechanism.

Failure three: the crawler sends both exact and maximum price headers. Cloudflare says only one price declaration header may be used. Treat a rejected or ambiguous request as invalid, not as a billing event.

Failure four: publisher and crawler logs show 80 successful paid responses, but the account record shows a different count. Stop expansion until request IDs and settlement records reconcile.

What the beta does not yet prove

The announcement says publishers currently define a flat price across the entire site. It discusses more dynamic pricing as future work, so do not advertise path-level demand pricing as current behavior. Private beta also means onboarding, interfaces and settlement details can change. A 404 on a guessed developer-doc route is another reason to cite the current announcement and its linked authentication material instead of inventing a stable public API surface.

Pay Per Crawl addresses crawler access. It does not automatically establish copyright rights, content licensing terms, data protection compliance or downstream model-use restrictions. Those need separate agreements and review.

Pause conditions

Pause when either party lacks beta access, crawler identity cannot be verified, payment details are incomplete, price headers disagree, paid events cannot be tied to 200-level responses, or account settlement cannot be reconciled. Also pause if the test content is private or the crawler could exceed the request/spend cap.

FAQ

Is Cloudflare Pay Per Crawl generally available?

The official announcement says it is in private beta as of this check. Teams without accepted access can review the protocol but cannot claim a live paid-crawl deployment.

Does every 402 create a charge?

No. A 402 communicates that payment is required. Cloudflare describes billing events for authenticated requests with payment intent that receive successful 200-level content and a crawler-charged header.

Can publishers set a different price for every path?

The announcement says the current control is a flat site-wide per-request price. More dynamic pricing is discussed as future work.

What should a team verify first?

Beta eligibility, crawler identity, publisher policy, request price headers, successful charged responses and reconciliation between both logs and the Cloudflare account.

Related reading

• AI news sources for builders
• China AI news sources in English
• RadarAI editorial standards