Should we try to change the world together? ~ Returning 402 Payment Required to AI crawlers. The "third option" realized through the x402 protocol and GEO strategy

This is Shigahi from Berlin. I was busy with DIY projects during the German Easter holidays.

Rather than assembling shelves, I was renovating my community project Berghain Klubnacht Complete Database. This is a structured database containing over 10,000 DJ performance records from Berlin's club Berghain spanning 16 years, running on Cloudflare Workers + D1 + KV + R2. It's a complete public resource with no ads, no cookies, and no trackers—not even Google Analytics. I've implemented clever techniques like multi-layer KV caching to reduce D1 raw reads, operating on the Cloudflare Workers Free plan (Thank you Cloudflare. I'll pay you back someday when I make it big).

This time, I implemented AI crawler monetization with the x402 protocol for this project, and then switched to a freemium + GEO strategy. I've published the implementation details in two articles on Zenn.

• Charging only AI crawlers: Implementing an AI paywall on Cloudflare Workers with Hono + x402
• The x402 paywall was too early: Switching to a freemium strategy that "lets AI read first"

In this article, I'll introduce Cloudflare's problem statement that motivated this project, then focus on the technical comparison of "systems for actually receiving money" that I couldn't fully cover in the Zenn articles.

Catalyst: AI Crawlers Breaking Caches

It all started with Cloudflare's blog post from March 2026: Why we're rethinking cache for the AI era.

32% of all traffic across the Cloudflare network is automated traffic, with AI bots alone generating over 1 billion requests per week. The core issue is that AI crawlers have fundamentally different access patterns than humans. Humans concentrate on popular content like homepages and rankings, while AI crawlers hit massive amounts of unique URLs across the long tail. Traditional CDN cache algorithms (like LRU) are designed on the premise of "keep frequently accessed content in cache," so the flood of unique requests from AI crawlers ends up evicting content that was cached for humans—creating a structural problem where cache quality for humans is degraded by AI crawling.

Cloudflare proposed solutions including a new architecture with separated cache layers for AI, AI Crawl Control to let site owners manage AI traffic, and Pay Per Crawl (discussed later).

This was a pressing issue for my project as well. After climbing to the top of Google search results for Berghain-related keywords through SEO efforts, AI crawler traffic surged. Within just 3 hours of starting to log to KV, I received 204 requests from 7 different AI crawlers: Amazonbot (184 requests), Meta-ExternalAgent (11 requests), PerplexityBot, GPTBot, ChatGPT-User, Claude-Web, ClaudeBot... all major AI companies were visiting. For a personal project on the Free plan, this directly impacts request limits (I've actually hit rate limits before).

I could block them with robots.txt to reduce load, but that would also mean losing citation opportunities in AI search. Doing nothing would let them extract data for free. Rather than choosing between blocking and allowing, I decided on a third option: requesting fair compensation through HTTP-compliant methods.

My DIY Project — 3-Line Summary

1. Implemented a paywall using the x402 protocol to charge only AI crawlers. OpenAI's crawlers (GPTBot, ChatGPT-User, OAI-SearchBot) actually paid through x402, proving that the "charge AI" concept works.
2. However, since only OpenAI crawlers supported x402, a full paywall killed visibility in AI search. After 3 days, I pivoted to a freemium model (only bulk exports are paid) + GEO/AEO strategy.
3. Added x402 signal headers to the free routes, advertising "it's free now, but we support x402," preserving AI search visibility while laying groundwork for future paywalls.

For implementation details (Hono middleware code, detection of 80+ crawlers, freemium design, GEO support with llms.txt, content negotiation, agentic-capabilities.json, etc.), please refer to the Zenn articles linked above. Broadly speaking, anyone can implement this using Hono.
From here on, I'll dive into the technical background of "facilitators that let you actually receive cash from AI crawlers."

Two Practical Solutions for AI Crawler Monetization

As of April 2026, there are two main systems that let you actually receive payments from AI crawlers as cash:

These two take fundamentally different approaches. x402 is an open protocol with cryptocurrency payments, while Pay Per Crawl is a fiat currency payment system integrated into the Cloudflare platform.

x402 + Coinbase CDP Facilitator

x402 Protocol Flow

x402 is an open protocol using the 402 Payment Required status code for machine-to-machine payments.

1. Client → Server: GET /api/export/artists
2. Server → Client: 402 + payment requirements (amount, wallet, network)
3. Client: Signs USDC transfer transaction on Solana
4. Client → Server: GET /api/export/artists + PAYMENT-SIGNATURE header
5. Server → Facilitator: /verify (signature verification)
6. Facilitator → Server: Verification result
7. Server: Returns content (200 OK)
8. Server → Facilitator: /settle (execute on-chain payment)
9. Facilitator: Submits transaction to blockchain → confirmation

Importantly, the facilitator is non-custodial, meaning it doesn't hold funds. It only verifies and submits client-signed transactions. Funds move directly from the client's wallet to the server's wallet.

Facilitator Options

Multiple x402 facilitators exist, but currently only the Coinbase CDP (Cloud Developer Platform) facilitator can actually receive USDC on Solana mainnet.

Since the x402.org facilitator only supports devnets, you need to switch to the Coinbase CDP facilitator to move real money.

Supported Networks and Tokens

The x402 protocol itself supports a wide range of networks.

EVM-based:

• Base mainnet (eip155:8453) / Sepolia (eip155:84532)
• Polygon mainnet (eip155:137) / Amoy (eip155:80002)
• Avalanche C-Chain (eip155:43114) / Fuji (eip155:43113)
• Sei, SKALE

Non-EVM:

• Solana mainnet (solana:5eykt4UsFv8P8NJdTREpY1vzqKqZKvdp) / Devnet
• Aptos mainnet (aptos:1) / Testnet
• Stellar mainnet (stellar:pubnet) / Testnet

The default payment token is USDC. On EVM chains, it uses ERC-20 tokens (via EIP-3009 or Permit2), while on Solana it uses SPL tokens.

Why Solana — Why EVM Chains Don't Work for x402

While the x402 protocol technically supports EVM-based L2s like Base, Polygon, and Avalanche, Solana is the only realistic choice for AI crawler payments.

The x402 payment flow requires on-chain transactions for every crawler request. You need to process micropayments of $0.01-$0.10 at crawler request speeds (seconds). This is where EVM chains have structural problems:

While EVM L2s are significantly faster and cheaper than Ethereum mainnet, they're still too heavy for handling micropayments at high speed like x402 requires. Even with EIP-3009 (gasless transfers) or Permit2, the transaction building → signing → confirmation cycle is more complex than Solana's SPL token transfers. Solana completes signing, submission, and confirmation in a single transaction in 400ms, making the "crawler sends HTTP request → receives 402 → pays → re-requests" flow happen with almost imperceptible delay.

It's no coincidence that OpenAI's crawlers, which currently support x402, pay with Solana USDC. For micropayment practicality, Solana is currently the optimal solution.

Coinbase CDP Facilitator Authentication

Requests to the CDP facilitator require ES256 JWT authentication.

// Coinbase CDP API authentication
const jwt = await createCDPJWT(
  env.CDP_API_KEY_ID,
  env.CDP_API_KEY_PRIVATE_KEY
);

const facilitatorClient = new HTTPFacilitatorClient(
  "https://api.cdp.coinbase.com/x402/facilitator",
  { createAuthHeaders: async () => ({ Authorization: `Bearer ${jwt}` }) }
);

CDP API Keys are available for free from the Coinbase Developer Platform. You can enable production use simply by setting CDP_API_KEY_ID and CDP_API_KEY_PRIVATE_KEY in Wrangler secrets.

Actual Implementation Cost

The implementation cost for x402 + CDP facilitator is effectively zero:

• x402 protocol: OSS (MIT license)
• @x402/hono package: Free
• Coinbase CDP API: Free (just need API Key)
• Solana network fees: ~$0.00025 per transaction
• Cloudflare Workers: Works on Free plan

The only cost is Solana transaction fees, which are paid by the facilitator, so the server-side cost is literally zero.

Challenge: Converting USDC to Japanese Yen

With x402, you receive USDC on Solana. To deposit Japanese yen into a bank account, you need to go through a crypto exchange, but currently no Japanese exchanges handle USDC on Solana. This requires DeFi conversion (to SOL or JPYC) or chain bridges, which might be the biggest hurdle for individual developers or corporations.

Cloudflare Pay Per Crawl

Overview

Cloudflare Pay Per Crawl is a feature of AI Crawl Control announced in July 2025 as a Private Beta. Cloudflare acts as the Merchant of Record between site owners and crawler owners, facilitating fiat currency payments through Stripe.

1. AI Crawler → Cloudflare Edge: GET /article/...
2. Cloudflare Edge: Checks Pay Per Crawl settings
3. Cloudflare Edge → AI Crawler: 402 + crawler-price: USD 0.01
4. AI Crawler → Cloudflare Edge: GET /article/... + crawler-exact-price: USD 0.01
   (+ Web Bot Auth signature headers)
5. Cloudflare Edge: Verifies signature + price matching
6. Cloudflare Edge → Origin: Forwards request
7. Origin → Cloudflare Edge → AI Crawler: 200 OK + crawler-charged: USD 0.01
8. Cloudflare: Pays site owner via Stripe

Crucial Differences from x402

Pay Per Crawl is a completely different protocol from x402. While both use HTTP 402, their header specifications differ.

x402:

# Response (Server → Client)
HTTP/1.1 402 Payment Required
X-PAYMENT: {"accepts":[{"scheme":"exact","payTo":"<wallet>","price":"$0.01","network":"solana:mainnet"}]}

# Request (Client → Server, with payment)
PAYMENT-SIGNATURE: <Solana transaction signature>

Pay Per Crawl:

# Response (Cloudflare Edge → Crawler)
HTTP/2 402
crawler-price: USD 0.01

# Request (Crawler → Cloudflare Edge, payment intent)
crawler-exact-price: USD 0.01
# or
crawler-max-price: USD 0.05

x402 places on-chain transaction signatures in headers, while Pay Per Crawl only includes payment intent in headers—actual charging is handled afterward by Cloudflare + Stripe. No cryptocurrency knowledge is required.

Site Owner Implementation

Implementing Pay Per Crawl requires zero code—just dashboard configuration:

1. Cloudflare Dashboard → AI Crawl Control → Settings
2. Enable Pay Per Crawl
3. Set default price (minimum $0.01/crawl)
4. Configure actions per crawler (Charge / Allow / Block)
5. Connect Stripe for payouts

Pricing is zone-wide by default, but as of December 2025, custom pricing (per-path settings) was added.

AI Crawler Owner Requirements

Crawlers using Pay Per Crawl need:

1. Cloudflare Account: Crawler owners also need Cloudflare accounts
2. Web Bot Auth: HTTP Message Signatures (RFC 9421 compliant) for crawler authentication, including payment headers in signature components
3. Cloudflare Verified Bot Registration: Pass Cloudflare's review to get on the Verified Bot list
4. Stripe Connection: Connect to Stripe via Cloudflare account to accept charges

Thus, Pay Per Crawl is not an "open protocol anyone can implement" like x402, but a managed service within the Cloudflare ecosystem.

Enterprise Strengths

Pay Per Crawl's true value is for enterprise users:

For organizations like newspapers and major media outlets, managing cryptocurrency wallets, converting USDC, and accounting for crypto assets are all hurdles. With Pay Per Crawl, payments process as Stripe revenue, fitting into existing accounting workflows.

Also, Web Bot Auth (HTTP Message Signatures, RFC 9421) provides cryptographically signed crawler authentication that prevents User-Agent spoofing. Pay Per Crawl offers crawler identity verification at the platform level, which is impossible with x402's User-Agent detection.

Challenges: Closed Beta and Cloudflare Lock-in

As of April 2026, Pay Per Crawl is in Closed Beta. To use it, you need to join the waitlist (Pay per crawl signup) or have an Enterprise contract.

Additionally, it requires a custom domain linked to a Cloudflare Zone, so it can't be used with subdomains like *.workers.dev. This was one reason why my project chose x402 over Pay Per Crawl.

Which Should You Choose?

Individual Developers/OSS: x402 + CDP facilitator. No code needed, $0 cost, no custom domain required, can start today. Converting USDC is a hassle, but it's ideal for micropayment experiments.

Enterprise/Media: Cloudflare Pay Per Crawl. Fiat currency payments via Stripe, robust authentication with Web Bot Auth, no code changes needed. However, it's in Closed Beta, requires a custom domain, and depends on the Cloudflare platform.

Both: Technically, you can use both. Pay Per Crawl is processed at the Cloudflare Edge, while x402 is processed at the application layer, so they don't conflict.

AI Crawler Transparency Dashboard

To support these implementations, I've published a dashboard that completely visualizes AI crawler traffic to the world:

https://berghain.ravers.workers.dev/dashboard/crawl-stats

It shows real-time detection status of 80+ AI crawlers, x402 payment records, blocking status, and revenue. I also provide a public API (/api/public/crawl-summary) without authentication, so anyone can access the data.

Who is crawling the web, and who is paying for it? This transparency report visualizes the ethics of GenAI companies.

The World Can Be Changed

To be honest, Berghain Klubnacht Complete Database is just a niche site, even if it ranks high in search. However, this project has proven that x402 actually works. OpenAI's crawlers have passed through the virtual noren (shop curtain) that displays the x402 protocol. I implemented it on a personal site running on the Cloudflare Workers Free plan.

Imagine all the people asking money against AI crawlers... What if newspapers and news media all implemented Pay Per Crawl? What if academic paper databases implemented x402?

AI companies are currently essentially extracting web content for free. robots.txt is just a gentlemen's agreement with no enforcement. But HTTP 402-based charging is different. x402 as an open protocol and Pay Per Crawl as Cloudflare's managed service are taking different approaches to solve the same problem.

HTTP 402 was a status code "reserved for future use" for 27 years. That "future" is now here. What's needed is for content providers to speak up. One niche site putting up a noren only got a response from OpenAI, but if the entire web raised HTTP 402 norens—AI companies would have no choice but to comply.

Let AI read your data. But not for free.

Detailed Implementation Articles (Zenn):

• Charging only AI crawlers: Implementing an AI paywall on Cloudflare Workers with Hono + x402
• The x402 paywall was too early: Switching to a freemium strategy that "lets AI read first"

Related Links:

• Berghain Klubnacht Complete Database: https://berghain.ravers.workers.dev
• AI Crawler Transparency Dashboard: https://berghain.ravers.workers.dev/dashboard/crawl-stats
• x402 Protocol: https://www.x402.org/
• Cloudflare Pay Per Crawl: https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/what-is-pay-per-crawl/