Web Bot Auth: Sign Agents to Skip CAPTCHA Walls

A hands-on guide to building a Web Bot Auth flow so your AI agent cryptographically proves it is a real, identified bot instead of getting CAPTCHA-walled.

What Web Bot Auth signed agents actually solve

Web Bot Auth signed agents attach a cryptographic signature to every outbound HTTP request so a website can verify the bot’s identity from a published public key, instead of guessing from a spoofable User-Agent string or an IP allowlist that breaks the moment the agent moves behind a new proxy. The result is the difference between an agent that sails through and one that hits a CAPTCHA wall it can never solve.

The mechanism is not a new invention. Web Bot Auth signed agents are built on RFC 9421 (HTTP Message Signatures), the IETF standard for signing HTTP messages, plus a Web Bot Auth architecture draft and a key-directory draft from Cloudflare. The IETF formally chartered the Web Bot Auth (webbotauth) working group after a BoF session at IETF 123, co-chaired by David Schinazi and Rifaat Shekh-Yusef, to standardize cryptographic authentication of automated clients.

Why does this matter right now? Legacy bot identification has three failure modes, and cryptographic verification is the only rail that survives all of them. User-Agent strings are trivially spoofed. IP allowlists go stale and punish agents that rotate egress IPs. Reverse-DNS checks are slow and frequently misconfigured. A signature does not care about the source IP, does not trust the UA, and needs no reverse-DNS lookup. It either verifies against the operator’s published key or it does not.

By the time you finish this tutorial you will have generated an Ed25519 key pair, published a signed JWKS-style key directory at a well-known path, signed a real outbound request with `Signature` and `Signature-Input` headers plus a `Signature-Agent` header pointing at your directory, and run a verifier that accepts the signed agent. That is the entire trust loop.

Step 1: Generate an Ed25519 key and compute the keyid

Start by generating an Ed25519 signing key and converting the public half to a JWK, because Web Bot Auth signed agents use the base64url JWK thumbprint of that public key as the keyid that ties every signature back to your directory. Ed25519 is the algorithm Cloudflare’s verifier supports today, so do not reach for RSA or P-256 here.

Cloudflare’s documentation walks through the OpenSSL path, then converts the PEM public key into a JWK with the `jwker` tool. The JWK for an Ed25519 key uses key type `OKP`, curve `Ed25519`, and an `x` parameter holding the base64url public key.

The keyid is not arbitrary. It must equal the RFC 7638 JWK thumbprint (base64url-encoded) of your public key. Compute it once and reuse it as the `kid` in your directory and the `keyid` in every signature. If the two ever disagree, the verifier cannot find your key and the request fails closed.

# 1. Generate an Ed25519 private key
openssl genpkey -algorithm ed25519 -out private-key.pem

# 2. Extract the matching public key
openssl pkey -in private-key.pem -pubout -out public-key.pem

# 3. Convert the public key to JWK format
go install github.com/jphastings/jwker/cmd/jwker@latest
jwker public-key.pem public-key.jwk

# public-key.jwk now looks like:
# {"kty":"OKP","crv":"Ed25519","x":"JrQLj5P_89iXES9-vFgrIy29clF9CC_oPPsw3c5D0bs"}

Step 2: Publish a signed key directory at the well-known path

Host a JWKS document containing your public key at /.well-known/http-message-signatures-directory over HTTPS, and sign the directory response itself so verifiers can confirm you control the keys you are publishing. This directory is the anchor the whole flow depends on: your `Signature-Agent` URL resolves to it, and a verifier fetches it to get the public key matching your keyid.

The directory is a JSON Web Key Set: a top-level `keys` array of JWKs. Serve it with the content type application/http-message-signatures-directory+json. The critical, easily-missed detail is that the directory response is itself signed, but with a different tag, http-message-signatures-directory, and it covers the @authority component with the req flag. That self-signature proves the host serving the keys also holds the corresponding private key.

In production this is a static file behind a CDN with a long cache TTL, plus the signing headers regenerated whenever the `expires` window lapses. A Cloudflare Worker, an S3 object fronted by CloudFront, or a single nginx location all work. Keep it boring and highly available, because if the directory is unreachable, your signatures cannot be verified.

// GET https://signature-agent.example/.well-known/http-message-signatures-directory
// Content-Type: application/http-message-signatures-directory+json
{
  "keys": [
    {
      "kty": "OKP",
      "crv": "Ed25519",
      "kid": "poqkLGiymh_W0uP6PZFw-dvez3QJT5SolqXBCW38r0U",
      "x": "JrQLj5P_89iXES9-vFgrIy29clF9CC_oPPsw3c5D0bs"
    }
  ]
}

Step 3: Sign an outbound request with the web-bot-auth library

Use Cloudflare’s open-source web-bot-auth npm package to build the Signature and Signature-Input headers, because it implements the RFC 9421 signing base correctly and computes the keyid thumbprint for you. The reference implementation lives at github.com/cloudflare/web-bot-auth and ships both TypeScript packages and Rust crates for signing and verifying Web Bot Auth signed agents.

The TypeScript side exposes a `signatureHeaders` helper plus an Ed25519 signer built from your JWK. You give it the request, the signer, and `created`/`expires` timestamps; it returns the `Signature` and `Signature-Input` values. You add the `Signature-Agent` header yourself, set to the quoted HTTPS URL of your directory. Keep the `expires` window short: Cloudflare’s guidance notes that a lifetime of about a minute is often sufficient, and a short window is your replay defense.

The companion packages are worth knowing: http-message-sig implements the raw RFC 9421 layer, and jsonwebkey-thumbprint implements the RFC 7638 thumbprint so your keyid is computed consistently on both ends.

import { signatureHeaders, Ed25519Signer } from "web-bot-auth";

// Your Ed25519 private JWK (kty=OKP, crv=Ed25519, with the `d` param)
const privateJwk = {
  kty: "OKP",
  crv: "Ed25519",
  kid: "poqkLGiymh_W0uP6PZFw-dvez3QJT5SolqXBCW38r0U",
  d: "<base64url-private-scalar>",
  x: "JrQLj5P_89iXES9-vFgrIy29clF9CC_oPPsw3c5D0bs",
};

async function signedFetch(url) {
  const request = new Request(url, { method: "GET" });
  const signer = await Ed25519Signer.fromJWK(privateJwk);

  const now = Math.floor(Date.now() / 1000);
  const headers = await signatureHeaders(request, signer, {
    created: new Date(now * 1000),
    expires: new Date((now + 60) * 1000), // 60s lifetime
  });

  return fetch(url, {
    headers: {
      "Signature": headers.Signature,
      "Signature-Input": headers["Signature-Input"],
      // points verifiers at your published JWKS directory
      "Signature-Agent": '"https://signature-agent.example"',
    },
  });
}

What the wire actually looks like

A correctly signed Web Bot Auth request puts three headers on the wire, and a verifier reconstructs the same signing base from the listed components to check the signature. Seeing the raw headers makes the abstraction concrete and makes debugging far faster when something is off by one component.

The `Signature-Input` line names the covered components in order. For an agent request that is typically ("@authority" "signature-agent"), meaning the signature commits to the host you are talking to and the directory you claim. That binding is what stops a captured signature from being replayed against a different origin or pointed at a different directory.

Note the structure: `sig` is just a label, `created` and `expires` are Unix timestamps, `keyid` is your JWK thumbprint, `alg` is `ed25519`, and `tag` declares intent as `web-bot-auth`. The `Signature` value is the base64 signature wrapped in colons per RFC 9421 byte-sequence encoding.

GET /products HTTP/1.1
Host: shop.example
Signature-Agent: "https://signature-agent.example"
Signature-Input: sig=("@authority" "signature-agent");\
  created=1735689600;\
  expires=1735689660;\
  keyid="poqkLGiymh_W0uP6PZFw-dvez3QJT5SolqXBCW38r0U";\
  alg="ed25519";\
  tag="web-bot-auth"
Signature: sig=:jdq0SqOwHdyHr9+r5jw3iYZH6aNGKijYp/EstF4RQTQdi5N5YYKrD+mCT1HA==:

Step 4: Verify the signed agent on the server

On the receiving side, parse the Signature-Input, fetch the JWKS from the Signature-Agent URL, select the key whose thumbprint matches the keyid, then verify the signature and enforce the created/expires window. The web-bot-auth package exposes a `verify` function plus a `verifierFromJWK` helper so you can accept a signed agent in a few lines, whether you run on Cloudflare Workers, Caddy via the repo’s plugin, or plain Node.

Verification is fail-closed by design. If the directory is unreachable, the keyid is unknown, the `expires` timestamp has passed, or the bytes do not match, you reject and fall back to your normal bot policy, which may include the CAPTCHA you were trying to avoid. The signature is an upgrade path, not a bypass: an unsigned or invalid request simply gets treated as it would have been before.

In practice you do not have to build the verifier yourself. Cloudflare’s Verified Bots Program now accepts Message Signatures, so agents authenticate cryptographically instead of via IP allowlists or User-Agent strings, and applications with well-formed Message Signatures are prioritized for approval. AWS WAF, Akamai, and HUMAN Security verify the same signatures. But running your own verifier is the best way to understand the trust model, and it is what you will reach for when you operate the origin.

import { verify, verifierFromJWK } from "web-bot-auth";

// Minimal origin-side check (Workers / Node)
async function acceptSignedAgent(request) {
  const agentUrl = request.headers
    .get("Signature-Agent")
    ?.replaceAll('"', "");
  if (!agentUrl) return false;

  // 1. Fetch the operator's signed JWKS directory
  const dir = await fetch(
    `${agentUrl}/.well-known/http-message-signatures-directory`
  ).then((r) => r.json());

  // 2. Build a verifier per published key, keyed by thumbprint
  const verifiers = await Promise.all(
    dir.keys.map((jwk) => verifierFromJWK(jwk))
  );

  try {
    // 3. verify() checks the signature AND the created/expires window
    await verify(request, async (params) =>
      verifiers.find((v) => v.keyid === params.keyid)
    );
    return true; // signed agent accepted -> skip the CAPTCHA
  } catch {
    return false; // fail closed -> normal bot policy applies
  }
}

Where signed agents already get you through the door

The same Web Bot Auth signature is now recognized across bot managers, payment networks, and at least one major search engine, so the engineering you just did is reusable far beyond a single origin. This is the strategic argument for adopting it early: sign once, get identified in many places, and stop maintaining brittle per-vendor allowlists.

On the infrastructure side, Cloudflare’s Verified Bots Program, AWS WAF, Akamai, and HUMAN Security all verify HTTP Message Signatures. Amazon Bedrock AgentCore Browser documents Web Bot Auth explicitly to reduce CAPTCHAs for browser agents; the feature is in preview, disabled by default, and enabled at browser creation, after which AgentCore automatically signs each request and attaches the verification headers. Vercel, Shopify, and other platforms have moved to support it as well.

On the commerce side, Web Bot Auth is the authentication foundation for Visa’s Trusted Agent Protocol and Mastercard’s Agent Pay. Both require agents to register a public key in a well-known directory referenced by the keyid, then sign requests so merchants and networks can verify them, ensuring each commerce request is verifiable, time-based, and non-replayable. And in search, Google has been testing Web Bot Auth to verify AI agent requests, a strong signal that cryptographic agent identity is becoming table stakes rather than a niche bot-management feature.

The verdict on shipping Web Bot Auth signed agents

If your agent makes outbound HTTP requests at any meaningful volume, implement Web Bot Auth signed agents now, because the standard has graduated from a Cloudflare experiment to an IETF-chartered effort with adoption across WAFs, browser runtimes, payment networks, and search. The implementation cost is one Ed25519 key, one signed directory file, and a few lines of signing code, against a payoff of provable identity that travels everywhere.

The honest caveat is that this is still early infrastructure. The reference library is not formally audited, Ed25519 is the only supported algorithm, and not every origin verifies signatures yet, so you will not eliminate every CAPTCHA on day one. Treat signing as a fail-open upgrade layered on top of your existing bot handling, never as the only path. But the trajectory is unambiguous: the era of identifying bots by guessing from headers and IP ranges is ending, and cryptographic identity is what replaces it.

Builder’s take

I have shipped agents that crawl and transact, and the single most wasteful failure mode is a perfectly legitimate bot getting hard-blocked because it could not prove who it was. Web Bot Auth signed agents fix the root cause rather than the symptom.

• The User-Agent string was never an identity. Treat it as a comment, not a credential. The day you ship Ed25519 signing, you stop arguing with bot managers and start presenting proof.
• Keep your `expires` window tight. I sign with a 60-second lifetime; a leaked signature that is already dead is a non-event. Replay protection is free when the signature is short-lived.
• The key directory is the part teams under-build. It is a static, signed JWKS file at a well-known path. Cache it, sign it, rotate it, and version your `kid`s so you can roll keys without an outage.
• Do not wait for one mega-standard. The same signature satisfies Cloudflare’s Verified Bots, AWS WAF, Akamai, and the Visa and Mastercard agent-commerce rails. Sign once, get recognized in many places.

Frequently asked questions

Primary sources

• Forget IPs: using cryptography to verify bot and agent traffic — Cloudflare
• Message Signatures are now part of our Verified Bots Program — Cloudflare
• Web Bot Auth — bot verification reference — Cloudflare Docs
• cloudflare/web-bot-auth — sign and verify orchestrated HTTP requests — GitHub
• Web Bot Auth (webbotauth) working group — IETF Datatracker
• Reduce CAPTCHAs for AI agents with Web Bot Auth (Preview) in Amazon Bedrock AgentCore Browser — AWS
• Securing agentic commerce: helping AI Agents transact with Visa and Mastercard — Cloudflare
• Google Testing Web Bot Auth To Verify AI Agent Requests — Search Engine Journal

Last updated: June 1, 2026. Related: Identity Provenance.