Your agent is signed, your keys are published, and Cloudflare still throws 403. Here is the symptom-to-fix checklist the docs never give you, with exact header strings.
Why is Cloudflare blocking my AI agent even though it is signed?
Cloudflare is blocking your AI agent because a signed request is not the same as a verified or allowed request — a 403 almost always means your Web Bot Auth signature was malformed, expired in transit, or signed by a key Cloudflare does not yet recognize, OR your identity verified fine but the site’s bot rules still challenge unverified automated traffic. Those are two separate problems, and most builders waste hours debugging cryptography when the real issue is that their key was never added to Cloudflare’s directory.
Web Bot Auth is Cloudflare‘s implementation of HTTP Message Signatures (RFC 9421) for bots and agents. Your agent attaches three headers — Signature, Signature-Input, and Signature-Agent — and Cloudflare verifies the Ed25519 signature against a public key it discovers from your published JWKS directory. When everything lines up, the request is treated as a verified, signed agent. When any one detail is off, you get a generic 403 Forbidden with no explanation of which step failed.
This guide maps the five concrete failure modes to exact fixes. Work through them in order — they are sorted from ‘cheapest to check’ to ‘requires a Cloudflare application.’ If Cloudflare is blocking your AI agent and you only read one thing, read failure #4: most ‘my signature is perfect but I still get 403’ cases are simply not being in the signed-agents directory yet.
Send your signed request to https://crawltest.com/cdn-cgi/web-bot-auth. It returns HTTP 200 if your message is verified and the key is known, HTTP 401 if the message is well-formed but the key is unknown, and HTTP 400 if the message itself is malformed. That single call tells you whether you have a formatting bug (400), a key/directory bug (401), or a clean signature (200).
Failure 1: Signature-Agent is not an https:// value or not double-quoted
If your Signature-Agent header is not an https:// URI, or the URI is not enclosed in double quotes, Cloudflare fails verification — because Signature-Agent is a structured field and an unquoted or non-https value is invalid by definition. This is the single most common formatting mistake, and it produces a 403 (or HTTP 400 on crawltest) that looks identical to every other failure.
The Signature-Agent header tells Cloudflare where to fetch your public key directory (your JWKS). It must point at an https:// URL and, because it is an RFC 8941 structured field, the string value must be wrapped in literal double quotes inside the header. A bare hostname, an http:// value, or an unquoted URL all fail.
Here is the difference, copy-pasteable:
Some HTTP libraries strip or normalize header quoting. Log the exact bytes you put on the wire before they are sent — not the value you passed to your client — and confirm the double quotes survive.
# BROKEN — no double quotes (structured-field violation)
Signature-Agent: https://signature-agent.test
# BROKEN — http:// instead of https://
Signature-Agent: "http://signature-agent.test"
# BROKEN — bare host, no scheme
Signature-Agent: signature-agent.test
# CORRECT — https URI, enclosed in double quotes
Signature-Agent: "https://signature-agent.test"Failure 2: signature-agent is missing from the Signature-Input component list
If you send a valid Signature-Agent header but do not list signature-agent in the component list inside Signature-Input, Cloudflare fails verification — the signature must cover the Signature-Agent header, or Cloudflare treats the message as unsigned over that field. A correct header value with the wrong component list still gets your AI agent blocked.
The Signature-Input header declares which parts of the request the signature covers, plus the signature metadata: the keyid (your Ed25519 key thumbprint), the created and expires Unix timestamps, and a tag that must equal “web-bot-auth”. If you include a Signature-Agent header but omit “signature-agent” from that covered-component list, verification fails because the signed payload and the presented headers do not agree.
The fix is to make sure the component list inside the parentheses includes “signature-agent” whenever you send the Signature-Agent header:
Cloudflare rejects signatures that cover non-ASCII components or unsupported derived parameters such as @query-params, @status, sf, or bs. Keep your covered components to the basics: @authority and signature-agent are enough for most agents.
# BROKEN — Signature-Agent header is present but NOT in the component list
Signature-Agent: "https://signature-agent.test"
Signature-Input: sig1=("@authority")\
;created=1735689600;expires=1735689660\
;keyid="poqkLGiymh_W0uP6PZFw-dvez3QJT5SolqXBCW38r0U"\
;tag="web-bot-auth"
# CORRECT — "signature-agent" is included in the covered components
Signature-Agent: "https://signature-agent.test"
Signature-Input: sig1=("@authority" "signature-agent")\
;created=1735689600;expires=1735689660\
;keyid="poqkLGiymh_W0uP6PZFw-dvez3QJT5SolqXBCW38r0U"\
;tag="web-bot-auth"Failure 3: the expires timestamp is too short and dies in transit
If your expires timestamp is set too aggressively short, the signature can already be expired by the time the request reaches Cloudflare — Cloudflare checks the expires parameter and fails verification if the current time is past it, so a 5-second window plus normal network latency means a guaranteed 403. Cloudflare’s guidance is that ‘a minute is often sufficient.’
Both created and expires are Unix timestamps. created marks when your agent signed the message; expires marks when Cloudflare should stop attempting to verify it. The window exists to limit replay attacks, but builders over-tighten it. If you regenerate signatures per request and set expires to created + 5 seconds, any retry, queue delay, or clock skew pushes you past the window before verification runs.
Set a window of roughly 60 seconds, and make sure your signing clock is accurate (NTP-synced). The example below uses created + 60:
import time
# CORRECT — ~60s window, the Cloudflare-recommended duration
created = int(time.time())
expires = created + 60 # "A minute is often sufficient" per Cloudflare docs
signature_input = (
'sig1=("@authority" "signature-agent")'
f';created={created};expires={expires}'
';keyid="poqkLGiymh_W0uP6PZFw-dvez3QJT5SolqXBCW38r0U"'
';tag="web-bot-auth"'
)
# BROKEN — created + 5 can expire before Cloudflare verifies it
# expires = created + 5“A signature that is cryptographically perfect but expired in transit fails exactly like a forged one. The verifier cannot tell the difference, and neither can your logs.”
On why short expires windows quietly break agents
Failure 4: your agent is not in Cloudflare’s signed-agents directory yet
If crawltest returns HTTP 401 — message well-formed, key unknown — your signature is correct but your public key is not in Cloudflare’s directory of bots and agents, so every Cloudflare-protected site treats you as unverified and can challenge or block you. This is the answer to the long-tail ‘my agent is configured but still rejected’: configuration is necessary but not sufficient; you also have to be registered.
To get verified across Cloudflare’s network, you apply through the Bot Submission Form in the Cloudflare dashboard (Account → Configurations → Verified Bots). You specify whether you want to be considered for the verified bots list or the signed agents list — note that a bot cannot be registered as both. Once approved, your agent appears on Cloudflare Radar’s bots and agents directory, and sites that allow verified bots will let you through.
Until then, individual site owners can still allow you manually, but you cannot rely on network-wide acceptance. If you control the destination zone, the fastest unblock is a WAF custom rule on that zone that allows known good bots. Cloudflare’s documented field for this is cf.client.bot:
# Cloudflare WAF custom rule (Skip / Allow action) on a zone you control:
# Allow verified/known-good bots, challenge everything else automated.
(cf.client.bot)
# Example combined rule: challenge automated traffic UNLESS it is a known good bot
(not cf.client.bot and cf.bot_management.score lt 30)| HTTP status | What it means | Where to look next |
|---|---|---|
| 200 | Message verified and key is known to Cloudflare | You are good — debug the destination site’s WAF rules, not your signature |
| 401 | Message well-formed, but key is unknown | Failure 4 — apply to the signed-agents directory or get the site to allow your key |
| 400 | Message is malformed | Failures 1, 2, or 3 — fix Signature-Agent quoting, component list, or expires |
Failure 5: generic WAF or managed challenge versus a purpose violation
If your agent is signed AND in the directory but still gets 403 or a managed challenge, you are hitting one of two things: a broad WAF/anti-bot rule on the specific site that does not allow verified bots, or a signed-agents policy violation that got your agent rejected or de-listed. These are different problems with different owners — one is the site operator’s config, the other is your application’s honesty.
On the WAF side: AI agents frequently get caught in broad anti-bot rules even when human visitors get 200, because the site’s rule challenges all automated traffic and was never updated to allow verified bots. There is nothing you can fix in your signature here; the site owner has to add a cf.client.bot allow rule. If it is your own property, add the rule from Failure 4. If it is someone else’s, you need them to allow-list verified bots or your specific agent.
On the policy side: signed agents must have a publicly documented purpose and benign, helpful behavior across multiple zones. Cloudflare’s policy explicitly states that ‘price scraping direct e-commerce competitors is not a valid use case,’ and it prohibits scalping, credential-stuffing, directory-traversal scanning, excessive data scraping, and DDoS behavior. If your disclosed purpose does not match your actual traffic, Cloudflare can remove you from the signed list — at which point your perfect cryptography stops mattering.
Pros
Cons
The full symptom-to-fix checklist for an AI agent blocked by Cloudflare
Run this ordered checklist whenever Cloudflare is blocking your AI agent: test on crawltest first, then fix the formatting failures (Signature-Agent, component list, expires), then resolve directory registration, then chase the destination site’s WAF and your policy compliance. Working top-down means you never debug cryptography for a problem that is actually a missing directory entry.
Each row pairs the exact symptom with the precise fix and the header or field you change. This is the one-screen reference the abstract spec docs and shallow blog posts never assemble in a single place — keep it next to your signing code.
| # | Symptom | Root cause | Exact fix |
|---|---|---|---|
| 1 | 403 / HTTP 400 on crawltest; Signature-Agent present | Signature-Agent not https:// or not double-quoted | Set Signature-Agent: “https://your-directory” with literal double quotes and an https scheme |
| 2 | 403 / HTTP 400; header value looks correct | signature-agent missing from Signature-Input components | Add “signature-agent” to the component list: sig1=(“@authority” “signature-agent”) |
| 3 | Intermittent 403; works locally, fails under load | expires window too short, expires in transit | Set expires = created + 60 and NTP-sync your signing clock |
| 4 | HTTP 401 on crawltest; clean signature | Key not in Cloudflare’s directory | Apply via the dashboard Bot Submission Form (Verified Bots → signed agents list) |
| 5 | Signed + listed but still 403 or managed challenge | Site WAF blocks all bots, or policy de-listing | Get the zone to add a cf.client.bot allow rule; confirm your documented purpose matches traffic |
Builder’s take
I run agent traffic into Cloudflare-fronted sites every day for Cyntr and Loomfeed, and the gap between ‘cryptographically signed’ and ‘actually let through’ is where everyone loses a day. Web Bot Auth is genuinely good design, but the failure modes are silent: a 403 tells you nothing about whether your signature was malformed, expired, or simply not on the directory yet.
- Treat the structured-field rules as load-bearing: an unquoted Signature-Agent or one missing from your component list fails closed, with the same 403 as ‘unknown key.’ Always test against crawltest.com first to separate a formatting bug (HTTP 400) from a key-not-known case (HTTP 401).
- Signing correctly only gets you a verifiable identity, not access. Until your key is in Cloudflare’s directory, a perfectly signed request is still ‘unverified’ and a site’s WAF can challenge it. Separate the two problems before you debug for hours.
- The policy layer is real and bites at approval time, not request time: if your documented purpose is ‘price-scrape competitors,’ you will be rejected or de-listed regardless of how clean your crypto is. Write an honest, benign, multi-zone purpose.
Frequently asked questions
Because the site’s WAF or anti-bot rule challenges all automated traffic and either your Web Bot Auth signature is malformed/unrecognized, or your verified identity is not being allowed by that specific zone. Test on crawltest.com/cdn-cgi/web-bot-auth: a 400 means fix your headers, a 401 means your key is not in Cloudflare’s directory, and a 200 means the destination site’s rules are blocking you, not your signature.
It means Cloudflare could not validate your HTTP Message Signature. The common causes are a Signature-Agent value that is not an https:// URI or is not enclosed in double quotes, a missing signature-agent entry in the Signature-Input component list, an expires timestamp that already passed, or a keyid that points to a key Cloudflare does not know.
Signature-Agent is an RFC 8941 structured field, so its value must be an https:// URI wrapped in literal double quotes, like Signature-Agent: “https://signature-agent.test”. A bare hostname, an http:// value, or an unquoted URL all fail. You must also include “signature-agent” in the Signature-Input component list, or verification fails even when the header value is correct.
Cloudflare’s documentation says a minute is often sufficient. Setting it too short — for example created + 5 seconds — risks the signature expiring in transit due to network latency or clock skew, which produces a 403. Use roughly 60 seconds and keep your signing host NTP-synced.
Submit the Bot Submission Form in the Cloudflare dashboard under Account → Configurations → Verified Bots, and choose whether you want to be considered for the verified bots list or the signed agents list. A bot cannot be on both. Once approved, your agent appears on Cloudflare Radar’s bots and agents directory and sites that allow verified bots will accept it.
No. Cloudflare’s signed-agents policy requires a publicly documented, benign purpose and explicitly states that price scraping direct e-commerce competitors is not a valid use case. It also prohibits scalping, credential-stuffing, directory-traversal scanning, excessive data scraping, and DDoS behavior. Violating the policy can get your agent removed from the signed list regardless of how correct your signature is.
Primary sources
- Web Bot Auth reference — Cloudflare Docs
- Signed agents concept — Cloudflare Docs
- Signed agents policy — Cloudflare Docs
- Allow traffic from verified bots — Cloudflare Docs
- The age of agents: cryptographically recognizing agent traffic — Cloudflare Blog
- Forget IPs: using cryptography to verify bot and agent traffic — Cloudflare Blog
- web-bot-auth reference implementation — Cloudflare Research (GitHub)
Last updated: June 6, 2026. Related: Identity Provenance.
