Allowing a crawler and letting it through are two different things. The first is decided in a three-line text file. The second is decided much further down, in a layer the marketing team never sees and the security team has no reason to connect to a visibility problem.

Why does an AI crawler get a 403 when robots.txt allows it?

Because robots.txt and infrastructure are two separate gates, and only the second decides actual access. OpenAI documents this plainly: web protection services such as Cloudflare, Akamai and similar providers can mistakenly block legitimate crawlers, often returning 403 Forbidden errors. OpenAI's crawlers resemble automated traffic patterns — that is what they are — and may therefore be denied unless specifically allowlisted.

Robots.txt answers one question: has the site declared that this crawler is permitted? It does not answer the one that matters: can the crawler actually get through the infrastructure and retrieve the page?

A site can therefore publish User-agent: OAI-SearchBot followed by Allow: /, pass every check in a conventional technical audit, and remain unreachable.

What is the difference between declared accessibility and effective reachability?

Declared accessibility is what a site permits. Effective reachability is what a crawler receives. The first is expressed in robots.txt, meta robots and HTTP headers. The second is observed at the end of the full chain: DNS, CDN, web application firewall, bot management, authentication, JavaScript challenges, CAPTCHA, rate limits, geographic rules, then the response code and the content actually returned.

A site can pass the first test and fail the second. When it does, it carries what we call a discoverability gap: the page is public to a human browser and non-existent to the system that will build the answer.

The test that counts

Reading a file proves nothing. The useful check replays the complete path: declared permission → DNS/CDN → firewall and WAF → bot management → application layer → HTTP code → final content. It is the only check that answers "what does the crawler actually get".

Which layers block AI crawlers?

Seven mechanisms can make a public page unreachable to a robot while leaving it perfectly accessible to a human. OpenAI identifies several of them explicitly in its crawler access guidance:

Each one works exactly as designed. A security configuration can fulfil its mission perfectly while degrading machine accessibility without anyone noticing.

Why does OAI-AdsBot get through where OAI-SearchBot is blocked?

Because OpenAI's two crawlers do not hold the same status with protection providers. OpenAI states that OAI-AdsBot is officially verified and allowlisted by Cloudflare. OAI-SearchBot carries no such verification.

In other words: the crawler that validates advertising landing pages passes the gate by default at the most widely deployed protection provider. The one that governs citation in ChatGPT's organic search has to be allowed by hand.

That asymmetry deserves to be stated as it is. Nothing about it is malicious — a contracted commercial crawler is simply easier to verify than a general-purpose indexing crawler. But the consequence is concrete: a company buying advertising on ChatGPT will see its pages validated, while the same pages stay invisible to the system that builds the organic answers. Paid visibility gets through; earned visibility stays outside.

A crawler allowed in robots.txt and refused at the firewall raises no alert. It produces nothing at all, which is precisely the problem.

What do a 403 and a 429 mean for AI visibility?

A 403 means the crawler reached the infrastructure and was denied access. A 429 means it got through and is being throttled. In conventional monitoring, these are technical anomalies among others. For AI visibility, they are the two signals that say the first prerequisite is failing.

A 403 usually comes from WAF rules, firewall policies, automated bot blocking, IP restrictions or a geographic rule. A 429 comes from request-rate rules, bot-protection thresholds or an automated security trigger.

OpenAI recommends reviewing HTTP response codes — naming 429 specifically — along with firewall and CDN logs, bot mitigation events, request throttling rules and traffic analytics around the time the crawler attempted access. Edge logs therefore become an instrument of AI visibility diagnosis. Not because they say whether a brand is recommended. Because they say whether it is even a candidate.

Is being reachable enough to be cited?

No. Reachability is the first of seven stages, and the only one under full control. OpenAI phrases its own position carefully: allowing OAI-SearchBot helps make content eligible to be discovered and surfaced. It promises neither inclusion nor position.

The KAIS AI Visibility Chain separates the stages precisely to prevent that confusion.

The KAIS AI Visibility Chain

Be Discovered → Be Retrieved → Be Understood → Be Trusted → Be Cited → Influence the Answer → Be Recommended.

Firewall blocking sits entirely within the first stage, and splits it in two: declared accessibility, then effective reachability. Until the second one clears, everything downstream is theoretical.

The value of that decomposition is that it forbids treating a problem with the wrong remedy. A crawler blocked by the WAF is not fixed by publishing twenty articles. A retrieval relevance problem is not fixed by changing firewall rules. Weak external corroboration is not fixed by adding another FAQ. Each stage calls for a different intervention, and diagnosis begins with knowing which one gave way.

How do you verify your site's reachability?

By replaying the complete path with the relevant crawler's user agent, rather than reading a file. Eight checks are enough to produce a defensible finding:

  1. confirm that the relevant AI crawlers are intentionally allowed in robots.txt;
  2. verify that the page returns a successful HTTP code to a request carrying the crawler's user agent;
  3. inspect CDN and WAF rules for false positives on bot traffic;
  4. check JavaScript challenges, CAPTCHAs and authentication or session requirements;
  5. review 403 and 429 events around crawler requests;
  6. check geographic access restrictions;
  7. validate the final content actually returned after redirects, without JavaScript execution;
  8. keep that access finding separate from later retrieval and citation measurements.

The seventh point deserves particular emphasis. A crawler does not render JavaScript. A page whose main content only exists after client-side execution returns a perfectly valid 200 and an empty body. That is the most deceptive case of all: the response code is fine and there is nothing to retrieve.

Do you have to weaken your security?

No. The answer is not to remove protection, but to distinguish malicious automation from legitimate crawler traffic. The two objectives — block unwanted automation, and let legitimate robots reach public content — are not contradictory. They are simply owned by two teams that do not talk to each other.

OpenAI publishes its crawler user agents and its searchbot IP addresses precisely to make targeted allowlisting possible. Its guidance is to check both bot identity and infrastructure-level blocking rules rather than relying on robots.txt alone. Google states the same requirement in its AI features documentation: crawling must be allowed in robots.txt and by any CDN or hosting infrastructure.

The target configuration is controlled accessibility: public zones open to identified and verified robots, member areas, internal tools and customer dashboards closed. And the exception written into the security policy, so a routine update does not wipe it six months later.

The starting point is not where you think

AI visibility does not begin when an engine cites you. It begins with the possibility that your information can enter the evidence environment the system has access to at all. And at that precise point, one green checkmark is not enough.

Your robots.txt can say yes. Your HTML can say yes. Your marketing team can say the page is public. While at the edge of your infrastructure, the firewall says no.

That is not a content problem. It is not an authority problem. It is not even a citation problem. It is a discoverability gap — and until it is closed, everything else is a theoretical conversation.

Sources