Your AI Crawler Rules May Be More Fragile Than You Think

Most site owners never read OpenAI’s crawler documentation directly. They read a summary of a summary, copy one robots.txt snippet, and move on. That is usually fine, until the underlying documentation changes in a way the summary did not anticipate.

OpenAI’s current Overview of OpenAI Crawlers includes this line for both OAI-SearchBot and GPTBot: the version number in the example user-agent string may change. At the time of writing, the published examples show OAI-SearchBot/1.4 and GPTBot/1.4. Those numbers are not permanent identifiers, and a rule that matches the complete string, including the version, is one OpenAI update away from silently failing.

This does not mean a standard robots.txt directive is unreliable. User-agent: OAI-SearchBot matches on the stable product token and will keep working after a version bump. The risk lives further down the stack, in the web application firewall, the CDN, and the plugin that someone configured two years ago and hasn’t opened since.

What OpenAI’s crawlers actually do

The most common governance mistake is treating every OpenAI-associated request as the same thing. It is not. OpenAI currently documents five distinct agents, each with a different purpose and a different robots.txt implication. Recording a policy for “OpenAI” as a single line item skips the actual decision.

OpenAI is explicit that OAI-SearchBot and GPTBot are independent settings. A site can allow OAI-SearchBot so it remains eligible for ChatGPT search citations while disallowing GPTBot to keep that same content out of training data. “Should we block OpenAI” is really five separate questions, and answering all five with one robots.txt line means guessing at four of them.

One nuance worth sitting with: OpenAI’s Publishers and Developers FAQ notes that if a page is disallowed for OAI-SearchBot, ChatGPT Atlas may still obtain its URL from a third party and surface the title and link on its own. A noindex meta tag prevents that, but only if the crawler is allowed to reach the page in the first place. Blocking crawling and preventing presentation are not the same control, which is a distinction anyone who has managed a traditional technical SEO audit will recognize.

Why matching the full user-agent string is fragile

A user-agent string bundles several things into one line: browser engine markers, operating system information, the bot’s product token, its version number, and a link to documentation. OpenAI’s published examples currently look like this:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot

A rule that checks for that entire string is fragile in two directions. First, OpenAI has told site owners directly that the version number may change, so a hard match on OAI-SearchBot/1.4 can start failing the moment OpenAI ships 1.5. Second, OpenAI’s documentation notes that requests specifically for the robots.txt file may carry an additional marker in the user-agent string, to help operators tell robots.txt fetches apart from other requests in logs that do not record the request path. A rule written against one captured example will not necessarily match every legitimate request from the same crawler.

The stable part of the identifier is the product token: OAI-SearchBot, GPTBot, ChatGPT-User. Standard robots.txt directives already match on that token and do not need to change. The place this actually breaks is custom infrastructure, most often:

• Web application firewall rules written against a full captured string
• CDN custom expressions with a hard-coded version number
• Security plugins configured once and never revisited
• Server-side bot-detection scripts or middleware
• Log-analysis filters and SIEM alerts tuned to an old pattern
• Cached IP allowlists with no recorded retrieval date

None of that means version pinning was a bad idea at the time. It means a crawler policy needs the same maintenance discipline as any other piece of infrastructure, not a one-time setup.

A user-agent string is a claim, not proof

Anyone can send a request with User-agent: GPTBot in the header. Matching on that string tells you what a requester claims to be, not who actually sent the request. This is where most published guidance stops short, and where OpenAI’s own verification methods are more specific than they get credit for.

For its automated crawlers, OpenAI publishes maintained IP-range files that site owners can check incoming traffic against:

• OAI-SearchBot: openai.com/searchbot.json
• GPTBot: openai.com/gptbot.json
• ChatGPT-User: openai.com/chatgpt-user.json
• OAI-AdsBot: openai.com/adsbot.json

ChatGPT agent traffic works differently, and it is the least-documented piece of this stack on most SEO blogs. Per OpenAI’s agent allowlisting guidance, every outbound ChatGPT agent request is signed using the HTTP Message Signatures standard defined in RFC 9421. Each request carries Signature and Signature-Input headers plus a Signature-Agent header set to https://chatgpt.com, and the corresponding public key is discoverable at a published well-known URL so an edge service can verify the signature is genuine.

In practice, most sites will not verify that signature by hand. Several infrastructure providers already do it for you: Cloudflare lists ChatGPT agent as a signed entry in its Bots and Agents directory under detection ID 129220581, Akamai categorizes it under Artificial Intelligence bots in its validated bots list, HUMAN treats it as a trusted agent in both Sightline and AgenticTrust, and Vercel added it to its Verified Bot Directory so requests are permitted automatically. If a site sits behind one of those, the practical step is turning on the existing verified-bot category, not writing a custom rule.

The layers a crawler policy actually has to pass through

robots.txt states a preference. It does not enforce one. A site can allow OAI-SearchBot in robots.txt and still block it at the edge, or allow it at the edge while robots.txt says otherwise, and both mismatches are common enough to be worth checking on a schedule rather than assuming.

A useful way to think about ownership: whoever runs monthly SEO or technical work on the site typically owns layers 1 and 4, while security or infrastructure usually owns layers 2 and 3. If nobody owns the handoff between them, that is usually where the mismatch lives.

Referral traffic is a different signal from crawl access

A page being crawled and a page sending visitors are not the same measurement, and conflating them makes AI visibility harder to report on accurately. OpenAI’s publisher FAQ confirms that ChatGPT automatically appends utm_source=chatgpt.com to referral links from its search results, which means Google Analytics or any standard analytics platform can isolate that traffic without extra tagging. Pair that referral data with server-log crawl activity and a clearer picture emerges: whether a page is being crawled at all, and separately, whether that crawling is translating into actual visits.

Run the self-audit

The checklist below mirrors the workbook further down this page. Check what is actually true today, not what you configured when the site first launched.

Get the workbook

The article above is free to use. The companion piece is an editable workbook built to carry the audit past a single read-through: a Crawler Policy Matrix, a robots.txt review sheet, an edge-control audit, a verification register for IP ranges and signed requests, a test plan, a search-safety check for Googlebot and Bingbot, a quarterly review checklist, and a change log. It is the same structure used above, in a format built for SEO, security, and infrastructure teams to work from together.

A short checklist for the next 30 minutes

• Search your CDN, WAF, and application code for the literal string GPTBot/1.4 or OAI-SearchBot/1.4. Replace any exact match with a match on the stable token.
• Confirm OAI-SearchBot and GPTBot have separate, deliberate robots.txt directives, not one shared line.
• Check whether your CDN has a verified-bot category for ChatGPT agent and whether it is switched on.
• Pull the current IP-range files for OAI-SearchBot, GPTBot, ChatGPT-User, and OAI-AdsBot, and note the date you retrieved them.
• Re-test Googlebot and Bingbot after any AI-bot rule change, since broad "block bots" rules catch traditional search crawlers more often than site owners expect.
• Assign an owner and a next-review date to the whole policy. A rule with no owner is the one that breaks quietly.

None of this requires blocking everything or allowing everything. It requires treating "should we allow OpenAI" as five separate decisions, documenting each one somewhere more durable than a single robots.txt line, and checking it on a schedule instead of assuming a two-year-old firewall rule still matches what OpenAI ships today. If a full crawlability and indexation review would help confirm where your site actually stands, send a message or a Loom and we will take a look.

Frequently Asked Questions

Sources

• Overview of OpenAI Crawlers - OpenAI Developer Platform
• Publishers and Developers FAQ - OpenAI Help Center
• ChatGPT Agent Allowlisting - OpenAI Help Center
• OAI-SearchBot published IP ranges
• GPTBot published IP ranges
• ChatGPT-User published IP ranges
• OAI-AdsBot published IP ranges

Crawler documentation changes over time. This article was last reviewed on July 15, 2026 against the OpenAI pages listed above. Compare your production rules with OpenAI's current documentation before deploying any access-control change.