Should you allow GPTBot, PerplexityBot and ClaudeBot?
Most public marketing websites should allow GPTBot, PerplexityBot and ClaudeBot to crawl useful, indexable pages, but not every URL. Allowing a crawler can make your content available for an engine's search, retrieval or model-related systems. It cannot guarantee that ChatGPT, Perplexity or Claude will mention your company.
Start with the pages that explain your category, products, services, use cases and point of view. Exclude customer portals, account areas, internal search results, staging sites, duplicate filter URLs and pages containing information that should not be broadly discovered. A robots.txt rule is a site-level access instruction, not a visibility strategy.
The practical decision is whether the potential discovery benefit outweighs the operational cost of requests and the sensitivity of the content. A small company with a mostly public brochure site will usually have a simpler decision than a company with a large documentation library, gated resources or user-generated content. Review the choice by crawler and path, rather than making one permanent allow-or-block decision for all engines.
For more context, read How Often Do Ai Answers Change.
What does robots.txt actually control?
Robots.txt tells compliant crawlers which paths they may request, but it does not control citations, rankings, training use or what an assistant ultimately says. The file is a public set of crawler instructions placed at the root of a domain, such as example.com/robots.txt.
A crawler can still be unable to use a page because of authentication, a server error, a network rule, a noindex directive or poor page rendering. Conversely, allowing a crawler does not make a page trustworthy, relevant or easy for ChatGPT, Perplexity, Gemini, Claude, Grok or Google AI Overviews to select.
Robots.txt also should not be treated as a security boundary. Blocking a path does not remove a previously exposed URL, prevent all automated requests or protect a secret. Keep confidential material behind authentication and remove sensitive data from public responses. Use robots.txt for crawl preferences, use access controls for protection, and use page-level indexing directives when search index handling is the actual requirement.
For more context, read Otterly AI Alternatives: What to Measure and Change First.
How do you write rules for the three named crawlers?
Add one User-agent block for each crawler and list the public paths that crawler may access. A basic file can contain the following plain-text rules:
User-agent: GPTBot Allow: /
User-agent: PerplexityBot Allow: /
User-agent: ClaudeBot Allow: /
The slash allows paths under the site root, so do not use it without first checking that the public site contains no private or unsuitable areas. A narrower pattern is safer when only selected directories should be available. For example, an organisation might allow /guides/ and /products/ while disallowing /account/ and /internal-search/.
Place the file at the domain root, save valid plain text, and check that the server returns it successfully. Directives are evaluated by user-agent token and path, so spelling matters. Do not assume that a generic wildcard rule expresses the same intention as a named rule. Before publishing, test the resulting rules with a robots.txt tester where available and inspect the file from an external connection.
Which pages should remain blocked?
Block pages that are private, low-value, duplicative or likely to create operational risk, rather than blocking an entire site by default. Common candidates include login and account paths, checkout flows, internal search results, preview URLs, staging environments, empty tag archives, session URLs and pages with personal information.
Marketing teams often miss faceted navigation. Filters can create many near-identical URLs, each of which consumes crawl activity without adding a distinct answer for a reader. Blocking every guide or product page to avoid duplicates creates the opposite problem, because the pages that could establish expertise become unavailable to crawlers.
Create a short path inventory before editing the file. Mark each path as public and valuable, public but repetitive, private, or unknown. Allow the first group, review the second, protect the third through authentication, and investigate the fourth. A disallow rule can also prevent a crawler from seeing links or context inside a blocked area, so use it selectively. If a page should not appear in search results, robots.txt may be the wrong control because a blocked crawler cannot reliably see a page-level noindex instruction.
How can you verify that the real crawler reached the site?
Verify crawler access in server logs and identify requests using more than the user-agent string. GPTBot, PerplexityBot and ClaudeBot can be named in a request header, but user-agent text can be imitated by unrelated software. Logs should show the requested path, response status, timestamp, user-agent, response size and any relevant network identity.
Begin by requesting the published robots.txt file and confirming that the server returns the intended version, not an old cached file or a redirect to an unexpected host. Then look for requests to an allowed page and compare them with requests to a deliberately disallowed path. A disallowed path receiving traffic does not automatically prove a robots failure, because the requester may not be a compliant crawler.
Separate verified crawler traffic from ordinary automated traffic in your analysis. Check the documentation for each crawler's current verification method before treating a request as genuine, because identification methods and published guidance can change. A spike in bot requests is not evidence of increased brand visibility, and no requests may simply mean the crawler has not selected your pages yet.
How should you roll out a robots.txt change?
Roll out a robots.txt change as a small, reversible experiment on a defined set of public pages. Save the current file, document the paths being opened or closed, and agree on a rollback version before publishing. A narrow change makes it easier to identify whether a later traffic or infrastructure issue is related.
First inspect server capacity, response errors and crawl activity for the selected paths. Then publish the change during a period when someone can review logs and restore the previous file if needed. Confirm the live response from the root domain, including the protocol and any important subdomains. A robots.txt file on one host does not automatically control another host.
Review the results over an appropriate observation period rather than expecting an immediate assistant mention. Record crawler requests, response status, average response time and errors separately for each named user agent. If opening access causes heavy load, narrow the allowed paths, improve caching, reduce duplicate URLs or adjust server controls. Avoid repeatedly changing the file without keeping a change log, because fluctuating instructions make later results difficult to interpret.
What should you measure after allowing a crawler?
Measure crawler access and business visibility as separate outcomes after changing robots.txt. The first outcome tells you whether a crawler could request selected pages. The second asks whether those pages are being used, cited or associated with your company in ChatGPT, Perplexity, Gemini, Claude, Grok or Google AI Overviews.
For access, track requests by crawler, allowed path, status code, response time and content type. For visibility, use a fixed set of category questions and record the engine, date, location where relevant, cited URLs, named competitors and whether the answer accurately represents your company. Keep the prompts stable enough to compare observations, while recognising that generated answers vary.
Do not treat crawler hits as citations, referrals or conversions. A crawler may access a page for a purpose that never produces a visible mention, and an answer may cite a page discovered through another route. The useful sequence is access, retrieval, citation, qualified visit and business outcome. Robots.txt affects the first stage directly, so change later stages through content quality, clear entity information, technical accessibility and evidence that answers the question better.
When should you change robots.txt instead of content?
Change robots.txt when the right page is inaccessible to a compliant crawler; change the content when the page is accessible but unhelpful, ambiguous or unsupported. This distinction prevents teams from using crawl permissions to solve a relevance problem.
For example, suppose a service page appears in server logs for GPTBot and PerplexityBot, but assistants describe the category without naming the company. Opening the same page to more crawlers is unlikely to fix the omission. Review whether the page clearly states what the company does, who it serves, where it operates, what evidence supports its claims and how its offer differs from alternatives. Add direct answers and maintainable facts rather than writing copy aimed only at a bot.
If the service page never receives crawler requests, check the robots.txt path, server response, canonical setup, internal links and rendering before rewriting the page. If only low-value duplicate URLs are crawled, improve URL controls and internal linking instead of allowing every variation. The decision rule is simple: permission problems belong in technical access; interpretation problems belong in content and site structure.
Related reading
- Understanding AI Visibility Metrics: What to Measure
- AI Visibility Platform: How AI Engines See You | Cituna
Sources consulted
- OpenAI (platform.openai.com)
- Perplexity (docs.perplexity.ai)
- Anthropic (anthropic.com)
- Google Search Central (developers.google.com)
Drafted with AI assistance from our own research and Search Console data, and reviewed by Rahul A before publishing. Rules and prices change; check the linked official source before you act.