AI engines read your pages before they answer from them. Cituna’s AI crawler analytics shows that reading: which crawlers requested which pages, how often and for what purpose. Cituna also asks seven AI engines your buyers’ questions every day, so it can put both on the same row for each page: read by crawlers, and cited in answers or not.
What it records
A collector on your site reports each crawler request to Cituna: the page path, the user agent, the time, and the status code when your collector sends one. Cituna matches the user agent against 16 known crawlers and files the request by purpose.
| Purpose | Crawlers | What a request means |
|---|---|---|
| Search indexing | OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot, Bingbot, Applebot | The engine is building the index it searches when it answers. |
| On-demand fetch | ChatGPT-User, Perplexity-User, Claude-User | An assistant opened the page for a person, during a conversation. |
| Training | GPTBot, ClaudeBot, anthropic-ai, Amazonbot, Bytespider, meta-externalagent, CCBot | The page is being collected for model training or a dataset, not for a live answer. |
The reports show crawler requests over time, split by purpose, the requests by AI platform, one row per crawler with its share and when it was last seen, and your most visited pages. When your collector sends status codes, they also count HTTP errors and aborted responses.
Ways to connect
Pick the source that matches how your site is served. A site has one source at a time, so no request is counted twice.
| Your site | Collector | Coverage |
|---|---|---|
| Is on Cloudflare, in an active zone you control | Cloudflare Worker + Queue | Full coverage at the edge, including cache hits |
| Is a Next.js app, on Vercel, Netlify or your own server | Next.js / Vercel / Netlify | Partial: no status codes |
| Runs on a long-lived Node server, such as Express | Node / Express | Partial: requests that reach your server |
| Is served by nginx, including static sites | nginx / static hosting | Partial: requests that reach your server |
| Runs elsewhere, or has a log pipeline that can post JSON | Incoming server webhook | Partial: what your server sends |
| Runs WordPress | WordPress plugin | Partial: requests that reach WordPress |
| Has older access logs you want to load | Historical log import | Partial: a one-off snapshot |
You verify the domain once, through a matching Google Search Console property or a DNS TXT record. After that, the collector’s token can only send events for that domain, and it cannot read your data. The setup guide in the app has the code for each collector and a test you can run before real traffic arrives.
From crawl to citation
This is what a server log cannot give you. Each page in your report shows its crawler requests and errors next to the answers in your latest Cituna scan that cited it, with the engine and the prompt for each one, and the Cituna work for that page, such as the articles AutoSEO published there and the fixes it suggested.
So the pages sort themselves: crawled and cited, crawled but not cited, and pages no AI crawler has asked for.
Why the two belong together:
Crawler access
Cituna also checks whether your robots.txt lets each crawler in, and whether you publish an llms.txt file. A crawler you block can still appear in your requests, because robots.txt is a request, not a lock: OpenAI says it may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores it. Our guide to allowing AI crawlers in robots.txt safely covers the rules, and the free AI visibility checker tests them without an account.
Privacy and retention
- Cituna stores the page path, the user agent, the time, and the status code when your collector sends one.
- No cookies, IP addresses, request headers or bodies are stored.
- Tracking and secret query parameters are removed from URLs before they are stored.
- Crawler events are kept for 90 days, then deleted.
What it cannot tell you
- User agents are claims, not verified identities: anyone can send a request that says GPTBot. Before you act on a surprising number, check the IP against the vendor’s published ranges, as our guide to reading AI bots in server logs shows.
- Googlebot does not say whether it fetched a page for Search, AI Overviews or AI Mode, so Google’s AI features cannot be split out of its requests.
- Every collector except Cloudflare’s sees only the requests that reach your server, so pages served from a CDN cache can be missing. The Next.js collector sends no status codes, so its reports show no errors.
- A crawl and a citation on the same page show that both happened, not that one caused the other.
For what each plan includes, see the pricing page. For how Cituna measures the answers themselves, see the Search Console integration and the ChatGPT rank tracker.
Drafted with AI assistance from our own research and Search Console data, and reviewed by Rahul A before publishing. Rules and prices change; check the linked official source before you act.
