AI Crawler overview
Rankshift’s crawl analytics shows how AI bots access your website, using first-party data from your own server logs. Server logs are the most direct signal of what AI platforms actually do with your content: which bots visit, which pages they request, and what your server answers.
Written By Rankshift
Once your data source is connected, Rankshift identifies AI bot hits in your logs, categorizes each bot, and fills the dashboard automatically.
What you can learn
• Which AI platforms visit your site, and how often.
• Which pages and folders they request most.
• Whether your server serves them correctly, or returns errors and redirects.
• What kind of visit it is: training, search indexing, or a live answer to a user’s question.
The dashboard
Filters
At the top of the page you can filter by:
• Date range
• Platform (for example OpenAI, Anthropic, Google, Perplexity)
• Bot (for example GPTBot, ClaudeBot)
• Bot type (Training, Search, User Query, Other)
• Status code
Crawl activity over time
A Column shows AI bot visits over time, with a separate line per platform or model. Use it to spot spikes after you publish new content, or sudden drops that may point to a blocked crawler or a server problem.
Platforms and bots
A breakdown of visits per AI vendor and per individual bot. Each bot is labelled with a type:
User Query bots deserve extra attention: a visit from one means an AI assistant read your page to answer a real person’s question right now.
Top pages and folders
A table of the URLs and folders AI bots request most, with the number of requests, the bots involved and the most common status code. Use it to see which of your content AI platforms care about, and which important pages they never visit.
Status codes
Shows how your server responded to AI bot visits.
These are all the status codes Rankshift shows, grouped as 2xx (success), 3xx (redirects), 4xx (client errors) and 5xx (server errors).
Crawl data and AI visibility
Crawl analytics tells you what bots do on your site. Rankshift’s visibility tracking tells you what AI answers say about your brand. Used together, they show whether a page that is crawled often is also cited, and why a page that is never crawled may never show up in answers.
Crawling does not guarantee citations. Treat crawl data as one signal next to prompt tracking, not as a replacement for it.
Crawl analytics and visibility data come together in the Page Tracker. There you can see which AI crawlers visit a page and how that page performs in AI answers.
Good to know
• Only AI bot traffic is stored. All other requests are discarded at ingest.
• Not influenced by bots. Rankshift data comes from your own logs, so it does not depend on third-party estimates.
• Data freshness. Live integrations update continuously. File uploads cover only the period in the file. [confirm timing]
• Availability. [confirm plans]
Frequently asked questions
Do I need a developer to connect my data? Not necessarily. A manual file upload needs only a log file, and the Cloudflare connection takes a few minutes if you can create an API token. The API endpoint needs someone to configure your own system.
Will it slow down my website? No. Rankshift only receives copies of AI crawler requests, and the integrations are built not to affect response times.
Does Rankshift store my visitors’ data? No. Requests that don’t come from a known AI bot are discarded.
What if an important bot is missing? Contact us if an important bot is missing. Send us the name of the bot and its user agent. We will look into it.
Was this helpful?
Still need help? Ask the team