Bots
Rankshift detects and tracks the AI crawlers and agents listed on this page. Every integration filters your traffic in the same way: only requests whose User-Agent matches one of these bots are stored. All other traffic is discarded at ingest.
Written By Rankshift
Each bot is assigned one of four types, so you can see not just who visits your site, but why.
Bot types:
Training bots:
Search bots:
User Query bots:
Other bots:
How to read this in Rankshift
• Training bots show whether AI companies are collecting your content. Blocking them in robots.txt stops future collection but does not remove what was already collected.
• Search bots show whether your pages are being indexed for AI search. If they never visit a page, that page is unlikely to be cited.
• User Query bots are the closest link to real demand. Frequent visits to a page usually mean AI assistants regularly use it to answer questions.
Controlling bot access
You decide which bots may crawl your site, typically through your robots.txt file. Training and search bots from the same company usually have separate user agents, so you can allow search visibility while opting out of training.
# Allow AI search, block AI training (example)
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /
Note that User Query bots act on behalf of a user and may not always follow robots.txt in the same way crawlers do. Check each vendor’s documentation.
Missing a bot?
The list is updated as new AI crawlers appear. If you see a bot in your logs that isn’t tracked, contact support.
Was this helpful?
Still need help? Ask the team