Connect your data
Crawl analytics in Rankshift works on your server logs. Connect a data source and Rankshift identifies AI bot visits in your traffic, categorizes each bot, and keeps your dashboard up to date.
Written By Rankshift
Only requests from known AI crawlers and agents are stored. All other traffic is discarded at ingest, so no human visitor data ends up in Rankshift. See Bots for the full list of supported bots.
Choose how to connect
Rankshift offers three ways to connect your crawler data. Open Connection Settings to pick one. You can manage or disconnect your data source there at any time.
Not sure which to pick? Use Cloudflare if your site runs through it. Use the API endpoint for any other live setup, such as another CDN or a custom stack. Use file upload when you want to look at the past.
Need something else? If none of these options fits your setup, get in touch and we’ll look at it together.
Manual file upload:
Upload server logs to analyze past crawler activity. This works with any hosting and you stay in full control of what you upload.
Highlights
• Supports historical data
• Works with any hosting
• Full control over what you upload
Good to know
• Data is processed after upload
• New data needs a new manual upload
• Accuracy depends on your log configuration
Best for: historical analysis, audits and backfilling existing data.
Upload a log file
1. In Connection Settings, choose Manual file upload.
2. Enter your domain (for example example.com) so Rankshift can build full URLs from the request paths.
3. Drag and drop your log file, or browse for it. Supported formats: .csv and Common Log Format (.log).
4. Rankshift processes the file and shows how many AI bot requests it found.
5. Click Upload to import them.
Only requests from known AI bots are imported. Everything else in the file is discarded.
CSV format. Column order doesn’t matter. Required columns:
timestamp, request_method, response_status, user_agent, request_url. Optional: client_ip, referer, country_code.
timestamp,request_method,response_status,user_agent,request_url
2026-10-14T12:34:56Z,GET,200,GPTBot/2.0,https://example.com/blog/post
Common Log Format (Apache / Nginx)
1.2.3.4 - - [14/Oct/2026:12:34:56 +0000] "GET /path HTTP/1.1" 200 1234 "https://referer" "GPTBot/2.0"
Cloudflare connection
Get live crawler data through your Cloudflare account. After setup, data flows in automatically.
Highlights
• Live and near-real-time data
• Fully automated after setup
• Highly consistent request data
Good to know
• Data starts from the moment you connect
• No historical backfill
• Limited to traffic that passes through Cloudflare
Best for: continuous monitoring and real-time crawler insights.
Create an API token
1. In Cloudflare, open your profile and select API Tokens.
2. Click Create Token, then Get Started on the first option.
3. Name the token.
4. Add these permissions:
– Workers Scripts → Edit
– Zone → Zone → Read
– Zone → Workers Routes → Edit
5. Continue to the summary and click Create Token.
Connect Cloudflare in Rankshift
1. In Connection Settings, choose Cloudflare connection.
2. Paste your API token and press Enter to validate it.
3. Select your Zone.
4. Click Deploy worker.
The Worker goes live for all requests to that zone and forwards AI crawler requests to Rankshift without affecting your site’s response times.
Cloudflare Worker limits
If the connection uses Cloudflare Workers, every request to your site counts toward your Workers quota, including requests from AI bots. The free Workers plan allows 100,000 requests per day. On high-traffic sites this can be reached quickly. When it is, two things can happen:
• Rankshift stops receiving data for the rest of that day, which leaves gaps in your crawl analytics.
• Visitors may see errors. By default Cloudflare fails closed and serves a 1027 error page.
To avoid this, set the route mode to Fail Open so your site stays available, or upgrade to a Workers Paid plan to remove the daily cap.
API endpoint
Send each crawler visit directly to Rankshift from your own system. This can be a CDN, log shipper, reverse proxy or custom application.
Highlights:
• Live and near-real-time data
• Customizable implementation
• Highly consistent request data
Good to know:
• Data starts from the moment you connect.
• You need your own implementation.
• Every field is mandatory in every request.
Best for: customizable implementation and direct API integration.
Set up the connection
1. Go to Connection Settings.
2. Select your domain.
3. Choose API endpoint.
4. Copy the endpoint URL and your API key.
5. Configure your system to send one POST request for each crawler visit.
6. Click Confirm once your system is sending data.
Endpoint:
POST https://crawler.rankshift.ai/ingest
Required headers:
Authorization: Bearer <your-api-key>
Content-Type: application/json
Request body:
Send one JSON object per visit.
{
"ts": "2026-08-04T14:32:18.123Z",
"input_type": "api",
"team_id": <your-team-id>,
"domain_id": "<your-domain-id>",
"zone_id": "<your-zone-id>",
"path": "/visitedPath",
"query": "?source=query",
"status": "200",
"ua": "chatgpt-user"
}
You find team_id, domain_id and zone_id in Connection Settings.
Fields
Important
• Every field is mandatory.
• If a field has no value, send an empty string. Example: use "query": "" when the URL has no query string.
• Keep your API key private.
• Regenerating the key invalidates the old one. Update your sender first.
Was this helpful?
Still need help? Ask the team