Private beta
Index Signal
A log-file analyzer built for AI crawlers, not just Googlebot.
The problem
Every log-file tool I tried was built around Googlebot behavior. GPTBot, PerplexityBot, and ClaudeBot crawl completely differently, and none of the existing tools separated them out in a way that was actually useful.
When I ran the crawl-budget research for this site, I ended up writing custom parsing scripts just to answer basic questions like which AI crawlers were hitting a site at all.
The research behind it
The crawl-budget research entry on this site used an early version of the parsing logic that became Index Signal. Six mid-size sites, thirty days of logs each, and a pattern that held across all of them: AI crawlers request far fewer pages per visit than Googlebot, and they skip most JavaScript-rendered content entirely.
Why I built it
I needed a tool that could tell me, at a glance, which pages GPTBot actually crawled last week versus which pages it should have crawled based on the sitemap. Nothing on the market separated AI crawler traffic from the rest of the log file cleanly enough to answer that.
What it does
AI crawler segmentation
Separates GPTBot, PerplexityBot, ClaudeBot, and Googlebot into distinct views instead of one combined crawl report.
Crawl gap detection
Flags important pages in your sitemap that AI crawlers have not requested in the tracking window.
JS-rendering check
Highlights pages where content likely depends on client-side rendering that AI crawlers may not execute.
Roadmap
- ShippedLog upload and crawler segmentation
- ShippedCrawl gap detection against sitemap
- In progressAutomated weekly digest email
- PlannedDirect server integration, no manual log upload
Who should use it
- Technical SEOs and developers who already read server logs and want AI-crawler-specific answers instead of a combined view.
- Sites with large page counts where crawl budget genuinely matters, not small brochure sites.
Lessons learned so far
Log format varies enough across hosting setups that a generic parser breaks constantly. Most of the early engineering time went into normalization, not analysis.
Want to try Index Signal when it's ready?
Join the waitlist