Crawler explorer
Crawlers that accessed the site
Crawler identity is based on user-agent classification plus verification evidence such as reverse DNS when available. These rows come from processed ClickHouse evidence, not simulated data.
Demo tenant: Demo Publisher · Date range: 2026-05-21 to 2026-06-20
Crawlers
Crawler identity is based on user-agent classification plus verification evidence such as reverse DNS when available.
| Vendor | Crawler | Purpose | Category | Requests | Verified | Likely | Suspicious | Unknown | Paywalled | Premium | Blocked | Allowed | Monitored | Top content class | Top path |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Googlebot | search | search | 2 | 1 | 0 | 1 | 0 | 1 | 0 | 0 | 1 | 1 | archive | /archive/2021/private-equity-report | |
| unknown | unknown_bot | unknown | unknown | 2 | 0 | 0 | 0 | 2 | 2 | 2 | 0 | 0 | 2 | premium_analysis | /premium/no-slash |
| human | browser | unknown | human | 1 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 1 | public_news | /markets/nvidia-earnings |
| OpenAI | OAI-SearchBot | search | ai | 1 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | public_news | /markets/nvidia-earnings |
| OpenAI | GPTBot | training | ai | 1 | 1 | 0 | 0 | 0 | 1 | 1 | 1 | 0 | 0 | premium_analysis | /premium/banking-analysis |