Sample dashboard | crawledby

ClaudeBot has a problem

In the last 30 days, 15% of ClaudeBot's visits ended in a 429 error because the server is throttling it.

104 425

bot visits in 30 days

+12 % vs. previous period

87 %

of visits went fine

13 464 ended otherwise

382 ms

average response to bots

good, under 500 ms

362

questions people asked AI

ChatGPT, Claude and others fetched a page for them · AI makes up 22% of visits

Most important first. Click to see the exact pages.

ClaudeBot is hitting errors
In the last three days 38% of its visits ended in an error, which used to be rare. See the bot details.
Bot details →
The server blocked 1 268 bot visits
The bot got a 403 or 429 and never reached the content. Usually a firewall, Cloudflare or a hosting limit is behind it.
Which pages →
463 server errors
The site failed while a bot was visiting. Neither the bot nor a person would have got the content.
Which pages →
AI can’t find the answer on 4 of 18 checked pages
The bot gets the page, but without JavaScript it’s missing text, the price or stock availability.
Which pages →

Show more (4) ↓

4 795 visits ended on a page that doesn’t exist
Old links, discontinued products or a sitemap error. For URLs bots keep asking for, we suggest a redirect.
Which URLs →
1 004× a bot left before the site answered
The server was too slow for the bot, so it left without the content.
Which pages →
1 667 visits from fake bots
Someone is posing as Googlebot or GPTBot from IP addresses that don’t belong to them. You can block those addresses.
Which bots →
AI used your site search 44×
See exactly what AI typed into your site search.
What it searched →
No bot has seen 646 of 2860 sitemap pages
Show pages
/sluchatka/audio-technica-ath-m50x/sluchatka/shure-aonic-50/reproduktory/bose-soundlink-flex/sluchatka/sony-wf-1000xm5/poradna/kodeky-bluetooth
932× a bot went where robots.txt forbids it
Compared with your current robots.txt. If you changed it recently, older visits may show up too. A bot that ignores the rules can only be stopped on your hosting or in Cloudflare.
Meta-ExternalAgent 932× /poradna/jak-vybrat-sluchatka-s-anc, /poradna/sluchatka-na-beh

GooglebotSeznamBotBingbotGPTBotClaudeBotMeta-ExternalAgentOAI-SearchBotApplebotAmazonbotOther bots