JeriloBot
JeriloBot is the crawler behind Jerilo. If you found it in your access logs, this page explains what it fetched and why.
User agent
Mozilla/5.0 (compatible; JeriloBot/1.0; +https://jerilo.com/bot)
Why it visited your site
Every visit starts with a person. There are two ways that happens.
-
Someone ran one of your pages through the free indexability checker.
One check fetches that page (following a few redirects), your
/robots.txtand one sitemap. Repeat checks of the same URL are served from cache for a few minutes, and we cap how often any one site can be checked. -
Someone added your site to Jerilo for monitoring.
Several times a day we fetch
/robots.txt, the homepage, the sitemap and a small set of pages listed in it. About once a day we read the sitemap again to see which pages were added or removed. When someone who got an alert asks us to confirm a fix, we fetch the affected URL one more time.
What it doesn't do
- It doesn't follow links, submit forms, run JavaScript or load images, CSS and scripts.
- It only sends
GETrequests and stops reading after a few megabytes. -
It doesn't obey
Disallowrules in robots.txt. Finding out what your robots.txt blocks is part of the job, and every visit was asked for by a person.
How to block it
Match the user agent at your server, CDN or firewall. In nginx:
if ($http_user_agent ~* JeriloBot) {
return 403;
}
If you monitor this site with Jerilo yourself, blocking JeriloBot will make every check fail.