Jerilo

JeriloBot

JeriloBot is the crawler behind Jerilo. If you found it in your access logs, this page explains what it fetched and why.

User agent

Mozilla/5.0 (compatible; JeriloBot/1.0; +https://jerilo.com/bot)

Why it visited your site

Every visit starts with a person. There are two ways that happens.

  • Someone ran one of your pages through the free indexability checker. One check fetches that page (following a few redirects), your /robots.txt and one sitemap. Repeat checks of the same URL are served from cache for a few minutes, and we cap how often any one site can be checked.
  • Someone added your site to Jerilo for monitoring. Several times a day we fetch /robots.txt, the homepage, the sitemap and a small set of pages listed in it. About once a day we read the sitemap again to see which pages were added or removed. When someone who got an alert asks us to confirm a fix, we fetch the affected URL one more time.

What it doesn't do

  • It doesn't follow links, submit forms, run JavaScript or load images, CSS and scripts.
  • It only sends GET requests and stops reading after a few megabytes.
  • It doesn't obey Disallow rules in robots.txt. Finding out what your robots.txt blocks is part of the job, and every visit was asked for by a person.

How to block it

Match the user agent at your server, CDN or firewall. In nginx:

if ($http_user_agent ~* JeriloBot) {
    return 403;
}

If you monitor this site with Jerilo yourself, blocking JeriloBot will make every check fail.