The SiteNorma crawler

SiteNorma uses it to analyse the accessibility, SEO and technical quality of web pages. If you have seen it in your logs, here is everything you need to know.

How it identifies itself

The crawler identifies itself by its name, SiteNorma, and by this page. When it loads a page with its browser —during a scan, a site preview or an ownership check with the meta tag or the file— the user agent ends like this:

… Safari/537.36 SiteNorma/1.0 (+https://sitenorma.com/rastreador/)

The beginning is that of an ordinary Chrome, with the version of the browser we use. When scanning for mobile it says Android, but the ending is the same. To read robots.txt and sitemaps and to check links we use this other user agent:

Mozilla/5.0 (compatible; SiteNorma/1.0; +https://sitenorma.com/rastreador/)

The crawler identifies itself with the user agent described on this page. If you need to verify a request, contact our team. When we publish fixed egress IP addresses, we will list them here.

When it visits your website

  • When a SiteNorma customer scans a website, usually their own or one belonging to a client who has given them permission.
  • When someone uses our free audit on a page of your website.
  • When we check ownership of a website (meta tag or file under /.well-known/).
  • To check the links on scanned pages and, in the future, the uptime of the websites our customers monitor.

Full and recurring scans can only be run on websites whose ownership has been verified. Without verification, a few pages at most are scanned.

What it does and does not do

  • It loads pages in a real browser, with their style sheets, images and scripts, because without them it is impossible to measure things such as contrast.
  • It closes the cookie banner when this is needed to analyse the content.
  • It does not submit forms, does not log in, does not solve CAPTCHAs or security challenges and does not try to get around paywalls. If it finds a challenge page, it marks it as blocked and moves on.
  • It does not collect your users' personal data.

It respects your robots.txt

Before scanning a website, the crawler reads its robots.txt file and follows its rules, in line with the RFC 9309 standard. If robots.txt does not exist (4xx error), we assume there are no restrictions; if the server fails (5xx error), we do not scan anything.

For robots.txt rules, the crawler's name is SiteNormaBot. It is one of the two names we look for in your rules; the other is SiteNorma, which is the one in the user agent of our requests, and we apply the rules of either. The word “bot” is left out of the user agent on purpose, because some servers throttle every request that contains it; we still identify ourselves by name and by this page, and we follow your robots.txt. We also follow the general rules in the User-agent: * group.

At a gentle pace

We scan only a few pages at a time on each domain and, when checking links, we make at most one or two simultaneous requests per server.

If your server answers with HTTP error 429 (or with a 503 and a Retry-After header), we wait as long as it says, between 3 and 30 seconds (about 5 if it gives none), slow down our requests to that server and load the page once more. If it keeps refusing requests sent after that wait, we extend the wait a couple of times and then stop asking for the remaining files of that page. If the server asks us to wait for more than 30 seconds, we do not load its pages again, but we carry on with the others.

If we still cannot load the page, we do not analyse it: it appears as “not analysed”, does not count towards the estimate and is not treated as a defect of your website. We stop asking when three pages in a row cannot be loaded: we load no more pages and we do not check that website's links either.

If you want SiteNorma to analyse your website at a normal pace, allow its requests on your server or firewall. You can recognise them by SiteNorma/1.0 in the user agent.

How to block it

To stop SiteNorma from scanning your website, add this to your robots.txt:

User-agent: SiteNormaBot
Disallow: /

You can also block only some sections:

User-agent: SiteNormaBot
Disallow: /mi-cuenta/
Disallow: /carrito/

Bear in mind that, if you block the crawler, you will not be able to scan your website with SiteNorma or with our free audit either.

Contact

If the crawler causes any problems on your website or you think it is acting improperly, email us at hi@gassiotllobet.com or use the contact form. We will reply as soon as possible.