Methodology: how we scan and how we score

Four principles

1. Automated tests only detect some of the barriers. Published studies put the share they detect at between 14% and 57%, depending on the tool and on how barriers are counted: between 14% and 38% of barriers in the academic comparison by Vigo et al. (2013); around 40% with the best tool in the test run by the UK Government Digital Service (2017); and 57% of issues by volume in the study by Deque (2021), a company that makes automated tools. They can only prove that something fails, never that it passes.
2. Manual review is essential. It completes what no automated test can decide. Its progress is shown separately; when it feeds the score, it will make up 40% and, without it, we will not show a final rating.
3. The score is not legal conformance. Conformance status comes from the compliance matrix, criterion by criterion, with human review.
4. Everything is versioned. Each scan records the version of the engine, the rule catalogue and the scoring formula.

How we scan each page

• With a real browser. We load each page in Chromium with its style sheets, images and scripts, just as a person would see it.
• On desktop and on mobile. A 1366 × 768 pixel window and a 390 × 844 touchscreen phone. Each finding is stored only once, tagged with the devices it appears on.
• With well-known rules. Automated tests based on WCAG 2.2, levels A and AA: the axe-core rules, with the target size rule switched on, plus our own checks: reflow at 320 pixels, text spacing, tabbing through the page, autoplay, automatic redirects and language. We also run the axe-core best-practice rules: they appear as recommendations, but they do not count towards the score.
• With care for your server. The SiteNorma crawler reads your robots.txt and scans only a few pages at a time.
• Without mistaking a block for a result. If a firewall shows us a challenge page, or the website limits our requests and we cannot load the whole page, we mark it as not analysed and do not score it. A refusal caused by our own request rate is not a defect of your website.

What automated tests can and cannot detect

Manual review

Manual review is the part that no tool can do for you: a person does it, criterion by criterion, following a guide. On the platform it works like this:

• One guide per criterion. Each guide says what you are going to check, how to do it, what should happen and how to record the result. Anyone can follow some of them (“guided review”); others warn that they may require specialist help. If an aspect has no guide, we say so and it stays pending review.
• Whenever you like. There is no need to wait until the problems found by the automatic analysis are fixed, and an automatic analysis with no detections never closes a pending review.
• Four answers. “I have not found any problems”, “I have found a problem”, “I could not check it” and “Not applicable”. With no answer, the criterion stays “Pending review”. A problem has to be described, “Not applicable” has to be justified, and “I could not check it” is not a favourable answer.
• What is kept. The page, the element, the device, the conditions (for example, keyboard only or with a screen reader), who did it, when and the evidence, together with the version of the instructions that were followed. A draft can be saved and resumed; a recorded review is never edited or deleted, and a new one is added to the earlier ones.
• Recorded does not mean passed. “Manual review recorded” only confirms that it was saved. A favourable answer holds for what was reviewed (one page, one element, certain conditions), not for the rest of the website.
• Problems are treated like any other. A problem found in a review is created as an issue with the origin “Detected in manual review”, linked to its guide and with the same priority as those from the automatic analysis. A later review that does not find it closes it, not an automatic analysis.
• If the page changes. When a new analysis detects that the structure or the text of the page has changed since the review, the criterion becomes “Needs a new review” and the earlier valuation stays until the review is repeated.
• After a fix. If the automatic analysis cannot decide whether an element has been fixed, the specific guide can be repeated and the new review recorded. A favourable manual review is one of the pieces of evidence that allow the work to be archived (see below).

Manual review progress is shown apart from the automatic analysis and from the follow-up of the work, and it is not a conformity result. Manual review records results and evidence per criterion. Look at the automated result and the review progress separately: for now, recorded reviews do not change the score.

The accessibility score

Today the score you see reflects only the automated tests. The planned formula combines two parts: automated tests (A) and manual review (M). When the manual part is connected to the score, without manual review the maximum will be 60.

P is the planned accessibility score: 60% from automated tests and 40% from manual review.

The automated part (A)

A equals the number e raised to the power of minus D divided by 25. D is the sum, over every rule with confirmed failures, of the rule's weight multiplied by the share of pages where it fails:

• Rule weight (w): 10 if the severity is critical, 7 if it is serious, 3 if it is moderate and 1 if it is minor.
• Share of pages (p): the proportion of scanned pages with that failure, on any device. Priority pages and pages in the review sample count double.
• Only WCAG level A and AA rules with confirmed failures count, including “Needs review” cases that your team has marked as “Detection confirmed”.
• Rules that pass do not dilute the score, and the number of affected elements does not count: a single style can produce 200 identical failures.
• Postponing a fix, marking it as applied or archiving its work does not raise the score. Only an analysis that stops detecting the problem does, or discarding a detection with a justification (see below).

When D is zero, A equals 1. The more severe and widespread the failures, the closer A gets to zero, without ever going negative.

The manual part (M)

M is the share of success criteria that pass after manual review:

• C is the set of success criteria in the chosen standard (55 with EN 301 549 v4.1.1, 50 with v3.2.1), minus those marked “Not applicable” with a justification. You cannot mark a criterion “Not applicable” if the engine detects that it applies: for example, a video makes the criteria under 1.2 applicable.
• A criterion adds 1 if it is marked “Passes”, its review is less than 12 months old and it has no open confirmed failures. Otherwise, it adds 0.

When we show a rating

Until you have reviewed at least 80% of the criteria, you will see “Manual review pending (x/55)” instead of a rating. After that, the score comes with one of these bands:

Accessibility score bands

Score	Rating
90 to 100	Excellent
75 to 89	Good
50 to 74	Needs improvement
Below 50	Poor

An example

An online shop has three rules with confirmed failures:

Example calculation of D

Rule	Severity and weight	Pages with the failure	Contribution to D
Buttons without an accessible name	Critical, 10	50%	5.0
Field with more than one label	Moderate, 3	100%	3.0
Accessibility label with an obsolete type	Minor, 1	20%	0.2

• D = 8.2, so A = e^−8.2/25 ≈ 0.72 and the automated part contributes 43.2 points.
• Today, with M not yet connected to the score, the figure shown is 43 (automated part only).
• When review feeds the score: without manual review, “Manual review pending (0/55)” will be shown; after reviewing 50 of the 55 criteria, with 45 that pass, M = 45/55 ≈ 0.82 and the manual part will contribute 32.7 points.
• The score will then rise to 76. As more than 80% of the criteria will have been reviewed, the rating “Good” will be shown.

To see how these figures appear in a report, read the sample report, which uses a different illustrative case.

SEO and quality

S is the SEO score or the quality score, each calculated separately. As with accessibility, D adds up the weight of each problem multiplied by the share of affected pages, using their own weights of 10, 7, 3 and 1. You will find the table of weights on the SEO and quality page.

Performance

When the performance module arrives, its score will be 0.6 times the median on mobile plus 0.4 times the median on desktop, rounded. Real-user data will be shown separately. Coming soon

Team decisions and fix checks

What your team records about each element (an image, a button, a field) is kept in three parts that do not stand in for each other, always per element: fixing one does not change the others.

• Work tracking: pending fix, in progress, fix applied awaiting verification or fix postponed. It does not change the figures or the score: a postponed problem stays in the results.
• Detection validation: pending validation, detection confirmed or detection discarded with justification. Confirming a detection does not mean it is fixed.
• Result of the latest check: still detected, not detected in the latest check, could not be checked or needs manual review.

How a discard affects the figures and the score

• Discarding a detection requires a justification, the evidence behind it and the owner or administrator role. Who decided it and when are recorded.
• A discarded element leaves the counts and the score. A problem stops counting on a page only when every element it has there is discarded; if the engine finds more elements than it lists, the page keeps counting.
• The original detection and its history are not deleted. Discards are counted apart, listed with their justification and can be lifted. The engine’s score, without discards, is shown next to the one that applies them.
• An item that needs review only counts in the score once a person confirms an element.

When a check counts a fix as checked

Changing a status in SiteNorma does not change your website or start any analysis. To check a fix you open the summary before the analysis, which says which pages will be checked and how many it will use, and only “Start analysis” runs it. An analysis can only say that an element is no longer detected if:

• it analysed the page, on every device where the element was detected;
• it used the same engine and rules as the analysis that detected it;
• the test found elements of that kind to judge on the page, and no other element with the same problem appears that could be the same one under another address in the code.

In any other case the result is “Could not be checked” or “Needs manual review” (for example, an element in a pop-up) and the fix stays awaiting verification. If the element is found, it is still detected. The work can only be archived after a comparable check that does not detect it, or a later favourable manual review; if the problem is detected again, the work reopens as pending. Archiving does not certify that the website complies with any regulation.

Calibration and versions

• The constants (25 and 30) and the weights are calibrated against a corpus of real Spanish online shops. They may be adjusted over time.
• Each scan records the version of the engine, the rule catalogue and the formula. That way, you can compare scans knowing what they were run with.
• If an engine update makes an issue appear or disappear, we mark it as an “engine change”, not as a regression or as a fix on your part.

What the score is not

The score does not tell you whether your website complies with the law. A website can have a high score and still fail an important success criterion that no automated test detects. Conformance is decided criterion by criterion in the compliance matrix, with human review and evidence.

Methodology version: draft 0.1, 25 September 2026. Scoring formula version: 1.2.0 (includes the treatment of discards).