Testing

WCAG audit: how it works and what the report's score counts

6 min read
Mikhail ShaymardanovWeb Quality and Accessibility Expert

In the WCAG audit a store owner gets back, a criterion that fails on one page counts as failed for every page tested. An audit against WCAG 2.1 Level AA checks 50 criteria on a sample of pages. The re-audit after the fixes therefore raises the score only for criteria met on all those pages.

From the first test to the new score


#How a WCAG audit works, step by step?

A WCAG audit combines an automated scan, which catches only part of the issues, with manual tests: keyboard only, a screen reader, contrast and 200% zoom, reflow at 320 px, and every interactive widget such as menus, forms and the checkout.

The auditor settles the pages and browsers with the business, then checks every page against each success criterion, a rule the page must pass. For WCAG 2.1, the international accessibility guidelines, that means 50 criteria up to the middle level, AA. The 2021 version of the European standard EN 301 549 counts a page that meets them all as meeting its web chapter.

  1. The pages. The auditor explores the site and picks the sample, the pages that get tested, following the evaluation method of the W3C, the body that writes these guidelines. The method encourages asking the owner about page types and earlier audits.
  2. The browsers. The auditor and the business choose them, along with assistive technologies such as screen readers. The developers' list of supported browsers and tools is a good starting point.
  3. The tests. Each page is first checked as it loads, then in use: the checkout up to the order, the forms and their error messages. Prepare a customer account and your payment provider's test mode, so the auditor can reach the order confirmation without paying.
  4. The results. The auditor records one result per criterion and per page: passed, failed, or "not present" when the page has nothing the criterion covers. The W3C's free report tool also has "not checked," for a criterion not yet evaluated. A delivered table that still shows it is unfinished.

Simmer & Co., our example store, has 60 employees and sells cookware online to consumers in the EU, so the European Accessibility Act covers its site. Its audit covers 17 pages and a PDF manual, and its results table has 850 cells: 50 criteria times 17 pages. One column is for Reflow, which asks that a page work in a window 320 px wide, scrolling only up and down, with no information or feature lost. Only content that needs two dimensions, like a data table or a map, is exempt. Someone who zooms a 1,280 px page to 400% sees that same window.

On the "Cookware" product list, though, the products keep four fixed-width columns. We opened the page at that width: the row is 960 px wide, so 640 px have to be scrolled sideways. Reflow fails on that page.

Screenshot: the Simmer & Co. Cookware page in a narrow window, with the header, the mustard banner, the Filters button, the whole Cast iron Dutch oven card, the next card cut off by the right edge and a horizontal scrollbar at the bottom.Marker 1. The row of products, wider than the window, cut off by the right edge.
The four product columns of Simmer & Co.marker 1 run past the right edge of a 320 px window, the view a 1,280 px page gives at 400% zoom.
Website accessibility audit: cost, quotes and how to choose an auditor

#How to read an accessibility audit report and its score?

A criterion counts as met for the sample only if it passes on every page where it applies. One failing element fails the criterion on its page, and in a process such as the checkout, one failing page keeps the whole process from conforming.

In the column for Reflow at 320 px, Simmer & Co. gets "passed" on 15 pages and "failed" on two, the product list and the search results. The criterion therefore fails, and its 15 passing pages earn it nothing.

Excerpt of the audit results for Simmer & Co.: 17 pages in rows, three criteria in columns, one result per cell and, at the bottom, the result for the whole sample.Marker 1. The two failed cells for Reflow: product list and search results. Marker 2. The Reflow result for the whole sample: failed.
In the Simmer & Co. results, Reflow passes on 15 pages and fails on 2marker 1: it counts as failed for the whole samplemarker 2.

A criterion with nothing to check, like captions on a site without video, is marked "not present" and counts as met. One that applies to only a few pages is met when it passes on each. Error prevention asks that an order or an account change can be undone, checked for errors or confirmed first. Here it applies only to Payment and the customer account, and passes on both.

#How the score is calculated?

WCAG gives no score. The W3C's method lets a report add one if it explains the calculation, and warns that such scores can mislead. The Simmer & Co. report counts the criteria met on every page, including those marked "not present": 36 of 50, or 72%. To conform, the site must meet all 50 on every page, so it doesn't conform yet.

For Simmer & Co., 36 of the 50 criteria are met on every page: 72%, short of the 100% that conformance takes.
Step of the calculationSimmer & Co.
WCAG 2.1 Level A and AA criteria50
Not present on any of the 17 pages, so met10
Passed on every page where they apply26
Failed on at least one page14
Criteria met10 + 26 = 36
Score36 ÷ 50 = 72%
Conforms to WCAG 2.1 Level AANot yet: that takes 100%

A report can also average each page's own score, and in that average the 15 pages that pass Reflow do count. Before comparing reports, check how each figure is calculated.

WCAG certification for a website: what an accessibility statement proves

#What to have retested at the re-audit?

Have every failed criterion retested on every page where it applies, including the page the report cites. The W3C's method treats a re-audit as a new evaluation, which can keep part of the first sample to compare results and swap about half for new pages. The Simmer & Co. price quote sets that scope, and here is why it matters.

#Why the score rises less than expected?

The W3C's method asks a report for at least one example of each failed criterion, and calls it good practice to flag issues that repeat. Only a page-by-page results table lists every failing page. The Simmer & Co. report gives 14 examples, one per failed criterion.

The team fixes all 14 and expects 100%. The re-audit, which retests every criterion on every page, comes back at 86%: 43 of 50. For Reflow, the report showed the product list, since fixed. The search module, though, builds its own product columns, untouched by the fix. We reopened its results page at 320 px: it still scrolls 640 px sideways, so Reflow still fails. Without zoom, a desktop visitor sees no change: the fixed list keeps its four columns.

Avoid

Screenshot: the search results for Dutch oven on the Simmer & Co. site, in a narrow window, with the whole Cast iron Dutch oven card, the next card cut off by the right edge and a horizontal scrollbar at the bottom.Marker 1. The product columns of the search module, cut off by the right edge.

Correct

Screenshot: the fixed Cookware page, in the same narrow window, with the same header, the same mustard banner and the Filters button, then the products in a single column, each with its photo to the left of its name and price.Marker 1. The fixed product list, in a single column.
At 320 px wide, the Simmer & Co. search results pagemarker 1, missing from the audit report, still scrolls to the right. The fixed product listmarker 1 fits in one column, under the same banner as before the fix.

Seven criteria fail the same way.

At the re-audit, seven Simmer & Co. criteria still fail on a page the report didn't cite.
What the criterion checksPage in the report, fixedPage where the issue remains
Content fits in 320 px wideProduct listSearch results
Text stays readable at 200%Product page “Cast iron Dutch oven”Care tips
Text has enough contrast with its backgroundHomeShipping and returns
Each link, in context, says where it goesHomeCare tips
Each field has a labelPayment (card number)Contact (“Message” field)
Each informative image has a text alternativeHomeCustomer account
Headings follow the page's structureCare tipsLegal notice

Fixing every issue the report cites can even leave the score where it was. Had each of the 14 criteria also failed on a second page, Simmer & Co. would have stayed at 72%.

#What to ask the auditor?

Ask for every failed criterion to be retested on all its pages, because a page that isn't retested keeps its first-audit result in the table. A fix made in a template, the layout several pages share, can repair other pages too. A re-audit limited to the 14 examples would still leave them "failed."

Also flag the pages changed since the first audit for reasons other than the fixes, so the auditor retests them. The results table shows each page as it was on the evaluation date.

Finally, ask for the updated, dated results table, with the score recalculated and every page and element still at fault. The method lists a report of every failure on every page among the extras a client can request.

The Simmer & Co. score, which tracks its progress between audits, goes from 72% to 86%, and the site still doesn't conform. The store can use the new results to update the accessibility information the European Accessibility Act requires in its terms and conditions or an equivalent document. In the re-audit's table, the search results page alone still makes Reflow fail for the whole sample. Fixing only the search module would bring the score to 88%, 44 of 50.

WordPress accessibility audit: where each issue comes from, and who fixes it

Before the re-audit, an Includdy scan of the 17 pages shows, page by page, what a tool can detect, even on a page the report never mentioned.


Share this article

Frequently asked questions

No. The Act never mentions an audit. An online store it covers must meet the accessibility requirements and explain how in its general terms and conditions, or an equivalent document. If a competent authority (a public body that enforces the Act) sends a reasoned request, the store must also give it all the information needed to show that its service complies. An audit is a practical way to gather that information.

Microenterprises that provide services are exempt: fewer than 10 employees, and annual turnover or a balance sheet total of €2 million or less. Who the Act applies to depends on what a business sells and to whom.

It states the scope: the site and pages evaluated, the target level (for example WCAG 2.1 Level AA), and the browsers and assistive technologies used. It then gives the results for each success criterion, page by page or for the whole sample, with at least one example of every criterion not met. The W3C's evaluation method also asks for the evaluator's name, the client's name and the date of the evaluation.

The W3C's free report tool walks through these sections one by one, so you can check that the report you received has them all.

The European Accessibility Act sets no frequency. It asks a store to have procedures that keep its service compliant, taking changes to the service into account, so plan a new audit after a redesign or a major change. The W3C's evaluation method also suggests running an evaluation again periodically, to monitor progress.

After each round of fixes, a re-audit updates the results. Book it in the same quote as the first audit: what to compare in a quote.