Correlation Between Manual and Automatic Accessibility Testing

In this post we provide a short introduction to two methodologies for measuring accessibility of public web sites; (1) Norge.no – an annual survey carried out by the Norwegian Agency for Public Management and eGovernment (DIFI) on quality of Norwegian public web sites and (2) eGovMon – a fully automatic tool based on the Unified Web Evaluation Methodology. Finally we present, despite methodological differences, a strong correlation between the results of these two surveys.

Norge.no

The survey is carried out by the Norwegian Agency for Public Management and eGovernment (DIFI) on quality of public web sites. The results are presented annually and are receive a lot of attention by web site owners, policy makers as well as in the popular media.

The survey consists of 34 indicators in the area of accessibility, usability and relevance. Twelve of the indicators are on accessibility from which seven are directly related Web Content Accessibility Guidelines 1.0 level 1,2 and 3. The survey is carried out manually by experts and the results are presented as stars on web site level; 6 stars are awarded to the best web sites while the web sites with most potential for improvement receive only 1 star.

eGovMon

The eGovMon tool is based on Unified Web Evaluation Methodology using the Web Content Accessibility Guidelines 1.0 level 1 and 2.  The
eGovMon system is an implementation of the fully automated monitoring application scenario. In total the system consists of  23 web accessibility tests both on (x)HTML and CSS. On web site level, the results are presented as number of barriers found over number of tests applied (UWEM score). The most accessible web sites will receive 0%, while those with most improvement potential will receive 100%.

Correlation between results.

We evaluated automatically 414 of the Norwegian municipalities using the eGovMon tool and compared these to the results presented by Norge.no. In the figure below we plot the results from both Norge.no and eGovMon. From this figure, we can see that the average eGovMon web site score improves the more stares appointed by Norge.no. From this we can conclude that there is a strong correlation.

This is true for all groups except the web sites which received six stars, which indicates that good accessibility (six stars by Norge.no) can not be identified by automatic evaluation alone but needs to be supported by manual assessments.

egovmon_vs_norgeno

This is based on a scientific publication Benchmarking and improving the quality of
Norwegian municipality web sites
for 5th International Workshop on Automated Specification and Verification of Web Systems 2009 by Morten Goodwin Olsen, Annika Nietzio, Mikael Snaprud and Frank Fardal.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: