Correlation Between Manual and Automatic Accessibility Testing

July 27, 2009

In this post we provide a short introduction to two methodologies for measuring accessibility of public web sites; (1) – an annual survey carried out by the Norwegian Agency for Public Management and eGovernment (DIFI) on quality of Norwegian public web sites and (2) eGovMon – a fully automatic tool based on the Unified Web Evaluation Methodology. Finally we present, despite methodological differences, a strong correlation between the results of these two surveys.

The survey is carried out by the Norwegian Agency for Public Management and eGovernment (DIFI) on quality of public web sites. The results are presented annually and are receive a lot of attention by web site owners, policy makers as well as in the popular media.

The survey consists of 34 indicators in the area of accessibility, usability and relevance. Twelve of the indicators are on accessibility from which seven are directly related Web Content Accessibility Guidelines 1.0 level 1,2 and 3. The survey is carried out manually by experts and the results are presented as stars on web site level; 6 stars are awarded to the best web sites while the web sites with most potential for improvement receive only 1 star.


The eGovMon tool is based on Unified Web Evaluation Methodology using the Web Content Accessibility Guidelines 1.0 level 1 and 2.  The
eGovMon system is an implementation of the fully automated monitoring application scenario. In total the system consists of  23 web accessibility tests both on (x)HTML and CSS. On web site level, the results are presented as number of barriers found over number of tests applied (UWEM score). The most accessible web sites will receive 0%, while those with most improvement potential will receive 100%.

Correlation between results.

We evaluated automatically 414 of the Norwegian municipalities using the eGovMon tool and compared these to the results presented by In the figure below we plot the results from both and eGovMon. From this figure, we can see that the average eGovMon web site score improves the more stares appointed by From this we can conclude that there is a strong correlation.

This is true for all groups except the web sites which received six stars, which indicates that good accessibility (six stars by can not be identified by automatic evaluation alone but needs to be supported by manual assessments.


This is based on a scientific publication Benchmarking and improving the quality of
Norwegian municipality web sites
for 5th International Workshop on Automated Specification and Verification of Web Systems 2009 by Morten Goodwin Olsen, Annika Nietzio, Mikael Snaprud and Frank Fardal.


Measuring Transparency

July 22, 2009

Transparency is seen as an important part of good governance. It is even one of the eight characteristics of good governance according to UN Economic and Social Commission for Asia and the Pacific (UNESCAP), defining transparency as following:
“Transparency means that decisions taken and their enforcement are done in a manner that follows rules and regulations. It also means that information is freely available and directly accessible to those who will be affected by such decisions and their enforcement. It also means that enough information is provided and that it is provided in easily understandable forms and media”.

Even though addressing open governments is a common goal for all transparency indicators, there exists no uniformly accepted list of indicators. The approaches are often from different perspectives such as political, economic, media, human rights or business and thus see transparency slightly different.

Following are some of the most well known methodologies/surveys on assessing transparency:

  • Sunshine Review: The Sunshine Review presents a list of checklist important for a transparent government. The survey is on web sites in the United States of America and aims at putting pressure on web site owners to become more transparent. Unfortunately, there is no justification of why the items are important.
  • eGEP (eGovernment Economics Project): The eGEP project has developed a framework to assess the impact of e-government services in three main areas: efficiency, democracy and effectiveness. Democracy is further divided into openness, transparency accountability and participation. The data sources include third party assessment, administrative records data, web metrics data and automatic web crawler software.
    It should be noted that a second version of eGep, eGEP 2.0, is under development.
  • monitor: assess Dutch web sites on six different areas including transparency to the public. The data is mostly collected manually.
  • BEGIX (Balanced E-Government Index): BEGIX is a self-assessment questionnaire for e-government and e-democracy. consisting oif in total 27 questions. Six of these questions are on transparency.
  • The Norwegian Agency for Public Management and eGovernment (Direktoratet for forvaltning og IKT, DIFI) assess public Norwegian web sites annually. The methodology consists of three parts, accessibility, usability and useful content. Useful content is highly related to transparency as they look at whether users can find basic information, if the web site provides feedback, etc.

This post is based on information from the State-of-the-art Review: transparency indicators originally prepared by Lasse Berntzen, Annika Nietzio, Morten Goodwin Olsen, Ahmed AbdelGawad, Mikael Snaprud

Accessibility of PDF documents

July 16, 2009

Similar to (x)HTML and CSS, which are the technologies use for traditional web pages, PDF documents may be designed with accessibility in mind. Today, most of the public PDF documents are inaccessible.

Scanned PDFs

A typical example of an inaccessible PDF is a scanned document originally consisting of mostly text. Assistive technologies, such as screen readers used by for example visually impaired users, are designed to read normal text. However, if a PDF is scanned, even when it consist of only text, it is perceived as an image. These images can not easily be interpreted as text by a screen reader, and the document would for the users who require text to be read out loud, be completely inaccessible.

Structured elements.

PDF documents may contain structured elements, often referred to as “tags”.
Tags in PDFs are used to denote the semantics of the text such as headings, images, links etc. As an example, when headings are properly tagged, users can navigate through the document using the chapter and section headings. In contrast, if the PDF is untagged, headings and paragraph texts are indistinguishable and navigating using the headings is not possible. Clearly, for large documents, navigating through the document is challenging if not close to impossible.

It should be noted that as with (x)HTML, tags can be used improperly. Letting a PDF be tagged does not by itself guarantee
that the document is barrier free. This is rather prerequisite for accessible PDF documents.

Access to content.

Occasionally, PDF documents restrict users to copy the text. Unfortunately, by enabling this option, it affects the assistive technologies in a negative manner. These technologies, such as screen readers, are restricted from extracting the content.

Language specified.

When a text is read out it is essential that it is read in correct language / dialect as the text is written. As an example, synthetic reading of an English text with German pronunciation will be impossible to understand. It is therefor important that the language is specified in the PDF so that the proper pronunciation could be applied by screen readers. Most PDFs are missing language specification.

Test PDF accessibility.

To test for the above issues PDFs can be done with eAccessibility Checker and providing a URL toa PDF document. Note that the eAccessibility Checker is experimental and may at times be unavailable.

European Web Accessibility Ranking

July 6, 2009

The declaration of the UN Convention on the Rights of Persons with Disabilities by the United Nation has drawn attention to the topic of inclusion worldwide.
Among others, the convention includes:

  • Promote other appropriate forms of assistance and support to persons with disabilities to ensure their access to information;
  • Promote access for persons with disabilities to new information and communications technologies and systems, including the Internet;

This paper presents results from an accessibility evaluation of European governmental web sites, showing that Europe is far from reaching the goal of having all public web sites barrier free within 2010. In fact, 28% of all applied tests discover barriers which causes significant problems for people with special needs.

The evaluation has been carried out using the first fully automatic implementation of the Unified Web Evaluation Methodology. Evaluation of web accessibility is traditionally carried out manually which includes several challenges. In contrast, over approach avoids the usual bias part of manual accessibility evaluation. It further makes it possible for the first time to have a statistically sound comparison of web accessibility between European countries.

Country Ranking List

Accessibility of public web sites in Europe as number of barriers over applied tests ranking from 17% (most accessible) to 38% (least accessible). A dark (red) color means that the contry has inaccessible web sites.   The more inaccessible the web sites within the county the darker (more red) the country is.

Accessibility of public web sites in Europe as number of barriers over applied tests ranking from 17% (most accessible) to 38% (least accessible). A dark (red) color means that the contry has inaccessible web sites. The more inaccessible the web sites within the county the darker (more red) the country is.

European Web Site Scores

Ranking Country Score (Percentage of fail tests)
1 United Kingdom 17%
2 Sweden 20%
3 Czech Republic 21%
4 Netherlands 22%
5 Denmark 23%
6 Ireland 24%
7 Iceland 25%
8 Germany 26%
19 Italy 26%
10 Poland 27%
11 Norway 27%
12 Austria 28%
13 Slovenia 28%
14 Switzerland 29%
15 Portugal 30%
16 France 30%
17 Cyprus 31%
18 Belgium 31%
19 Hungary 32%
20 Luxembourg 34%
21 Romania 34%
22 Spain 35%
23 Bulgaria 38%
Europe Average 28%

This survey has been carried out using the eGovMon tool for measuring accessibility and was first published in Journal of Ph.D. of Papers in Technology and Science, 2008 at Aalborg University as How Accessible is the Public European Web by Morten Goodwin Olsen.