Towards Automated eGovernment Monitoring

September 26, 2011

Morten Goodwin’s Ph.D. thesis, with the title Towards Automated eGovernment Monitoring, is now available online.

Illustration photo of digital government

EGovernment solutions promise to deliver a number of benefits including increased citizen participation. To make sure that these services work as intended there is a need for better measurements. However, finding suitable approaches to distinguish the good eGovernment services from those which need improvement is difficult. To elucidate, many surveys measuring the availability and quality of eGovernment services are carried out today on local, national and international level.

Because the majority of the methodologies and corresponding tests rely on human judgment, eGovernment benchmarking is mostly carried out manually by expert testers. These tasks are error prone and time consuming, which in practice means that most eGovernment surveys either focus on a specific topic, small geographical area, or evaluate a small sample, such as few web pages per country. Due to the substantial resources needed, large scale surveys assessing government web sites are predominantly carried out by big organizations. Further, for most surveys neither the methodologies nor detailed result are publicly available, which prevents efficient use of the surveys results for practical improvements.

This thesis focuses on automatic and open approaches to measure government web sites.

The thesis uses the collaboratively developed eGovMon application as a basis for testing, and presents corresponding methods and reference implementations for deterministic accessibility testing based on the unified web evaluation methodology (UWEM). It addresses to what extent web sites are accessible for people with special needs and disabilities. This enables large scale web accessibility testing, on demand testing of single web sites and web pages, as well as testing for accessibility barriers of PDF documents.

Further, the thesis extends the accessibility testing framework by introducing classification algorithms to detect accessibility barriers. This method supplements and partly replaces tests that are typically carried out manually. Based on training data from municipality web sites, the reference implementation suggests whether alternative texts, which are intended to describe the image content to people who are unable to see the images, are in-accessible. The introduced classification algorithms reach an accuracy of 90%.

Most eGovernment surveys include whether governments have specific services and information available online. This thesis presents service location as an information retrieval problem which can be addressed by automatic algorithms. It solves the problem by an innovative colony inspired classification algorithm called the lost sheep. The lost sheep automatically locates services on web sites, and indicates whether it can be found by a real user. The algorithm is both substantially tested in synthetic environments, and shown to perform well with realistic tasks on locating services related to transparency. It outperforms all comparable algorithms both with increased accuracy and reduced number of downloaded pages.

The results from the automatic testing approaches part of this thesis could either be used directly, or for more in-depth accessibility analysis, the automatic approaches can be used to prioritize which web sites and tests should be part of a manual evaluation.

This thesis also analyses and compares results from automatic and manual accessibility evaluations. It shows that when the aim of the accessibility benchmarking is to produce a representative accessibility score of a web site, for example for comparing or ranking web sites, automatic testing is in most cases sufficient.

The thesis further presents results gathered by the reference implementations and correlates the result to social factors. The results indicate that web sites for national governments are much more accessible than regional and local government web sites in Norway. It further shows that countries with established accessibility laws and regulations, have much more accessible web sites. In contrast, countries who have signed the UN Convention on the Rights of Persons with Disabilities do not reach the same increased accessibility. The results also indicate that even though countries with financial wealth have the most accessible web sites, it is possible to make web sites accessible for all also in countries with smaller financial resources.

Full disclosure: I am the author of the thesis.

Global Web Accessibility

April 7, 2011

Cover of Journal of Information Technology and Politics

A scientific publication titled Global Web Accessibility Analysis of National Government Portals and Ministry Web Sites (Morten Goodwin, Deniz Susar, Annika Nietzio, Mikael Snaprud, Christian S. Jensen) was recently published.

The publication presents web accessibility benchmarking methodology, and uses this methodology to present a survey on the accessibility of public web sites in the 192 United Nations Member States. It further identifies common properties of Member States that have accessible and inaccessible Web sites and shows that implementing antidisability discrimination laws is highly beneficial for the accessibility of Web sites, while signing the UN Rights and Dignity of Persons with Disabilities has had no such effect yet. The article also demonstrates that, despite the commonly held assumption to the contrary, mature, high-quality Web sites are more accessible than lower quality ones. Moreover, Web accessibility conformance claims by Web site owners are generally exaggerated.

The countries with web sites that receive the best accessibility scores are:

  1. Germany
  2. Portugal
  3. Spain

The survey also shows that the economy of a country influences the accessibility of web sites, so that, not surprisingly, wealthy countries have more accessible web sites than poor countries. However, the study shows that accessibility laws have more impact than the financial status. Thus, it is not necessarily costly to make web sites accessible. It is however important to have well established accessibility laws which are actively followed up.

(Full disclosure: I am a co-author of the paper)
Morten Goodwin

Automatically finding inaccessible alternative texts in web pages

September 24, 2010

A publication on Automatic Checking of Alternative Texts on Web Pages (Olsen, Snaprud, Nietzio) was recently published.

Often alternate texts for images, maps of audio files, are generated by web publishing software or not properly provided by the editors. For humans it is relatively straightforward to see which alternative texts have been generated automatically as the texts are in no way describing the corresponding image. Examples include texts such as ”Image1”, texts which resemble filenames such as ”image12.png” or ”insert alternative text here”.

The proper method for adding images to a document is when an editor uploads an image for article and can / must provide an alternative text in the CMS.

There are however several improper methods which results in in-accessible automatically generated alternative texts:

  • The editor uploads an image and uses the default alternative text.
  • The editor uploads an image for an article and the CMS generates some (often strange) alternative text.
  • The editor uploads an image but have no possibility to write an alternative text.

Following are some example of automatically generated alternative texts (Image source wikipedia)

A picture of a dog eating with a correct alternative text: Golden Retriever Eating

Correct Alternative text "Image12.png" HTML: <img alt=”Golden Retriever Eating” ... />

A picture of a dog eating with a wrong alternative text: image12.png

Wrong Alternative text "Image12.png" HTML: <img alt=”image12.png” ... />

A picture of a dog eating with a wrong alternative text: image12.png

Wrong Alternative text "Image12.png" HTML: <img alt=”image12.png” ... />

For people who cannot see non-textual content alternative texts are crucial to understand and use the content and automatically generated alternative texts may impose web accessibility barriers. Most automatic accessibility checkers only detects for the existence of alternative texts. The above mentioned texts, which are not describing the corresponding image well and are thus not considered accessible, will not be detected.

The paper introduces a pattern recognition approach for automatic detection of alternative texts that may impose a barrier. The introduced algorithms reach an accuracy of more then 90%, which should hopefully be a step towards improving the usefulness of automatic accessibility checking. Additionally, it could be useful input of manual accessibility checking.

(Full disclosure: I’m a co-author of the paper)

Remaining challenges of measuring the accessibility of web sites according to WCAG 2.0

August 11, 2010

The Web Content Accessibility Guidelines (WCAG 1.0) was launched in 1999 and was followed up by WCAG 2.0 in 2008. These guidelines have been the de facto standard for how to make web sites accessible for all people, including people with special needs.

Accessibility Sign

During the 9 year period from 1999 from 2008, many measurement methodologies for WCAG 1.0 was created. Furthermore, many national and international surveys have benchmarked the accessibility of public web sites according to WCAG 1.0. Since WCAG 2.0 differ from WCAG 1.0 in significant ways, the existing measurement methodologies cannot easily be translated to WCAG 2.0. Thus, very few applications for evaluation according to WCAG2.0 has been produced. Only two tools claiming to be WCAG 2.0 compliant are known to the authors: AChecker and TAW. The details of these tools are not known.

A paper titled Evaluating Conformance to WCAG 2.0: Open Challenges (Alonso, Fuertes, Gonzalez, Martínez) presented the remaining challenges of measuring accessibility of public web sites according to WCAG 2.0. In this paper, the authors identify the main challenges with measuring measuring accessibility in web sites in accordance to WCAG 2.0. The lessons have been learned by applying WCAG 2.0 tests in practice by university students.

The paper identifies the following challenges. The described challenges are in the authors experience unclear parts WCAG 2.0, which often means that the testers need interpret the texts and take decisions of how it should be understood. This could easily lead to inconsistency among testers as the testers may understand the texts differently.

Accessibility supported Technologies

WCAG 2.0 describes that only accessibility supported technologies can be relied upon for accessibility. It further states that the technology is accessibility supported only when user’s assistive technology will work with it. Since no list of supported technologies is provided, nor any formal way to measure if a technology is supported or not, this causes a challenge. There are no established method of saying that using one technology is accessibility, while using another is not.

Testability of Success Criteria

WCAG 2.0 consists of testable techniques. A technique is testable if it can be tested ether with machine or by human judgment. It is believed that around 80% of the criteria are testable by humans. However, the authors show that some of the description of the techniques for testing causes confusion. For example: in the sentences, “the test sequence of elements should be meaningful”, it is not evident what is meant by the wording meaningful. What is understood as “meaningful sequence of elements” for one person may not be meaningful for others. This is likely to cause confusion, which leads to inconsistency in any testing results.

Openness of Techniques and Failures

WCAG 2.0 is divided to separate documents: the guidelines and techniques. The guidelines are stationary and technology independent. In contrast the techniques is a living document which is updated as technology evolves. This makes it possible to update WCAG 2.0 with hands on techniques as the technologies used on the web evolve. One challenge is that W3C updates the techniques document for non-proprietary software only. This means that there will be no techniques collected by W3C for proprietary software, such as for example Adobe Flash. Thus, there will be no techniques from W3C on how to make Adobe Flash accessible.

Aggregation of Partial Results

How to present data from successfull techniques and common failures have not been presented by W3C. WCAG 2.0 identifies two types of criteria an element can match:

  • Positive: Elements which meet the criteria of successfull techniques. Any elements which uses the successfull techniques are known to be accessible.
  • Negative: Elements which is a common failure. Any elements which uses a common failure, is known to be in-accessible.

It is not so that the successfull techniques and common failures are opposite measures. Thus, not following a success technique does not mean that a barrier exist. Similarly, it is not so that avoid a common failure necessarily means that the element is accessible. Therefor, elements which nether match the successfull techniques nor common failures fall into some unknown state and cannot be claimed to be accessible nor in-accessible.

How to present data from a web page with common failures and successfull techniques are not clear.


The author further present some recommendations when measuring web accessibility according to WCAG 2.0. The recommendations are as following:

  • Accessibility-supported techniques should be clearly defined, and a methodology to identify if a techniques is accessible-suppported, or not should be established.
  • More experiments are needed for the testability of the techniques, failures and success criteria. This should be a step towards creating a common understanding of how the tests should be interpreted.
  • W3C should define how test results from successfull use of techniques, common failures, and not applicable should be aggregated and presented as a single result.

Web Accessibility Checking

April 22, 2010

eGovMon logoA new version of the eAccessibility Checker has been launched by the eGovMon-project.

The tool targets checking how accessible web pages and web sites are for people with special needs. This new release focus on being understandable both for content providers and web developers. People no longer need to be web accessibility experts to find out both the accessible status of a web page and how to improve it.
The tool also includes an accurate presentation of the code ((X)HTML and CSS) which creates barriers. As well as good and bad examples of web accessibility.

Can you make your web site accessible and get the Checked by eGovMon-logo?

Smarter, Faster, Better eGovernment

December 10, 2009

Capgemini has released its 8th eGovernment Benchmark Measurement called Smarted, Faster, Better eGovernment.

International eGoverment surveys are always interesting. This is especially true when surveys include an aim at measuring user experience, as the Capgemini survey does.

In this survey, user experience is divided into six sub categories:

  1. Usability
  2. Accessibility – Manual
  3. User Satisfaction
  4. One-Stop-Shop
  5. User Focus
  6. Accessibility – Automatic

From the above user experience categories, there are some interesting points to take note from. The manual evaluations were limited to checking if the national portals could be viewed with various font sizes. It turns out that this was possible for almost every portal and the metric was finally dropped.

Another interesting point is that Capgemini has chosen to measure web accessibility automatically using the technology from the European Internet Accessibility Observatory (EIAO). Capgemini states that technology conducts a series of automated tests following the  Unified Web Evaluation Methodology. From each national portal, 6000 pages were downloaded and about 300 pages examined. (It should be noted that EIAO is no longer being developed. The project measuring Web accessibility automatically, building on the EIAO application, is eGovMon.)

According to Capgemini, the top five countries in Europe when it comes to Web accessibility is:

  1. Austria,
  2. Netherlands,
  3. Denmark,
  4. Norway and
  5. Germany

This ranking list is further supported by Capgemini stating that Austria regularly proceeds with self assessment of government Web sites. Furthermore, that the Netherlands have a strict Web accessibility policy; all Web sites launched after September 1st 2006 in the Netherlands must conform to strict accessibility guidelines within the end of 2010.

Online Tools for Analyzing your Web Site

October 19, 2009

It is unfortunately not possible to get a complete overview of how well a web site is working for the users by only using automatic analysis tools. However, there exists many tools which give valuable indications.

Web Accessibility

It should be clearly noted that in order to make a web site successfully working for all users, including users with disabilities, automatic accessibility measurements is not sufficient. In fact, only 20% of the tests defined in the Unified Web Evaluation Methodology can be measured automatically. However, many tools of automatic accessibility evaluation exists. Following are some of these;

  • The eAccessibility Checker detects web accessibility barriers using tests from the Unified Web Evaluation Methodology.
  • Walidator is another online tool based on the Unified Web Evaluation Methodology. In addition to automatic evaluation, the Walidator also assists in manual evaluation.
  • WAVE is interesting tool for web accessibility analysis which, in contrast to most other tools, does not a report or list of all the detected barriers. It presented the evaluated web page visually as well as clearly marking the detected barriers.
  • TAW is a tool which, in addition to WCAG1.0 and WCAG2.0 could present accessibility according to the W3C mobileOK Basic Tests.

Code Validation

(x)HTML and CSS are the most used technologies on web pages. There are many reasons to have valid (x)HTML and CSS. To mention a few; valid code is important to make sure the web pages are rendered similarly in different browsers, to help faster loading in web browsers and to reduce maintenance. The World Wide Web consortium (W3C) has also gathered opinions from the web community validation of web pages.

I would be surprised if many disagree with me when I say that hard to take a professional web site seriously if there has not even valid HTML. Thankfully, making sure web pages are implemented with valid (x)HTML and CSS is both easy and fool proof.

  • The classic and de facto standard of (x)HTML validation is the W3C Markup Valid Service.
  • Similarly, the de facto standard for CSS validation is the W3C Jigsaw.
  • Additionally, the CSE HTML Validator Lite does similar work as the W3C Markup Validator, but it has a little more detailed interface.

Broken Links

Another issue which can easily be checked automatically is detecting links which are no longer working (broken links). When you create a web site you typically link to the relevant resources – both internal and external. As the web site grows, chances are that the number of links to external resources also grows. Naturally, it would be a tedious process to manually verify that your links are still working, which is why we have automatic tools for doing exactly this:

  • One such tool is the SiteReportCard. In addition to broken links, it also checks for spelling errors, search engine hits and more. This is definitely worth a try.
  • Another similar tool to check for broken links, HTML syntax, etc. is Dr Watson.


Web Site performance can also be carried out automatically.

  • has a web page speed test.  This tool provides information on the web page size and download duration with various connection rates. In addition, it provides suggestions on improve the performance of the evaluated web site such as adding HTTP compression, reducing the size of scripts etc.
  • Load Impact is a more  thorough performance analysis tool. Among the useful features, it sends multiple requests to the evaluated web site in order to get an overview of how many simultaneous requests (users) the web site can handle.