The Challenge of Evaluating Product Accessibility

This blog post was written by Alex Li, Senior Accessibility Policy and Standards Strategist at Microsoft - Alex works with national and international standards organizations to improve the level of Accessibility for people with disabilities.


Throughout my career I have often been asked the question, “Is product X accessible?” by people new to the field as well as experts in accessibility.  At first consideration, the concept of a product being either “Accessible” or “inaccessible” seems reasonable; however, the complexities of technology, paired with the unique combination of abilities we all possess, make the concept of a product being “Accessible” less black and white.

There are many different types of disabilities, each requiring unique solutions.  Because of this, any given product or service will perform differently depending on the needs of the individual. This makes assessing a product or service far more involved than simply running through a yes or no checklist.  Such a binary approach fails to consider the complexity that comes with the gamut of disabilities, each with different performance requirements.

In my work, the term Accessible is often used to describe whether something meets a defined technical standard.  However, there is no single Accessibility standard that covers every type of disability. Should a product fail to meet an Accessibility standard, the designation does not necessarily differentiate between a failure of a core feature, a little used feature, or with all features. Conversely, using a binary approach to define a product as Accessible or not also fails to indicate where a product or service might actually exceed the requirements and recommendations. 

Imagine being a parent and only being able to see a single pass/fail grade for your child’s entire school year.  Wouldn’t it make it difficult to understand and help improve her individual performance in mathematics, literature, science, etc. if all of those details were reduced into a single mark? Evaluating student performance is impossible without more detail. Similarly, when we merely ask whether a product is Accessible or not we are not getting the full picture.

The more meaningful discussion for our industry is to discuss how products and services are Accessible, and for whom, rather than whether they are accessible. This framework provides a much richer context from which we can better understand and prioritize the improvements that are needed to meet the needs of people of all ages and abilities. 

Comments (4)
  1. Jay Wyant says:

    What taxonomy would you suggest we use for such a paradigm? Standards specifications such as that in WCAG 2.0 or Section 508? Or a different framework altogether?

  2. Jay thanks for the question: Alex has the following response…

    Obviously, there is a difference in scope between WCAG 2.0 and Section 508.  WCAG applies only to web content and Section 508 includes the web, software, telecommunication, etc.  But more relevant to the question is that Section 508 has the concept of functional performance criteria, best meet and equivalent facilitation.  WCAG 2.0’s conformance requirements, on the other hand, require complete fulfillment of all success criteria for all scenarios at the various levels of conformance (eg. Level A, AA, and in theory AAA).

    What we see as a best practice of Section 508 is that competing products and services are compared side-by-side and that preference is given to the vendor with the better overall accessibility performance. I believe that if more customers made their purchasing decisions based on relative accessibility performance (similar to 508) rather than just "Meets WCAG", the incentive for making more accessible products would be substantial.

  3. Jonathan Hassell says:

    Hi Alex,

    I couldn't agree more! WCAG is a great tool for enabling creators of products to understand some of the things they need to take into account to ensure they don't needlessly exclude some disabled or elderly audiences from being able to use their product. Unfortunately, the crude A, AA and AAA level system – one step removed from your single pass/fail grade – isn't so helpful, as it doesn't take into account anything about the product being created, and the costs and benefits of each of the checkpoints a product would need to comply with to get its badge.

    Add to that the user-research I have that huge numbers of disabled people who would need an Assistive Technology to get an accessible experience from a product or website actually don't have that Assistive Technology for reasons of cost, complexity or lack of awareness that ATs exist (google 'Web Accessibility Myths 2011 Hassell Inclusion' to find the research).

    So, understanding the whole perspective of a product's purpose, its target audiences, and what technologies those audiences have and can use, is essential for understanding how to make a product usable by as many people as possible. And user-testing with real users is essential in understanding whether the product you've made using that understanding, and guidelines like WCAG, has actually succeeded in your aim.

    Here in the UK we have developed a digital inclusive design framework called BS 8878 which includes all of these ideas and has been adopted as the British Standard for accessibility. And it's getting a lot of interest from all around the world.

    Anyone who agrees with Alex's views should check out summaries of it on the Hassell Inclusion site (google 'bs8878 Hassell Inclusion')

    Alex – I'd love to know what you think!


  4. Alex Li says:

    Thank you Jonathan!  I'd also recommend others to check out BSI 8878.  For those who are unfamiliar with it, I think it would be fair to characterize it as a development workflow for accessibility.  It is especially useful for those redesigning/reengineering their development process.

    Jonathan, I'll share more of my opinion with you offline.

Comments are closed.

Skip to main content