Be Wary of Accessibility Guarantees from Vendors
In my ~20 years of responding to RFPs/RFQs, once organizations started to realize the value of accessibility (or fear of lawsuits), I saw more and more requests include a note on accessibility. In most cases this was just a single line item among many, often with nothing more than a checkbox: “Is your solution Section 508 compliant?”
Typically these requesting organizations don’t have the expertise within to evaluate whether a solution that has been delivered is genuinely accessible. More than once I have walked into a project after completion simply to deliver the bad news that the vendor failed to make good on its promise.
While at CSUN I saw a few conversations about third-party tools that have no accessibility built in at all, along with those that fail to meet claims made in their marketing or by their support teams.
As so many web sites and applications today are often built by hiring third-party developers or cobbling together features from different vendors, this something with which many organizations struggle.
Evaluating Vendors & Tools
Based on that ongoing frustration, I decided to propose a list of questions to ask and steps to take when evaluating a vendor or tool.
Look for an Accessibility Policy
Visit the web site of the vendor or tool maker. Look for an accessibility statement or policy, ideally linked prominently in the footer or similar. Read it and see what guarantees are made about the site, the organization, and how these are worded. This can be insight into an otherwise opaque tool or organization.
Look for Accessibility Tickets or Pull Requests
If you are evaluating an open source tool or if it maintains a bug tracking feature that you can see, then take a look through it to see what tickets are open. Look for an accessibility category. Look for how long tickets are open or merges sit. Check Stack Overflow for those tools that offload support there, as well as those that don’t.
Look for Accessibility References on Social Media
Take to the Twitters (and/or the social platform where you have the most experience) with the company or tool name and the #a11y hashtag. See if people are raising issues, sharing tips, or discussing work-arounds. Pay attention to the tone and frequency.
Look at Community Engagement
Similarly to looking for references on social media, look to see if the organization or its representatives are engaging users at all, and then compare that to how they engage users with accessibility issues. A large consultancy may not have a developer relations team, nor would that be appropriate for a company that builds bespoke solutions, but a tool maker might (or should, at a certain size).
While social media platforms may be great tools to use from your chair, remember that there are plenty of places to get together and share information in meat space. Perhaps among your team, at local meet-ups, co-working spaces, conferences, and so on. My guess is you don’t interact with all those people on social media (nor do they all spend time there).
Ask for Demos
If a vendor is selling a solution or if you are just choosing an add-on for your site, ask for a demo that shows the accessibility features. It may be a kitchen-sink style hodge-podge, a previous project, or just some widgets. Take some time to evaluate them, starting with the quickest test you know first. After all, how ARIA may be used on overlays is moot if you can’t tab through the page (probably the easiest first test).
Ask for Posts / Articles
Some organizations have written about solutions, best practices, or just want to share some neat problem and solution combinations related to accessibility (or not). Look to see if these exist and pay attention to how they might have been solved.
Ask for a Written Guarantee
Finally, if you are paying for a solution and the vendor has assured you that it will provide accessibility, ask for a written guarantee. Make it a separate document; just one page. When people are asked to sign promises they tend to think a little harder about that promise. Whether you choose to ask for it up front or at the end (telling the vendor at the start to expect it) is up to you and your process, but make sure the vendor knows it is coming.
Write Testing into the Contract
Shortly after I posted this, Lainey Feingold pointed out another good tip:
What to Do When You’ve Been Burned
File bugs. Contribute code fixes. Blog. Share the experience.
In the end, sometimes the only option that works is shaming an organization. Unfortunately, depending on the nature of the work your best bet may be to sue everybody.
It would be uncharacteristic of me to fail to highlight organizations that could do better. While none of them makes a promise to be accessible, the burden still lies on those making the request, so the techniques I outline above would quickly disqualify them.
TypeForm is currently a popular survey tool because of its visual style, but it is useless with a screen reader. Its opaque method of submitting bugs and requests, coupled with its stated (to me in email) unwillingness to incorporate even simple fixes, makes it a poor choice in any context. Though if you saw its (now corrected) failure to render results on Chrome/Android in the SitePoint CSS survey, you’d probably already have suspected this. A little testing would disqualify this as a tool immediately.
Slack isn’t a tool you add to a site, but it has become a popular collaboration platform for teams. While it has had an unfilled position for an accessibility product manager for a couple years, its unwillingness to eat its own dog food by requiring this person to move to San Francisco suggests that Slack is unwilling to make even a basic investment in making its product accessible. This says nothing of its use of a CAPTCHA on the application form. A little social media surfing should quickly disqualify Slack as a platform.
Wix is billed as a quick way to build a web site that is attractive. Both its tools and the sites it builds are known to be inaccessible. Some bug spelunking would reveal this and should immediately disqualify it as a viable solution, especially since adding accessibility isn’t something you decide to do only after it has enough votes.
If you want a third-party threaded discussion form, Disqus is quite the popular choice. It is also a pain to use with some assistive technology. In this case, if testing the tool is tricky you can always ask around. At CSUN more than a few hallway conversations were had, though sadly nobody could come up with a comparable accessible solution.
Polar is a polling tool (originally developed by Luke Wroblewski) that does not work with keyboards (as I noted in a mis-attributed bug report with WebCompat). Given that three of original developers are no longer there, its GitHub pages aren’t seeing any activity (as of 30 November 2016, the pages at github.com/input-factory have been purged and are not in the Internet Archive), and the Twitter account is dormant, you can expect accessibility issues probably won’t be getting addressed.
Update on Polar
Polar officially closed shop on May 18, 2016, as announced via a tweet.
Sitemorse (added July 12)
Sitemorse sells an automated testing product that presents its customers with a dashboard to show how a site is performing on a few different metrics, including accessibility. Unfortunately it appears from all the knowledge base materials that the accessibility testing is based on WCAG 1.0 (you have to expand the Accessibility bar first, then you can read about WCAG 1.0 and tables and WCAG 1.0 and link text). Given Sitemorse’s double claim that it is not an expert on WCAG (the second statement was in a comment it has since deleted), you should maybe take Sitemorse at its word and move along.
Update: March 28, 2016
Hah! VPAT = Very Packaged Alternate Truth (not Voluntary Product Accessibility Templates).
Update: April 27, 2016
I am of the opinion that offering tips on how to validate a vendor (as I have done above) is good. Requiring people to provide their contact and employer information to get access to those tips is only intended to turn them into sales leads, and not so good.
After all, if you have just identified a lead then isn’t the intent to sell them your own services? That means those tips are a sales tool and should be treated with appropriate suspicion.
Update: July 12, 2016
Beware WCAG 1.0
If a vendor builds an entire testing model, automated or not, around WCAG 1.0 then you should be suspect. Given that WCAG 1.0 was superceded by WCAG 2.0 in 2008, and given that international laws and now plenty of case law all reference or incorporate WCAG 2.0, you should consider throwing any vendor out of your office if it even breathes WCAG 1.0.
There is a very real risk you will be addressing recommendations that will not only fail to protect you in the case of a lawsuit, but because of differences between WCAG 1.0 and 2.0, can open you up to greater risk.
Update: January 30, 2017
I have seen an increase in individuals and companies spinning up sites in WordPress and selling their overall UX consulting services, usually with accessibility as a featured line item. Everything I touched on above still applies. In addition, another quick test is to identify if the theme they are using is an off-the-shelf theme and, if so, if it is an accessibility-ready theme. Couple that with a quick check in the WayBack machine and you may find the site, and the business, only has an eight month shelf life.
Update: February 1, 2017
Sadly, even with all those checks in place you can get screwed.
This is breach of contract. The next step is to seek damages. A concurrent step is to make others aware that this vendor is probably not up to the task.
Update: September 6, 2017
When there is no VPAT (because it is custom development work) you can use a questionnaire. Here is an example (it uses a Microsoft Word document):
At the Texas Department of Information Resources, we developed one such form, The Vendor Accessibility Development Services Information Request. Vendors must complete this form as part of any solicitation where development services are being procured. It asks for qualitative information regarding a vendor’s accessibility policies, organization, tools, training, corrective actions, and examples of prior accessible work. The completed form can be a significant indicator as to a vendor’s accessibility development capabilities; and can be used to gauge the level of confidence in the vendors testing and results, should that vendor be selected.
The rest of the post has some nuggets. Its overall message is that accessibility testing should always be the vendor’s responsibility, not the client’s.