Accessibility testing, with 'pa11y-ci'

Caveat: this blog post is aimed mainly at software developers, and those with some familiarity with Node and npm.

Caveat 2: the examples below relate to pa11y-ci 1.3.* and pa11y 4., and may need adjusting for pa11y-ci 2.. and pa11y 5.x.

I’ve recently come across an open source, automated accessibility testing tool called pa11y-ci, and I’m really impressed with how easy it is to set up, and the usefulness of the results.

Note, tools like pa11y-ci are not of course a replacement for accessibility testing with humans.

pa11y-ci is written in Javascript for Node.js. It is built on top of pa11y, and HTML_CodeSniffer. And, it integrates well with continuous integration platforms like Travis-CI.

Here’s a run down on how I’m using it so far. (You’ll need Node.js, which includes npm.)

1. Install pa11y-ci

To get started, run this command in your terminal:

npm install pa11y-ci@^1.3 --save-dev

2. Configure pa11y-ci

Create a configuration file named .pa11yci.json, specifying a standard, some urls and other options. Here’s an example:

{
  "#": "Automated accessibility testing ~ https://github.com/pa11y/pa11y-ci",

  "defaults": {
    "standard": "WCAG2AAA",
    "timeout": 5000,
    "wait": 2000,
    "verifyPage": "gaad-widget-js"
  },
  "urls": [
    "http://127.0.0.1:9001/embed/?gaadwidget=force&...",
    "http://127.0.0.1:9001/test/static.html?..."
  ]
}

3. Add a pa11y-ci script

I’m eschewing Grunt et al, and simply adding scripts to my package.json:

{
  "devDependencies": {
    "pa11y-ci": "^1.3.1"
  },
  "scripts": {
    "pa11y-ci": "pa11y-ci --config .pa11yci.json"
  }
}

To run the above script, type this in your terminal:

npm run pa11y-ci

Here’s a more complete package.json:

{
  "name": "gaad-widget",

  "devDependencies": {
    "live-server": "^1.2.0",

    "pa11y-ci": "^1.3.1"
  },
  "scripts": {
    "test": "echo Testing ...",
    "serve-ci": "live-server --port=9001 -V --no-browser --watch=DUMMY",

    "pa11y": "pa11y --standard WCAG2AAA http://127.0.0.1:9001/#..",
    "pa11y-ci": "pa11y-ci --config .pa11yci.json"
  }
}

4. Add pa11y-ci to continuous integration

And, here is a simplified .travis.yml file:

language: node_js

node_js: 8

git:
  depth: 8

install: npm install

before_script: npm run build

script:
  - npm test
  - npm run serve-ci & sleep 5; npm run pa11y-ci;

Note: & after a command in bash pushes it to the background.

Example outputs

The above results in this Travis-CI job output:

> pa11y-ci --config .pa11yci.json
Running Pa11y on 2 URLs:
GET /embed/?gaadwidget=force&_ua=pa11y-ci 200 28.105 ms - 1708
GET /test/static.html?_ua=pa11y-ci 200 30.170 ms - 1792
GET /dist/gaad-widget.js 200 4.013 ms - 12631
GET /style/gaad-widget.css 200 3.276 ms - 1125
 > http://127.0.0.1:9001/embed/?gaadwidget=force&_ua=pa11y-ci - 0 errors
 > http://127.0.0.1:9001/test/static.html?_ua=pa11y-ci - 0 errors
✔ 2/2 URLs passed

The good:

• The Cloudworks home-page
• gaad-widget
• …

The bad:

• The comments on my blog (see job output) :(
• …

An example snippet follows, containing an error from the Travis-CI job output for my blog:

$ npm run pa11y-ci

> nfreear.github.io@1.0.0 pa11y-ci /home/travis/build/nfreear/nfreear.github.io
> pa11y-ci --config .pa11yci.json

Running Pa11y on 1 URLs:
 > https://nick.freear.org.uk/2017/05/14/gaad-widget.html?gaadwidget=force&_ua=pa11y-ci - 8 errors

Errors in https://nick.freear.org.uk/2017/05/14/gaad-widget.html?gaadwidget=force&_ua=pa11y-ci:

 • This form does not contain a submit button, which creates issues for those
   who cannot submit the form using the keyboard. Submit buttons are INPUT
   elements with type attribute "submit" or "image", or BUTTON elements with
   type "submit" or omitted/invalid.

   (#IDCommentPopupInner > form)

   <form> <label for="txtResolveEmail">P...</form>

...

✘ 0/1 URLs passed

If you’d like to use pa11y-ci, why not add a badge to your README? For example:

Articles by Andrew Mee, Ire Aderinokun and others contain more useful tips.

Update

16 March 2018: when you don’t want to publicise the test URL, for example, for a hidden acceptance server, you can set an environment variable, and dynamically generate the configuration using Javascript.

See this Gist, .pa11yci.conf.js for an example.

Update 2

17 May 2018: I’ve also got pa11y-ci running within GitLab CI.

See the .gitlab-ci.yml configuration file.