Accessibility Scan¶
This Jupyter notebook processes the necessary data file of data_out/errors-different-counts-a11y-analyze-errors-summary.csv generated from the various processing scripts in various phases of the pipeline. The result of this notebook when executed will result in the generation of Figure 7 as described in the accompanying paper.
This file generates 2 figures;
- 7(a) Heatmap respresents errors reported by aXe engine
- 7(b) Heatmap respresents errors reported by HTMLCS engine
Required Libraries:¶
matplotlibnumpypandasseaborn
In [1]:
import matplotlib
import matplotlib.pyplot as plt
import numpy
import pandas
import seaborn as sns
matplotlib.rcParams.update({'font.size': 20, 'pdf.fonttype': 42, 'ps.fonttype': 42})
In [2]:
different_errors = pandas.read_csv('data_out/errors-different-counts-a11y-analyze-errors-summary.csv')
In [3]:
error_grid = different_errors[different_errors['Type'] == 'error'].pivot(index='Theme', columns=['Runner', 'DetailCode'], values='count')
Figure 7(a): Heatmap of Errors Types from aXe Engine¶
In [4]:
axe_errors = error_grid['axe']
In [5]:
axe_error_columns = axe_errors.columns
reassignment = {}
for i, column_name in enumerate(axe_error_columns):
reassignment[column_name] = f'AXE-E{i+1}'
reassignment_df = pandas.DataFrame.from_dict(reassignment, orient="index")
reassignment_df.rename(columns={0: 'Error Code'}, inplace=True)
reassignment_df
Out[5]:
In [6]:
axe_errors.rename(columns=reassignment, inplace=True)
axe_errors
Out[6]:
In [7]:
for updated_column in axe_errors.columns.tolist():
normalizer = axe_errors[updated_column].max()
axe_errors[updated_column] = axe_errors[updated_column] / normalizer * 100.0
axe_errors
Out[7]:
In [8]:
fig7a_description = """Figure 7(a) The heatmap of errors for aXe indicates the percentage of errors of different
types. The type of errors are presented on the x axis and the themes along the y axis with each cell ranging
between 0 and 100.0. The corresponding legend of the color bar is to the right side of the figure.
"""
In [9]:
fig, ax = plt.subplots(nrows=1, ncols=1, figsize=(10, 8))
sns.heatmap(axe_errors, robust=True, linewidths=0.1, linecolor='black', ax=ax, vmin=0, vmax=100)
ax.set_ylabel('Notebook Theme', fontsize=20)
ax.set_xlabel('Errors Reported by Axe Accessibility Engine', fontsize=20)
plt.xticks(rotation=90)
plt.yticks(rotation=0)
plt.savefig('plot_out/fig-7a-errors-axe-heatmap.pdf', bbox_inches='tight')
plt.savefig('plot_out/alt-embedded-images/fig-7a-errors-axe-heatmap.png',
metadata={'alt': fig7a_description},
bbox_inches='tight')
Figure 7(a) The heatmap of errors for aXe indicates the percentage of errors of different types. The type of errors are presented on the x axis and the themes along the y axis with each cell ranging between 0 and 100.0. The corresponding legend of the color bar is to the right side of the figure.
Figure 7(b): Heatmap of Errors Types from HTMLCS Engine¶
In [10]:
htmlcs_errors = error_grid['htmlcs']
htmlcs_error_columns = htmlcs_errors.columns
htmlcs_reassignment = {}
for i, column_name in enumerate(htmlcs_error_columns):
htmlcs_reassignment[column_name] = f'HTMLCS-E{i+1}'
htmlcs_reassignment_df = pandas.DataFrame.from_dict(htmlcs_reassignment, orient="index")
htmlcs_reassignment_df.rename(columns={0: 'Error Code'}, inplace=True)
htmlcs_reassignment_df
Out[10]:
In [11]:
htmlcs_errors.rename(columns=htmlcs_reassignment, inplace=True)
htmlcs_errors
Out[11]:
In [12]:
columns = htmlcs_errors.columns.tolist()
for updated_column in columns:
normalizer = htmlcs_errors[updated_column].max()
htmlcs_errors[updated_column] = htmlcs_errors[updated_column] / normalizer * 100.0
htmlcs_errors
Out[12]:
In [13]:
fig7b_description = """Figure 7(b) is similar to 7a, but contains 9 error types instead of 10 in 7a.
The key for the error codes along the x axis for both figures are presented in the table below.
"""
In [14]:
fig, ax = plt.subplots(nrows=1, ncols=1, figsize=(10, 8))
sns.heatmap(htmlcs_errors, robust=True, linewidths=0.1, linecolor='black', ax=ax, vmin=0, vmax=100)
ax.set_ylabel('Notebook Theme', fontsize=20)
ax.set_xlabel('Errors Reported by HTMLCS Accessibility Engine', fontsize=20)
plt.savefig('plot_out/fig-7b-errors-htmlcs-heatmap.pdf', bbox_inches='tight')
plt.savefig('plot_out/alt-embedded-images/fig-7b-errors-htmlcs-heatmap.png',
metadata={'alt': fig7b_description},
bbox_inches='tight')
Figure 7(b) is similar to 7a, but contains 9 error types instead of 10 in 7a. The key for the error codes along the x axis for both figures are presented in the table below.
| AXE | HTMLCS | WCAG 2AA | Error Description | Impact |
|---|---|---|---|---|
| AXE-E1 | HTMLCS-E1 | 1.4.3 G18 Fail | Text elements must have sufficient color contrast against the background | Serious |
| AXE-E2 | HTMLCS-E2 | 1.1.1 H37 | Images must have alternate text | Critical |
| AXE-E3 | HTMLCS-E7 | 1.4.3 F24 | Links must be distinguished from surrounding text in a way that does not rely on color | Serious |
| AXE-E4 | 2.4.4 H77, H78, H79, H80, H81, H33 | Links must have discernible text | Serious | |
| AXE-E5 | 2.4.1 G1, G123, G124, H69 | Page must have means to bypass repeated blocks | Serious | |
| AXE-E6 | 1.2.1 G159, G166 | <audio> elements must have a captions <track> |
Critical | |
| AXE-E7 | aria-hidden elements do not contain focusable elements |
Serious | ||
| AXE-E8 | ARIA input fields must have an accessible name | Serious | ||
| AXE-E9 | Certain ARIA roles must be contained by particular parent elements | Critical | ||
| AXE-E10 | All page content must be contained by landmarks | Moderate | ||
| HTMLCS-E3 | 4.1.1 F77 | Duplicate ID attribute value found on the web page | Minor | |
| HTMLCS-E4 | 1.3.1 H43, H63 | Tables not using header or scope attributes | Critical | |
| HTMLCS-E5 | 1.3.1 H43 Headers Required | Table Header Required and currently not used | Critical | |
| HTMLCS-E6 | 4.1.2 H91 EmptyNoId | Anchor elements with no ID or link content | Serious | |
| HTMLCS-E8 | 4.1.2 H91 NoContent | Anchor elements with valid link but no link text content | Serious | |
| HTMLCS-E9 | 4.1.2 H91 Button Name 4.1.2 H91 [Element] Name |
This [Element type] does not have a name available to an accessibility API. | Critical |