Automated Accessibility Testing Techniques For Native Mobile Apps

Introduction

Mobile applications should be accessible to everyone. Mobile accessibility testing enables this by simplifying the creation and testing of mobile apps. This is integral to creating a good user experience and is more than just a top priority.

But what are the best automated accessibility testing techniques for native mobile apps?

Find the answer to this question here. Examples will be used from the catalog page of the native demo app by SauceLabs to demonstrate the procedures used at Axelerant.

Automated Accessibility Testing For Native Mobile Apps

Accessibility test automation is a strategic choice that makes finding common problems in new features easier and quicker and reduces regressions in existing features. At its most basic, automating accessibility testing includes running a set of accessibility rules on a mobile app's most crucial user interfaces.

The main benefit of automated accessibility testing is that it can find errors that are easy to fix because of mistakes in the code. This frees up engineering time and addresses frequent flaws that are usually ignored.

Another advantage of mobile accessibility testing is that it prevents regressions. A sudden rise in the number of problems will be a signal that there are regression errors. For instance, to combat this,  BrowserStack's automated accessibility lets teams monitor the health of regression builds with simplified automation. Run the builds as usual, and accessibility scans will trigger automatically wherever a DOM change is detected.

In an environment with daily builds where it is not practical to manually test every new release, this becomes a crucial tool to run in the CI/CD pipelines. Accessibility test automation rules are also run during continuous integration to address issues before release and prevent code-level accessibility issues.

The continuous integration pipeline at Axelerant incorporates several open-source and paid automation frameworks. And the experts at Axelerant use a mix of automated, semi-automated, and human-driven testing to find a wide range of accessibility problems in mobile apps.

Automated Accessibility Testing Using Espresso For Android

Espresso is an open-source Android user interface testing framework developed by Google. It is created to speed up and simplify UI testing. In addition, it enables interaction with the UI elements in the app that is being tested.

Integrating accessibility checks with Espresso is an important step in automated accessibility testing. With Google's Accessibility-Test-Framework / ATF, users can add accessibility checks to existing Espresso tests. Please be aware that the application's complete source is required for this solution.

Configure The Solution With Accessibility Checks

The steps for configuring the solution with accessibility checks are:

1. Begin by importing the dependencies into the build.Gradle file.

2. Next, import the accessibility checks into your test file.

3. Add an accessibility check to the test file using the @BeforeClass annotation in the Espresso test.

4. The AccessibilityChecks class will enable and configure accessibility testing. When this is done, the tests are automatically launched by executing any view action listed in ViewActions.

To evaluate a screen’s view hierarchy for every check, pass true while calling setRunChecksFromRootView().

5. A ViewAction from the ViewActions class is necessary for conducting Accessibility-Test-Framework / ATF checks. It should be noted that ATF checks are not performed if a ViewAction is not used to interact with a view.

6. Use addGlobalAssertion to turn on accessibility checking globally above the tests.

7. Use removeGlobalAssertion to turn off the accessibility check.

8. Finally, there is a choice to overlook any particular guidelines. This is useful if one is aware that a specific rule will fail or is found to be problematic. Overlooking a rule should be the exception, not a practice, which can also be achieved by specifying a matcher.

Accessibility Coverage

When tests for accessibility checks are enabled, the following guidelines are applied:

• Touch Target Size: An error is thrown if the target height or width is less than 48 dp unless a touch delegation is detected.
• Text Contrast: An error is thrown if the contrast ratio, 4.5 for regular text and 3 for large text, is not met.
• Duplicate Speakable Text: Throws an error if two views in a hierarchy have the same speakable text.
• Speakable Text Present: Throws errors if the view lacks speakable text needed for a screen reader.
• Editable Content Description: An error is thrown if an editable text view has a content description.
• Clickable Span: Throws an error if Clickable Span is inaccessible.
  Redundant Content Description: Issues a warning if a redundant word appears in the content description.
• Duplicate Clickable Bounds: Throws an error if a clickable view has the same bounds as another.

The categories of the reported issues are listed below.

• This item's size is 20dp x 20dp. Consider making this touch target 48dp wide and 48dp high or larger.
• This item may need a label readable by screen readers.

The disadvantage of this approach is that it does not generate a high-quality, user-friendly, and readable report. The report does not contain any screenshots or marked-up text. It also provides the elements' coordinates and advice on addressing any accessibility problems in a cluttered manner.

Here is a screenshot of the report generated by the accessibility testing framework.

Automated Accessibility Testing Using XCUITest In iOS

The XCUITest framework is a sub-framework within the XCTest Testing framework provided by Apple for UI testing of iOS apps. A11yUITests is an XCTestCase extension that includes tests for typical accessibility problems that can be executed as part of an XCUITest suite. Tests can be run separately or integrated into existing XCUITests written in the Swift language. 

Configure The Solution With Accessibility Checks

• Import A11yUITests into the XCTestCase subclass where accessibility checks are required.
• Add A11yCheckAllOnScreen() to ensure that every component on the screen is accessible. Here's an illustration of a snippet.

• The solution can be run on GitHub CI, and having accessibility tests as part of a CI/CD suite ensures that the app has a certain level of accessibility quality.

Accessibility Coverage

It is possible to test for 12 accessibility checks using XCUI.

• Minimum Size
• Valid Accessibility Label
• Redundant Element Type
• Uppercase Button Labels
• Button Labels Are Not Sentences
• Redundant Image Labels
• Image File Names
• Short Labels
• Tap Target Size
• Duplicated Labels
• Overlapping Elements
• Text Outside a Scroll View

Once the automated framework has been run, improvements to accessibility will be reported. But what if, after running the tests, the build fails? In that case, a list of accessibility problems will be provided.

A report in HTML called tests.html will also be produced and stored in the "build/reports" directory. Check out XCPretty's numerous formatting options.

The framework identified the following categories of problems after running the solution for the catalog page:

• The interactive element is not tall enough
• The interactive element needs to be wider
• The button should not contain the word button in the accessibility label
• Labels are not meaningful
• Elements have duplicate labels
• The screen has no element with a header trait

Implement Automated Accessibility Tests Using Appium And The Evinced Flow SDK

Evinced Mobile SDKs, which come with a 14-day trial period, can be integrated with common mobile testing frameworks to detect accessibility issues. By adding a few lines of code, programmers can use the Evinced SDK to find accessibility problems.

The Evinced Appium SDK supports test frameworks like Appium Java, Espresso, and XCUITest. Unlike native solutions, the framework can be installed without the application's source code. A rich and detailed HTML or JSON report is generated at the end of the tests to track issues in any reporting tool.

Configure The Solution With Accessibility Checks

1. Add evincedAppiumSdk.analyze() to the test file to perform an accessibility check.

2. Add scrolling logic to the page to detect accessibility problems throughout the entire page.

3. Depending on the platform where the example test will be run, the execution OS variable should be set to "ANDROID" or "IOS" in the AppiumController class (AppiumController.java). For Android, it should run on the emulator and the device, and for iOS, it will only run on the simulator.

4. The capabilities required to run the Android framework are mentioned in the snippet. Similar configuration options are available for Android, iOS, and BrowserStack.

5. The snippet below describes how to run the automated tests for the My Demo Android App on Android and iOS.

6. To run the tests across various platforms, Evinced-Appium-SDK Solution offers an integration solution with Browserstack. Read more to learn about integration with Evinced.

The guidelines used by Evinced in its SDK and Mobile Flow Analyzer to check an app for accessibility are the same. This is why either one can be implemented. But most users still prefer using the SDK over a mobile flow analyzer.

This is because the SDK allows all URLs to be scanned simultaneously in a fully automated solution. But a mobile flow analyzer needs users to scan one page at a time while manually scrolling.

The Evinced recognizes the following problem types:

• Accessibility Not Enabled
• Accessible Name
• Colliding Controls
• Color Contrast
• Duplicate Role
• Interactable Role
• Label Capitalization
• Meaningful Accessibility Label Buttons
• Meaningful Alternative Text
• Screen Orientation Consistency
• Screen Orientation Lock
• Sentence Like Label
• Special Characters
• Tappable Area
• Text Resizing
• Type In Label

Once the automated framework has been executed, accessibility enhancements will be reported. The reported categories of problems are:

• The tappable area of this element is too small
• There is no accessible label for the element
• More than one interactable element has the same name
• This element's label contains one or more special characters that may not be readable

For Android:

For iOS:

A comprehensive HTML report is produced once the tests are completed. Screenshots of the report are given below when generated from an iOS device and an Android device, respectively.

Feature Comparison: ATF Vs. A11yUITests Vs. EvincedSDK

All three automated solutions discussed here can be used in different situations. Use the comparison chart below to determine the most suitable option for your organization.

Conclusion

The Evinced is a commercial library that provides an effective accessibility solution for both iOS and Android apps. It supports both black box and white box accessibility testing and generates an advanced HTML report that lists accessibility issues.

But if the tester can access the app's source code, native solutions with basic-level reporting can be used, like Espresso with AFT for Android and XCUI with AllyUITest for iOS. On the other hand, without access to an application's source code, testers can opt for Evinced SDK.

The experts at Axelerant use a hybrid approach of automated, semi-automated, and manual test solutions. Learn more about this approach here.