Evinced + BrowserStack

In our increasingly digital and mobile-first world, the speed of software development matters.

But sacrificing quality - and accessibility - for speed is a recipe for disaster. High-functioning teams today swear by testing as a hedge against such risks.

In this post, we will discuss how the Evinced Engine can be integrated with Appium tests on BrowserStack to detect accessibility issues before they slip into production.

Evinced is an enterprise-grade platform for accessibility engineering operations. With Evinced, software teams can weave accessibility into all aspects of development - from design, development, automated and QA testing to production monitoring.

Fragmentation makes accessibility testing a challenge

Appium is a popular open-source mobile automation tool with a significant advantage in its approach to automated mobile testing. It supports cross-platform testing to enable teams to test iOS and Android from the same framework.

With the addition of the Evinced Appium SDK, your Appium tests can return even more value by pinpointing accessibility issues. The Evinced API goes beyond what the iOS and Android accessibility APIs offer, helping you find more accessibility issues in your application. In addition, Evinced uses advanced algorithms to create a single actionable report that is easy to digest.

But the one thing missing from Evinced's powerful testing practice is the ability to test on a wide variety of devices and operating systems.

For Android, breadth of coverage is critical due to the large segmentation in the market - think Samsung, Google, LG, HTC, OnePlus, Huawei, etc. These manufacturers take the default Android OS image from Google and make it their own. This requires additional testing to ensure the best possible experience for all users.

For iOS, you may have noticed that once you upgrade your iOS device, it is impossible to roll it back to a previous version. Therefore, thorough testing requires a library of iOS devices on all the versions a potential customer might be using to ensure a great experience.

This can be quite a challenge for teams as hosting, maintaining, and updating devices and operating systems is an unnecessary overhead that takes the focus away from development.

Using the Evinced Engine with BrowserStack

Fortunately, BrowserStack has a comprehensive real device cloud where we can execute our Appium tests with the Evinced Engine to pinpoint accessibility issues on nearly any device and operating system combination.

Let's take a look at our Java JUnit Appium test running on a BrowserStack Device:

import com.evinced.a11y.validator.appium.java.core.A11yValidatorOptions;
import com.evinced.a11y.validator.appium.java.core.EvincedA11yValidator;
import com.evinced.a11y.validator.appium.java.core.Report;
import io.appium.java_client.MobileBy;
import io.appium.java_client.ios.IOSDriver;
import io.appium.java_client.ios.IOSElement;
import io.appium.java_client.remote.MobileCapabilityType;
import java.io.File;
import java.io.IOException;
import java.net.MalformedURLException;
import java.net.URL;
import java.util.List;
import org.openqa.selenium.remote.DesiredCapabilities;
import org.junit.AfterClass;
import org.junit.BeforeClass;
import org.junit.Test;
import org.openqa.selenium.support.ui.ExpectedConditions;
import org.openqa.selenium.support.ui.WebDriverWait;
public class EvincedA11yValidatorTest {
    public static URL url;
    public static DesiredCapabilities capabilities;
    public static IOSDriver<IOSElement> driver;
    @BeforeClass
    public static void setupAppiumDriver() throws
MalformedURLException, IOException {
        URL bsUrl = new URL("http://hub.browserstack.com/wd/hub");
        DesiredCapabilities capabilities = new DesiredCapabilities();
        capabilities.setCapability("device", "iPhone XS");
        capabilities.setCapability("os_version", "15");
        capabilities.setCapability("platformName", "iOS");
        capabilities.setCapability("app",
"bs://12345327ab8e755d13cf395490f2b22259c26306");
        capabilities.setCapability("automationName", "XCUITest");
        capabilities.setCapability("browserstack.user",
"<Your_BS_Username>");
        capabilities.setCapability("browserstack.key", "<Your_BS_Key>");
        driver = new IOSDriver<IOSElement>(bsUrl, capabilities);
        driver.manage().timeouts().implicitlyWait(20, TimeUnit.SECONDS);
    }
    @AfterClass
    public static void tearDown() {
        driver.quit();
    }
    @Test
    public void testStationsScreenIsAccessible() {
        IOSElement firstStationTableCell = driver.findElement(MobileBy.
className("StationTableViewCell"));
        firstStationTableCell.click();
        IOSElement playButton = driver.findElement(MobileBy.className
("XCUIElementTypeButton"));
        playButton.click();
        Assert.assertTrue(!playButton.isDisplayed());
    }
}

The first place we want to take a look at is the @BeforeClass method. This is where we will want to initiate the EvincedAppiumSdk class by passing in the IOSDriver instance that has created an Appium session on the BrowserStack device cloud. This will not impact any of the existing functionality of the driver, it will simply add the tools needed to add accessibility scans to our test. Check out the BrowserStack Capabilities Builder for all the information needed to select a cloud device. Next, add your Evinced service ID and API key to authenticate the Appium SDK. These credentials can be found by logging into the Evinced Product Hub.

@BeforeClass
public static void setupAppiumDriver() throws MalformedURLException,
IOException {
      URL bsUrl = new URL("http://hub.browserstack.com/wd/hub");
      DesiredCapabilities capabilities = new DesiredCapabilities();
      capabilities.setCapability("device", "iPhone XS");
      capabilities.setCapability("os_version", "15");
      capabilities.setCapability("platformName", "iOS");
      capabilities.setCapability("app",
"bs://12345327ab8e755d13cf395490f2b22259c26306");
      capabilities.setCapability("automationName", "XCUITest");
      capabilities.setCapability("browserstack.user",
"<Your_BS_Username>");
      capabilities.setCapability("browserstack.key", "<Your_BS_Key>");
    driver = new IOSDriver<IOSElement>(bsUrl, capabilities);
    driver.manage().timeouts().implicitlyWait(20, TimeUnit.SECONDS);
    // Init the EvincedAppiumSdk
    evincedSdk = new EvincedAppiumSdk(driver);
    evincedSdk.setupCredentials("<Evinced Service ID>","<Evinced API
Key>"); 
}

Now that we have added the capabilities to the driver, the next step is identifying where to add accessibility scans in the test for maximum coverage. This will likely be after a tap or a swipe that takes us to a new page or opens a menu. Taking a look at our test we can identify the following points:

@Test
public void testStationsScreenIsAccessible() {
    // The main application screen has loaded. Let's scan for a11y
issues!
    IOSElement firstStationTableCell = driver.findElement(MobileBy.
className("StationTableViewCell"));
    firstStationTableCell.click();
    // We have clicked on a cell that has taken us to a new page. Let's
scan for a11y issues!
    IOSElement playButton = driver.findElement(MobileBy.className
("XCUIElementTypeButton"));
    playButton.click();
    // We have now entered the player view. Let's scan for a11y issues!
    Assert.assertTrue(!playButton.isDisplayed());
}

Now that we have identified the points in our test that would be good to scan, let us add the code:

@Test
public void testStationsScreenIsAccessible() {
    // Let's scan for a11y issues!
    evincedAppiumSdk.analyze();
    IOSElement firstStationTableCell = driver.findElement(MobileBy.
className("StationTableViewCell"));
    firstStationTableCell.click();
    // Let's scan for a11y issues!
    evincedAppiumSdk.analyze();
    IOSElement playButton = driver.findElement(MobileBy.className
("XCUIElementTypeButton"));
    playButton.click();
    // Let's scan for a11y issues!
    evincedAppiumSdk.analyze();
    Assert.assertTrue(!playButton.isDisplayed());
}

In this test, we have added the scans directly to test code for demonstration purposes. For actual implementation, we would recommend adding the scans to existing page objects to make maintenance simple and easy.

The last thing we need to do is generate the report of all the scans. We can do that in the @AfterClass method.

@AfterClass
public static void tearDown() {
    List<Report> reports = evincedSdk.reportStored(true);
    driver.quit();
}

This single line of code will automatically compile all of the data from the scans and generate a set of easy-to-digest HTML and JSON report files. We are now ready to execute our test!

View of the BrowserStack dashboard

Not only can we run our test on any of the thousands of devices and operating system versions BrowserStack provides, we also gain additional insights from videos, logs, and screenshots to make debugging our functional test quicker and easier. This is extremely powerful combined with the Evinced HTML and/or JSON accessibility report containing all issues found, severity levels, descriptions, effect on end-users, and a link to actionable information on how to resolve the issues.

View of the Evinced report

For more information on getting started, check out Evinced and BrowserStack.

This post was originally written by Kevin Berg from Evinced and published by BrowserStack's editorial team.