Integrate BrowserStack App-Automate with Azure Pipelines
Integrate your Appium test suite with Azure Pipeline AKA Visual Studio Team Services (VSTS) and BrowserStack device cloud using our Azure Pipelines extension.
Azure Pipelines is a continuous integration tool used to integrate your test suites. It enables continuous testing, build, and deployment of iterative code changes. CI/CD tools help catch failures ahead of the production stage and mitigate them as they occur.
The information in this section applies to using the BrowserStack plugin with Azure Pipelines as part of the Azure DevOps Services (cloud) offering only.
In this guide, you’ll learn how to:
- Install the BrowserStack Azure DevOps Extension
- Configure the App Uploader
- Configuring BrowserStack Local
- Generating test reports
Prerequisites
Before you can start integration, ensure that:
- You’ve hosted your project on any of these source repositories supported by Azure.
- You are a maintainer or owner of the project.
- You are an administrator of your Azure DevOps organization.
- You’ve created a pipeline for the project’s repository. If you are yet to create a pipeline, check out how to create an Azure Pipeline.
- You have access to the BrowserStack account credentials, namely Access Key and Username.
Install the BrowserStack Azure DevOps Extension
The BrowserStack Azure DevOps Extension eases the process of configuring Azure Pipelines to run tests on BrowserStack. To install the extension follow these steps:
- In the Visual Studio Marketplace, click Azure DevOps.
- Search for BrowserStack Azure DevOps Extension.
- In the BrowserStack Azure DevOps Extension page, click Get it Free.
- The Sign in page opens to sign in to your Azure account. After you sign-in, click Select an Azure DevOps organization and select your organization.
- Click Install to add the extension.
BrowserStack’s extension is now installed in your Azure DevOps organization. This installs the BrowserStack Config
, BrowserStackResults
, and BrowserStackStopLocal
tasks to the list of available tasks for you to use them in your pipeline.
Configure BrowserStack in Azure Pipelines
For your tests to run on BrowserStack, the Azure Pipeline needs additional information, such as BrowserStack credentials.
You can create a pipeline using either a YAML editor or a classic editor. Depending on the editor you used to create the pipeline, refer to the steps in the following tabs to integrate BrowserStack in your pipeline.
-
Open the Azure portal page and click My Azure DevOps Organizations.
-
In the Azure DevOps Organizations page, click the project where you’ve created a pipeline for your project’s repostory from the list of projects that appear.
-
Click Project settings located at the bottom-left corner of your project’s page.
-
In the Project details page, click Service connections > Create service connection.
-
In the New service connection section, select BrowserStack from the given list, and click Next.
-
In the New BrowserStack service connection section, enter your BrowserStack Username in the Username field, and then enter Access key in the AccessKey field.
-
Click Verify to verify your BrowserStack credentials.
-
After verification succeeds, enter a service connection name in the Service connection name field, such as
browserstack-connection
, and then click Verify and save. The service connection gets added and you’re redirected to the Service connections page. -
Click the (Pipelines) icon.
-
In the Pipelines page, select your pipeline.
-
Click Edit to edit the pipeline.
-
In the pipeline editor, from the Tasks pane, search for the BrowserStack Config task, and then select it.
-
In the BrowserStack Config section, select the connection, created in Step 8, from BrowserStack service end point
-
Click Add.
-
In the pipeline editor, from the Tasks pane, search for the BrowserStack Results task, and then select it.
-
In the BrowserStack Results section, enter
automate
in the Enter the product in use field, and then click Add.
The following image shows the azure-pipelines.yml
file with all the BrowserStack tasks configured to run your test on BrowserStack.
Ensure to set the build
capability in your test script using the environment variable BROWSERSTACK_BUILD_NAME
. The extension will fail to embed test reports in your pipeline if this capability is missing. Check out Edit your test script to call environment variables to learn more.
-
Open the Azure portal page and click My Azure DevOps Organizations.
-
In the Azure DevOps Organizations page, select the project from the given list.
-
In the project’s page, click the Pipelines option.
-
In the Pipelines page, click to select your pipeline.
-
In the pipeline’s page, click Edit to edit your pipeline.
-
In the Tasks tab, click the Add (+) icon next to the Agent Job 1 card.
-
In the Add tasks pane, select the BrowserStack Config task from the tasks list, and click Add.
-
Click the BrowserStack configuration setup card and click + New.
-
Create a new service connection using the following steps in the New service connection screen:
a. Enter your BrowserStack Username.
b. Enter your BrowserStack Access key.
c. Click Verify.
d. After verification succeeds, enter a name for service connection.
e. Click Verify and save.
-
In the Tasks tab, click + on the Agent job 1 card to add the BrowserStack Results task.
-
In the Add tasks panel, from the tasks list, select the BrowserStack Results task and click Add.
-
Click the BrowserStack Results card and enter
automate
in the Enter the product in use field. -
Click Save & queue and select Save to save the changes.
The following image shows all the BrowserStack tasks configured in the pipeline to run your test on BrowserStack.
Ensure to set the build
capability in your test script using the environment variable BROWSERSTACK_BUILD_NAME
. The extension will fail to embed test reports in your pipeline if this capability is missing. Check out Edit your test script to call environment variables to learn more.
Your build pipeline is successfully configured to connect with the BrowserStack cloud for your project.
Configuring App Uploader
BrowserStack Azure Pipelines extension allows you to upload your app to the BrowserStack servers.
Select the BrowserStack App Uploader task from the right task pane in the YAML edit window. This will set the BROWSERSTACK_APP_ID environment variable with the App Id returned as a response after the app upload.
If you would like to set a custom identifier for your uploaded app, set the ‘App custom id’ value. In this case the BROWSERSTACK_APP_ID environment variable will be set to the custom id you provided. Custom Id is optional.
Follow the below steps to upload app from your local machine to the BrowserStack servers.
-
Select BrowserStack App Uploader and update the App path to point the app location on your local machine. This will set the
BROWSERSTACK_APP_ID
environment variable with the App Id returned as a response after the app upload. -
If you would like to set a custom identifier for your uploaded app, set the ‘App custom id’ value. In this case the
BROWSERSTACK_APP_ID
environment variable will be set to the custom id you provided. Custom Id is optional.
Configure BrowserStack Local in Azure Pipelines
Use BrowserStack Local Testing to test your web applications hosted on private, development, and internal servers. With the BrowserStack Local Testing binary, you can create a secure, private connection between the BrowserStack Selenium Grid and your internal servers.
Configuring BrowserStack Local involves:
- Setting environment variables
- Editing test scripts to use environment variables
- Starting the BrowserStack Local
- Stopping the BrowserStack Local
- Using BrowserStack Local when behind proxy
Set variables for BrowserStack Azure DevOps Extension
The BrowserStack Azure DevOps extension, by default, sets the following environment variables:
BROWSERSTACK_USERNAME
BROWSERSTACK_ACCESS_KEY
BROWSERSTACK_LOCAL
BROWSERSTACK_LOCAL_IDENTIFIER
BROWSERSTACK_BUILD_NAME
When you create a service connection, the BROWSERSTACK_USERNAME
and the BROWSERSTACK_ACCESS_KEY
are automatically added as variables during the configuration step.
In addition, when you add the BrowserStack Config task in your pipeline, the BROWSERSTACK_LOCAL
and BROWSERSTACK_LOCAL_IDENTIFIER
environment variables are set automatically by the extension.
The extension also adds the BROWSERSTACK_BUILD_NAME
variable that autogenerates a build name for your test runs.
Edit your test script to call environment variables
Edit your test script to add the environment variables for setting capabilities using the following code snippets:
Ensure to pass the browserstackLocal
and localIdentifier
capabilities to run tests on your local development servers.
String userName = System.getenv("BROWSERSTACK_USERNAME");
String accessKey = System.getenv("BROWSERSTACK_ACCESS_KEY");
String browserstackLocal = System.getenv("BROWSERSTACK_LOCAL");
String browserstackLocalIdentifier = System.getenv("BROWSERSTACK_LOCAL_IDENTIFIER");
String app = System.getenv("BROWSERSTACK_APP_ID");
public static void main(String args[]) throws MalformedURLException, InterruptedException {
DesiredCapabilities caps = new DesiredCapabilities();
caps.setCapability("device", "Samsung Galaxy S8");
caps.setCapability("app", app);
caps.setCapability("browserstack.local", browserstackLocal);
caps.setCapability("browserstack.localIdentifier", browserstackLocalIdentifier);
}
driver = new AndroidDriver(new URL("https://"+userName+":"+accessKey+"@hub-cloud.browserstack.com/wd/hub", caps);
userName = process.env.BROWSERSTACK_USERNAME
accessKey = process.env.BROWSERSTACK_ACCESS_KEY
browserstackLocal = process.env.BROWSERSTACK_LOCAL
browserstackLocalIdentifier = process.env.BROWSERSTACK_LOCAL_IDENTIFIER
app = process.env.BROWSERSTACK_APP_ID
var capabilities = {
"browserstack.user" : userName,
"browserstack.key" : accessKey,
"app" : app,
"device" : "Samsung Galaxy S8",
"browserstack.local" : browserstackLocal,
"browserstack.localIdentifier" : browserstackLocalIdentifier
}
driver = wd.promiseRemote("https://hub-cloud.browserstack.com/wd/hub");
driver
.init(capabilities)
//Write your code here
.fin(function() { return driver.quit(); })
.done();
userName = Environment.GetEnvironmentVariable("BROWSERSTACK_USERNAME");
accessKey = Environment.GetEnvironmentVariable("BROWSERSTACK_ACCESS_KEY");
browserstackLocal = Environment.GetEnvironmentVariable("BROWSERSTACK_LOCAL");
browserstackLocalIdentifier = Environment.GetEnvironmentVariable("BROWSERSTACK_LOCAL_IDENTIFIER");
app = Environment.GetEnvironmentVariable("BROWSERSTACK_APP_ID");
DesiredCapabilities caps = new DesiredCapabilities();
caps.SetCapability("browserstack.user", userName);
caps.SetCapability("browserstack.key", accessKey);
caps.SetCapability("app", app);
caps.SetCapability("device", "Samsung Galaxy S8");
caps.SetCapability("browserstack.local", browserstackLocal);
caps.SetCapability("browserstack.localIdentifier", browserstackLocalIdentifier);
AndroidDriver driver = new AndroidDriver(new Uri("https://hub-cloud.browserstack.com/wd/hub"), caps);
user_name = os.getenv("BROWSERSTACK_USERNAME")
access_key = os.getenv("BROWSERSTACK_ACCESS_KEY")
browserstack_local = os.getenv("BROWSERSTACK_LOCAL")
browserstack_local_identifier = os.getenv("BROWSERSTACK_LOCAL_IDENTIFIER")
app = os.getenv("BROWSERSTACK_APP_ID")
desired_cap = {
'browserstack.user': user_name,
'browserstack.key': access_key,
'device': 'Samsung Galaxy S8',
'app': app,
'browserstack.local': browserstack_local,
'browserstack.localIdentifier': browserstack_local_identifier
}
driver = webdriver.Remote("https://" + user_name + ":" + access_key + "@hub-cloud.browserstack.com/wd/hub", desired_cap)
user_name = os.getenv("BROWSERSTACK_USERNAME")
access_key = os.getenv("BROWSERSTACK_ACCESS_KEY")
browserstack_local = os.getenv("BROWSERSTACK_LOCAL")
browserstack_local_identifier = os.getenv("BROWSERSTACK_LOCAL_IDENTIFIER")
app = os.getenv("BROWSERSTACK_APP_ID")
desired_cap = {
'app': app,
'device': 'Samsung Galaxy S8',
'browserstack.local': browserstack_local,
'browserstack.localIdentifier': browserstack_local_identifier
}
driver = webdriver.Remote("https://"+user_name+":"+access_key+"@hub-cloud.browserstack.com/wd/hub", desired_cap)
user_name = ENV["BROWSERSTACK_USERNAME"]
access_key = ENV["BROWSERSTACK_ACCESS_KEY"]
browserstack_local = ENV["BROWSERSTACK_LOCAL"]
browserstack_local_identifier = ENV["BROWSERSTACK_LOCAL_IDENTIFIER"]
app = ENV["BROWSERSTACK_APP_ID"]
desired_caps = {
'device': 'Samsung Galaxy S8',
'app': app,
'browserstack.local': browserstack_local,
'browserstack.localIdentifier': browserstack_local_identifier
}
appium_driver = Appium::Driver.new({
'caps' => desired_caps,
'appium_lib' => {
:server_url => "https://#{user_name}:#{access_key}@hub-cloud.browserstack.com/wd/hub"
}}, true)
To address the Build x not found error, ensure you set the environment variable BROWSERSTACK_TEST_RUN_NAME
to match the auto-generated build name of the BrowserStack Azure plugin. This can be achieved using the following command:
export BROWSERSTACK_TEST_RUN_NAME=$BROWSERSTACK_BUILD_NAME
By doing this, the test name will align with the build name generated by the BrowserStack Azure plugin, preventing the error.
Add task to start BrowserStack Local
When you add the BrowserStack Config
task, you have an option to select BrowserStack Local to run your tests on any developement or staging environment. Based on the editor you used to create your pipeline, refer to the steps in the following tabs:
-
Navigate to your pipeline and click Edit.
-
In the Tasks pane, select BrowserStack Config from the tasks list.
-
Select the service endpoint you created earlier.
-
Select the BrowserStackLocal checkbox, and then click Add. Note: You can set additional configuration options, such as enabling verbose logging, for the Local binary in the BrowserStackLocal Options input field. Check out Set Additional Configuration options to learn more.
-
In the Tasks pane, select BrowserStack Results from the tasks list.
-
In the BrowserStack Results section, enter
automate
in the Enter the product in use field, and then click Add.
The following image shows the azure-pipelines.yml
file with all the BrowserStack tasks and local configured to run your test on BrowserStack.
-
Navigate to your pipeline page and click Edit.
-
In the Tasks tab, click the BrowserStack configuration setup card.
-
In the BrowserStack Config section, select the BrowserStackLocal checkbox. Note: You can set additional configuration options, such as enabling verbose logging, for the Local binary in the BrowserStackLocal Options input field. Check out Set Additional Configuration options to learn more.
-
In the Tasks tab, click + on the Agent job 1 card to add the BrowserStack Results task.
-
In the Add tasks panel, from the tasks list, select the BrowserStack Results task and click Add.
-
Click the BrowserStack Results card and enter
automate
in the Enter the product in use field. -
Click Save & queue and select Save to save the changes.
The following image shows the azure-pipelines.yml
file with all the BrowserStack tasks and local configured to run your test on BrowserStack.
You must pass browserstack.local
and browserstack.localIdentifier
capabilities to test on your local development servers.
Set Proxy URL in the Pipeline
If your application is behind a proxy setup, you can specify the proxy host and port details as environment variables in Azure DevOps. Adding this information as variables ensures that the Local Testing binary passes test traffic through it.
Based on the type of proxy, either HTTP or HTTPS, you can set the value for HTTP_PROXY
or HTTPS_PROXY
, respectively, as shown:
HTTP_PROXY = "http://proxy-host>:<proxy-port>"
HTTPS_PROXY = "https://<proxy-host>:<proxy-port>"
If the proxy requires authentication, pass the credentials in the format of a basic auth URL, as follows:
HTTP_PROXY = "http://<username>:<password>@<proxy-host>:<proxy-port>"
HTTPS_PROXY = "https://<username>:<password>@<proxy-host>:<proxy-port>"
Depending on the editor you used to create your pipeline, refer to the steps in the following tabs:
-
Navigate to your pipeline and click Edit.
-
Click Variables located at the top-right corner of the page.
-
In the Variables section, click New Variable.
-
In the Variables section, enter
HTTP_PROXY
orHTTPS_PROXY
in the Name field, and enter its value, such ashttp://127.0.0.1:8080
, in the Value field. -
Click OK.
-
In the Variables section, click Save.
-
Navigate to your pipeline and click Edit.
-
Click the Variables tab to view a list of predefined variables with their respective values in the Pipeline variables section.
-
Click + Add and enter
HTTP_PROXY
orHTTPS_PROXY
in the name field, and enter its value, such ashttp://127.0.0.1:8080
, in the value field. -
Click Save & queue and select Save to save the changes.
Set Additional Configuration options
With the Local binary, you can set additional configurations, such as verbose logging, excluding hosts, and so on. To set any additional configuration options, add the flags in the BrowserStackLocal Options section. For example, to enable verbose logging and to route all traffic via the proxy, set the following flags in the BrowserStackLocal Options field:
{"verbose": true, "forceProxy": true}
Check out Flag for Local Binary to learn about more Local Testing parameters.
Follow Camel case style when adding local modifier options. For example, enter the --force-local
option as forceLocal
in the BrowserStackLocal Options section.
Add task to stop BrowserStack Local
It is recommended to stop the BrowserStack Local binary after tests have been executed to optimize your resources. Add the BrowserStack Stop Local task in your pipeline using the following steps:
-
Navigate to your pipeline and click Edit.
-
In the Tasks pane, search for BrowserStack Stop Local from the tasks list, and then select it.
The following image shows the azure-pipelines.yml
file with all the BrowserStack tasks configured to run your local test on BrowserStack.
-
Navigate to your pipeline and click Edit.
-
In the Task tab, click + on the Agent job 1 card.
-
Search for BrowserStack Stop Local and click Add.
-
Click Save & queue and select Save to save the changes.
The following image shows all the BrowserStack tasks configured to run your local test on BrowserStack.
Generating test reports
The BrowserStack Azure Pipelines extension enables you to embed the test results from BrowserStack into Azure Pipelines. We currently support viewing test reports in Azure Pipelines with a select set of languages and frameworks, which are Java with JUnit/TestNG, and Node with Protractor and WebdriverIO. The sections below describe how to configure your project to generate BrowserStack test reports.
Java - TestNG and JUnit
Follow the below steps to generate test reports.
A prerequisite for embedding test reports is that the Java project must be built with Maven and use either TestNG or JUnit for testing.
To configure your project:
- Add the following dependencies to your
pom.xml
file:
<dependency>
<groupId>com.browserstack</groupId>
<artifactId>automate-testassist</artifactId>
<version>1.1.0</version>
</dependency>
- Add the following repositories to your
pom.xml
file:
<repositories>
<repository>
<id>sonatype-nexus-snapshots</id>
<url>https://oss.sonatype.org/content/repositories/snapshots</url>
</repository>
</repositories>
<pluginRepositories>
<pluginRepository>
<id>sonatype-nexus-snapshots</id>
<url>https://oss.sonatype.org/content/repositories/snapshots</url>
</pluginRepository>
</pluginRepositories>
- Add the following plugins to your
pom.xml
file:
<build>
<plugins>
<plugin>
<groupId>com.browserstack</groupId>
<artifactId>automate-maven-plugin</artifactId>
<version>1.1.0</version>
<configuration>
<source>${jdk.source.version}</source>
<target>${jdk.target.version}</target>
<complianceLevel>${jdk.source.version}</complianceLevel>
</configuration>
<executions>
<execution>
<goals>
<goal>test-compile</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
You must define the Maven properties, jdk.source.version and jdk.target.version to match the Java versions of your project.
To configure WebdriverIO project, add wdio-browserstack-reporter
as a dependency in your package.json
file.
{
"dependencies": {
"wdio-browserstack-reporter": "~0.1.1"
}
Add browserstack
as a reporter in your conf
file.
// wdio.conf.js
module.exports = {
// ...
reporters: ['browserstack'],
reporterOptions: {
browserstack: {
outputDir: './'
}
},
// ...
};
outputDir(optional): Defines a directory location for storing browserstack report files. See the GitHub respository link for more details.
Embedding test reports
Once you have completed the above steps, you can configure your Azure Pipelines setup to embed all the BrowserStack Appium reports and logs.
-
Select Agent job 1 and click on the + icon. Type browserstack in the search box.
-
Hover on BrowserStack Results and click on the Add button. Click on Control Options and enable Continue on error.
With this integration, you can now view the results of your Appium tests within Azure Pipelines. This is how the test results look in Azure Pipelines:
Conclusion
By following the steps outlined in this guide, you should have a seamless integration between your existing automation in Azure Pipelines, your Appium tests and the BrowserStack Appium grid. This will help you leverage all the benefits of test automation and the scale and coverage offered by BrowserStack.
We're sorry to hear that. Please share your feedback so we can do better
Contact our Support team for immediate help while we work on improving our docs.
We're continuously improving our docs. We'd love to know what you liked
We're sorry to hear that. Please share your feedback so we can do better
Contact our Support team for immediate help while we work on improving our docs.
We're continuously improving our docs. We'd love to know what you liked
Thank you for your valuable feedback!