Integrate Azure Pipelines with Quality Engineering Insights

Integrate your Azure Pipelines CI setup with BrowserStack Quality Engineering Insights and view insights on different pipelines and jobs configured in Azure Pipelines.

Connect to Azure Pipelines using Azure DevOps PAT

You can configure the integration using an Azure DevOps Personal Access Token (PAT).

Create an Azure DevOps Personal Access Token (PAT)

1. Follow this Azure DevOps guide to create a personal access token within your Azure DevOps user account.
2. Create a token for all organizations by choosing ‘All accessible organizations’ from the Organization dropdown and grant the following permissions to the token so that BrowserStack Quality Engineering Insights can collect data to generate insights:
   • Build: Read & Write
   • Code: Read
   • Graph: Read
   • Project and Team: Read
   • User Profile: Read
   • Work Items: Read & Write

Connect to Azure Pipelines

• Step 1: Click on the Connect button on the Azure Pipelines tile on the Integrations page.
• Step 2: Enter the PAT created in previous section and click on Continue button.

• Step 3: Select organizations and Azure DevOps projects to import to BrowserStack Quality Engineering Insights. All of the pipelines of the selected projects would be imported on Quality Engineering Insights.
• Step 4: Click on View Insights button to start viewing rich insights from your Azure Pipelines setup on Quality Engineering Insights.

Set up Automation metrics

BrowserStack Quality Engineering Insights provides insights into various CI/CD Automation metrics such as job frequency, stability, build flakiness and many more.

Configure Job Type for different jobs

From the Automation tab on the Settings page, you can configure the job type for all the jobs (i.e. pipelines) fetched from your Azure Pipelines setup.
You can tag your jobs according to testing type, such as Regression, Functional, Deployments, Sanity etc. All jobs labeled with job types other than ‘Deployments’ will be considered Testing jobs and will be used to populate Testing-related insights. Jobs labeled ‘Deployments’ will be considered Deployment jobs and will be used to populate Deployment-related insights. The following metrics will start getting computed upon the configuration of job types:

• Job Stability
• Total Jobs Configured
• Job Frequency
• Build Duration

Store test output as build artifacts

BrowserStack Quality Engineering Insights cannot track tests related metrics such as Build Stability, Total Automated Tests, Build Flakiness unless you publish the test output artifacts on Azure. Azure Pipelines provides the PublishBuildArtifacts@1 task to publish build artifacts to Azure Pipelines. The following script demonstrates the use of this task to publish artifacts:

steps:
- task: Maven@3
  inputs:
    mavenPomFile: 'pom.xml'
    mavenOptions: '-Xmx3072m'
    javaHomeOption: 'JDKVersion'
    jdkVersionOption: '11'
    jdkArchitectureOption: 'x64'
    publishJUnitResults: true
    testResultsFiles: '**/surefire-reports/junitreports/*.xml'
    goals: 'clean test -Dsurefire.suiteXmlFiles=config/passtest.xml'
- task: CopyFiles@2
  inputs:
    Contents: 'target/surefire-reports/junitreports/*'
    TargetFolder: '$(Build.ArtifactStagingDirectory)'
- task: PublishBuildArtifacts@1
  inputs:
    pathToPublish: '$(Build.ArtifactStagingDirectory)'
    artifactName: 'JUnitReports'
    publishLocation: 'Container'

Troubleshooting

Need help? Reach out to us here.