An important part of automated test monitoring and control is to monitor, inspect, and react to the results. This is best accomplished in the test management and bug tracking system that you use for your daily work.
In this article, you can review three different scenarios on how to trigger and create/update bugs in the Azure DevOps Server, formally known as Visual Studio Team Services (VSTS) or Team Foundation Server (TFS), based on the results of test cases running in LEAPWORK.
Use a plugin for Azure DevOps Servers
Use the LEAPWORK plugin to easily integrate with the Azure DevOps Servers. This extension can be installed from the Visual Studio Marketplace as follows:
Navigate to the Visual Studio Marketplace.
Type LEAPWORK in the search bar and select it.
Now go back to your Project in DevOps and Select the required project in DevOps.
Now create a release pipeline for the associated project. Select the project and in the sidebar, select the Pipelines option and choose the Releases.
To add a new pipeline, select New and choose the "New release pipeline" option.
Select a required job for this step or just start with an "Empty job'
After adding the job, click on the Save button.
Now to add a task, select the Task option.
To add a task to an agent job, click on the highlighted icon.
Type LEAPWORK in the search bar and press enter.
Click on Add.
Now the task will be shown on the left panel and choose that task by selecting the checkbox.
- Enter LEAPWORK controller hostname or IP-address.
- Enter your LEAPWORK controller API port, by default it is 9001.
- Enter your API access key. You can find it in LEAPWORK Controller Settings section.
- Enter time delay (in seconds, by default 5 seconds). While a schedule is running extension will check the schedule state with a specified delay before the schedule is finished.
- Select how the extension should process "Done" status: “Success” or “Failed”.
- Enter the JUnit report file name. The default name is "report.xml".
- Enter schedule titles you want to run, each one must be entered from a new line.
- Enter schedule ids, each one must be entered from a new line. Using id is preferable because the schedule title can be changed.
After entering all the required inputs, click on the Save button.
Now navigate back to the pipeline and click on the "Create a Release" option.
Click on a Stage(Stage 1) and click on create to create a release.
Now to deploy the release, click on either Release or Stages.
Click on Deploy.
Now the release will be queued.
Create bugs from schedules
To run a collection of test cases in one or more environments (machines, devices, etc.) on a scheduled basis or by triggering test cases through the LEAPWORK REST API, you must define a schedule. In a schedule, you can select one or more test cases from a LEAPWORK project and pair them with one or more environments. Furthermore, you can set the duration and frequency of scheduled test case runs.
In a schedule, you can select Add action and define one or more actions that should be performed during a test case. One example of an action is to send an e-mail when a schedule has ended. Another is to call a specified web service if more than X number of test cases in the schedule has failed.
In this case, a PowerShell action is triggered when the schedule is finished. This action is used to call a local PowerShell script and pass the result of the schedule to the PowerShell script. An example of how to specify this action is shown in the following figure:
The command tells LEAPWORK to run a PowerShell script, c:\scripts\createTFSbugs.ps1.
The following parameters have been added:
- -result: This parameter is read by the PowerShell script and is the result of the entire schedule. To add the result data, right-click in the command field and select Insert token -> RESULTS-JSON.
Warning: The inserted token must be in between
to include the line breaks. This structure ensures that PowerShell transfers the JSON result format correctly.
- -rootPath: This parameter is read by the PowerShell script and is the file path to the folder containing the videos and screenshots recorded during the execution of the test cases.
The PowerShell script, CreateTFSbugs.ps1, will connect to the Azure DevOps Server REST API and create or update bugs based on the failed test cases in the schedule.
To use the PowerShell script, you need to specify a few settings about you Azure DevOps Server installation:
domain: If you are using Visual Studio Online (VSO), insert your visualstudio.com domain name, for example, https://LEAPWORK.visualstudio.com
pat: Personal Access Token (PAT). You can find more information about setting up a PAT here.
logfile: The path to the integration log file. All log entries are appended to this file
Trigger schedules from API and create bugs in Azure DevOps Server
Integrating LEAPWORK with any third-party system through the REST API can be done in a few simple steps. This enables you to easily set up the automatic creation or updating of bugs in Azure DevOps Server when test automation cases fail in LEAPWORK.
As part of the integration, we have a Windows Powershell script that runs a pre-defined LEAPWORK run list, polls for the results until they are available, and then loops through all failed cases and creates bugs in JIRA as appropriate using API's.
The following changes are required:-
- Generate an AccessKey from LEAPWORK Studio->Settings->Add Key.
- Replace the generated key with Accesskey value
- Get your schedule name from LEAPWORK Studio->Runlist->Expand Runlist and replace it in Line 66 with schedule name
#Run the LEAPWORK schedule "SmokeRunSchedule" and get the results.
$runResult = RunScheduleAndGetResults("SmokeRunSchedule")
- Create a Personal Access token in Azure DevOps and use it to authenticate into Azure DevOps and use the below link to generate a token - For more information about how to setup PATs and how to perform Azure DevOps REST calls, see the following links https://docs.microsoft.com/en-us/azure/devops/organizations/accounts/use-personal-access-tokens-to-authenticate?view=azure-devopshttps://docs.microsoft.com/en-us/rest/api/azure/devops/?view=azure-devops-rest-5.1&viewFallbackFrom=azure-devops
# Create an authorization header using a Personal Access Token.
$pat = "YOUR PERSONAL ACCESS TOKEN HERE"
- Enter the Azure DevOps Instance URL
$urlAlreadyExists = "https://YOURINSTANCE.visualstudio.com/DefaultCollection/YOURPROJECT/_apis/wit/wiql?api-version=1.0"
After running the PowerShell script, the schedule will be executed and for failed flows, a bug will be logged in Azure DevOps.
For any clarification, please contact our Priority Support.