Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
Azure DevOps Services | Azure DevOps Server | Azure DevOps Server 2022
Automated testing in Azure Test Plans connects your automated test code to test case work items. You can run tests on demand from test plans, track results alongside manual tests, and measure requirement-level quality. This article walks through the complete workflow from writing tests to analyzing results.
End-to-end automated testing workflow
Setting up automated testing involves the following steps:
- Create automated tests — Write tests in a supported framework and check them into source control.
- Set up a build pipeline — Create a pipeline that compiles your test project and produces test binaries.
- Associate tests with test cases — Link each automated test method to a test case work item for traceability.
- Run automated tests — Execute tests on demand from Test Plans or as part of CI/CD pipelines.
- Review results and track quality — Analyze pass/fail results, track requirement coverage, and identify trends.
Prerequisites
| Category | Requirement |
|---|---|
| Project access | Project member. |
| Access levels | - To view test-related work items or run test cases: At least Basic access. Users with Stakeholder access have no access to Test Plans. - To exercise the full range of test-related features: Basic + Test Plans access level or one of the following Visual Studio subscriptions: - Enterprise - Test Professional - MSDN Platforms |
| Permissions | Area Path permissions: - To add or modify test plans, test suites, test cases, or other test-based work item types: Edit work items in this node set to Allow. - To modify test plan properties such as build and test settings: Manage test plans set to Allow. - To create and delete test suites, add and remove test cases from test suites, or change test configurations associated with test suites: Manage test suites set to Allow. Project-level permissions: - To view, create, or delete test runs: View test runs, Create test runs, and Delete test runs set to Allow. - To manage test configurations or test environments: Manage test configurations and Manage test environments set to Allow. - To add new tags to test-based work items: Create tag definition set to Allow. - To permanently delete test-based work items: Permanently delete work items set to Allow. |
For more information, see Manual test access and permissions.
Additionally:
| Category | Requirements |
|---|---|
| Pipeline permissions | To run automated tests from test plans, you need release permissions (Create releases, Manage deployments, Edit release stage). For more information, see Release permissions. |
| Tools | Visual Studio 2017 or later (Enterprise or Professional) to associate tests from Visual Studio. Not required if you associate tests from the Azure DevOps web portal. |
Step 1: Create automated tests
Write automated tests by using any supported framework. Check the test project into Azure Repos or a connected GitHub repository.
Supported frameworks
| Framework | Associate in Visual Studio | Associate in Azure DevOps |
|---|---|---|
| MSTest v1/v2 | Yes | Yes |
| NUnit | Yes | Yes |
| xUnit | Yes | Yes |
| Selenium | Yes | Yes |
| Coded UI tests | Yes | Yes |
| Python (PyTest) | No | Yes |
| Java (Maven/Gradle) | No | Yes |
For guidance on writing UI tests that run reliably in CI/CD pipelines, see UI testing considerations. For a step-by-step Selenium example, see Perform UI tests with Selenium.
Step 2: Set up a build pipeline
Create a build pipeline that compiles your test project and produces artifacts containing the test binaries. Both Classic and YAML pipelines are supported.
- Create your first pipeline if you don't have one.
- Make sure your pipeline builds the test project and publishes the test binaries as build artifacts.
- Use the Visual Studio Test task or the Azure Test Plan task to run and report tests in your pipeline.
- If your pipeline runs tests, the results appear automatically on the Tests tab of the pipeline run.
To publish test results from non-Microsoft test runners, use the Publish Test Results task.
Step 3: Associate automated tests with test cases
When you associate a test method with a test case work item, you get these benefits:
- On-demand execution from Test Plans
- Requirements traceability — when test cases link to user stories, automated results show requirement-level quality
- Unified reporting — automated and manual test results appear together in the Test Run Hub
You can associate tests from Visual Studio or from the Azure DevOps web portal:
- Visual Studio — Open Test Explorer, select a test method, and choose Associate to Test Case. Supports .NET frameworks (MSTest, NUnit, xUnit, Selenium, Coded UI).
- Azure DevOps portal — Associate from pipeline test results or directly from a test case work item. Supports all frameworks including Python and Java.
For complete instructions, see Associate automated tests with test cases.
Important
You can associate a test method with multiple test cases, but each test case can only have one associated test method.
Step 4: Run automated tests
Run from test plans (on demand)
Trigger automated tests directly from Azure Test Plans without setting up scheduled builds. Select specific test cases and run them against a build and release pipeline configured in the test plan settings.
For step-by-step instructions, see Run automated tests from test plans.
Run from pipelines (CI/CD)
Run automated tests as part of your build or release pipeline. Tests execute automatically on every commit or deployment, and results appear on the pipeline's Tests tab.
- Run tests in parallel — Distribute tests across multiple agents to reduce execution time. See Run VSTest tests in parallel or Run any tests in parallel.
- Run only impacted tests — Use Test Impact Analysis to automatically select tests affected by code changes. See Test Impact Analysis.
Step 5: Review test results and track quality
Test results
- In Test Plans — View pass and fail results for automated and manual tests together in the Test Run Hub. Select any test run to see detailed results, error messages, and stack traces.
- In Pipelines — View test results on the Tests tab of any pipeline run. See Review test results.
Analytics and trends
- Test Analytics — Identify top failing tests, track pass rate trends, and analyze failure patterns across builds. See Test Analytics.
- Code coverage — Measure which parts of your code are exercised by tests. See Review code coverage results.
- Flaky test management — Detect and manage tests with non-deterministic results to prevent false build failures. See Manage flaky tests.
Requirements traceability
When you link test cases to requirements (user stories, PBIs), automated test results show requirement-level quality. Track which requirements pass, fail, or lack test coverage. See Requirements traceability.