I work in a company where we use MsTest as testing framework. TFS is our integration platform. For a long time we have been struggling with a problem that inconclusive tests are breaking the whole team build because mstest.exe returns exit code -1. My friend asked a question in StackOverflow where he summarized our requirements:
We are using
Assert.Inconclusivemainly for tests which are:
- Not implemented yet
- Somehow broken or incomplete = requires futher attention
- When test body is for any reason commented out
We are doing this because:
- Inconclusive test can have message
- We want to see such tests in test results on TFS
I did not find any simple way how to make TFS build understand that inconclusive tests should not break the build. Finally, I found a workaround. It is based on analysis of trx file after completing test phase of the build. Trx is the XML file containing all details about the test run. Among others, it contains a node <ResultSummary>:
<?xml version="1.0" encoding="UTF-8"?> <TestRun id="2811e37a-3c6e-4e9f-90a0-6cf0d6e7f44c" name="TestResults" runUser="MACHINE\user" xmlns="http://microsoft.com/schemas/VisualStudio/TeamTest/2010"> ... <ResultSummary outcome="Inconclusive"> <Counters total="487" executed="487" passed="485" error="0" failed="0" timeout="0" aborted="0" inconclusive="2" passedButRunAborted="0" notRunnable="0" notExecuted="0" disconnected="0" warning="0" completed="0" inProgress="0" pending="0" /> </ResultSummary> ... </TestRun>
If there are any inconclusive tests and no other tests failed then the attribute outcome will have value Inconclusive. Analysis of the trx file in the build script was implemented with MsBuld community tasks:
<Target Name="AnalyzeTestResultsFile"> <ItemGroup> <TrxFilePath Include="$(TestResultsRoot)\*.trx" /> </ItemGroup> <!-- Namespace should be specific for MsTest version --> <XmlRead Prefix="n" Namespace="http://microsoft.com/schemas/VisualStudio/TeamTest/2010" XPath="/n:TestRun/n:ResultSummary/@outcome" XmlFileName="@(TrxFilePath)"> <Output TaskParameter="Value" PropertyName="TestOutcome" /> </XmlRead> <Error Text="Test outcome was $(TestOutcome)." Condition="'$(TestOutcome)' == 'Error' or '$(TestOutcome)' == 'Failed' or '$(TestOutcome)' == 'Timeout' or '$(TestOutcome)' == 'Aborted'" /> </Target>
If you want to have such behavior in all your builds the the simplest way is to create a file in $(MsBuildExtensionPath) (usually C:\Program Files\MSBuild\):
<?xml version="1.0" encoding="utf-8" ?> <Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003"> <Import Project="$(MSBuildExtensionsPath)\Microsoft\VisualStudio\TeamBuild\Microsoft.TeamFoundation.Build.targets" Condition=" '$(TeamBuildVersion)' == '' "/> <Import Project="$(MSBuildExtensionsPath)\MSBuildCommunityTasks\MSBuild.Community.Tasks.Targets" Condition=" '$(MSBuildCommunityTasksLib)' == '' " /> <PropertyGroup> <TestConfigurationDependsOn> ResetTestWorkspace; $(TestConfigurationDependsOn); AnalyzeTestResultsFile; </TestConfigurationDependsOn> </PropertyGroup> <Target Name="ResetTestWorkspace"> <RemoveDir Directories="$(TestResultsRoot)" /> <PropertyGroup> <StopOnTestFailure>false</StopOnTestFailure> </PropertyGroup> </Target> <Target Name="AnalyzeTestResultsFile"> <ItemGroup> <TrxFilePath Include="$(TestResultsRoot)\*.trx" /> </ItemGroup> <!-- Namespace should be specific for MsTest version --> <XmlRead Prefix="n" Namespace="http://microsoft.com/schemas/VisualStudio/TeamTest/2010" XPath="/n:TestRun/n:ResultSummary/@outcome" XmlFileName="@(TrxFilePath)"> <Output TaskParameter="Value" PropertyName="TestOutcome" /> </XmlRead> <Error Text="Test outcome was $(TestOutcome)." Condition="'$(TestOutcome)' == 'Error' or '$(TestOutcome)' == 'Failed' or '$(TestOutcome)' == 'Timeout' or '$(TestOutcome)' == 'Aborted'" /> </Target> </Project>
I redefined property $(TestConfigurationDependsOn) so that target ResetTestWorkspace is automatically called right before test run is invoked. Target AnalyzeTestResultsFile is called after test run when trx file is already available.
With this TFS template all your builds will not fail when you have some tests inconclusive.
Hi, thanks for taking the time to write up this tutorial, I’ve been searching everywhere for something like this!
I’ve tried to follow your example so that for all builds, ‘Inconclusive’ tests will not fail the build.
I have the Community Tasks installed, but I don’t understand:
1) What I need to call the file ? IgnoreInconclusiveTests.proj ?
2) How do I tell MSBuild to use the aforementioned file ?
Any help you can provide would be greatly appreciated!!
You can save the template above for example to c:\Program Files (x86)\msbuild\YourCompany\teamBuild.targets.
Then, in your TFS build for the project, change
$(MSBuildExtensionsPath)\Microsoft\VisualStudio\TeamBuild\Microsoft.TeamFoundation.Build.targets
with $(MSBuildExtensionsPath)\YourCompany\teambuild.targets so that zou project build does not use default TFS template but uses extended template instead.
On TFS 2010, where do you make the change to use $(MSBuildExtensionsPath)\YourCompany\teambuild.targets ?
How to do with TFS2010 ?
Unfortunately we still do not use TFS 2010 in my company. As soon as we start using it I will update this post with the steps needed for TFS 2010.
For TFS 2010 try looking at creating a TestCategory for Inconclusive tests and then in your TFS build definition under Test Assembly, set your Category Filter to filter out your Test Category. You’ll still want to add Assert.Inconclusive() so that when you run a local build you can see how many tests are incomplete but the build server will skip over them. Unfortunately this means that your build output will not show the amount of inconclusive tests…