Posts Tagged ‘TFS’

Inconclusive Tests in TFS Build Should Not Break the Build

Tips | Posted by Petr Kozelek
Feb 27 2011

I work in a company where we use MsTest as testing framework. TFS is our integration platform. For a long time we have been struggling with a problem that inconclusive tests are breaking the whole team build because mstest.exe returns exit code -1. My friend asked a question in StackOverflow where he summarized our requirements:

We are using Assert.Inconclusive mainly for tests which are:

  • Not implemented yet
  • Somehow broken or incomplete = requires futher attention
  • When test body is for any reason commented out

We are doing this because:

  • Inconclusive test can have message
  • We want to see such tests in test results on TFS

I did not find any simple way how to make TFS build understand that inconclusive tests should not break the build. Finally,  I found a workaround.  It is based on analysis of trx file after completing test phase of the build. Trx is the XML file containing all details about the test run. Among others, it contains a node <ResultSummary>:

<?xml version="1.0" encoding="UTF-8"?>
<TestRun id="2811e37a-3c6e-4e9f-90a0-6cf0d6e7f44c"
 name="TestResults" runUser="MACHINE\user"
 xmlns="http://microsoft.com/schemas/VisualStudio/TeamTest/2010">
 ...
 <ResultSummary outcome="Inconclusive">
   <Counters total="487" executed="487" passed="485"
     error="0" failed="0" timeout="0" aborted="0"
     inconclusive="2" passedButRunAborted="0"
     notRunnable="0" notExecuted="0" disconnected="0"
     warning="0" completed="0" inProgress="0" pending="0" />
 </ResultSummary>
 ...
</TestRun>

If there are any inconclusive tests and no other tests failed then the attribute outcome will have value Inconclusive. Analysis of the trx file in the build script was implemented with MsBuld community tasks:

<Target Name="AnalyzeTestResultsFile">
 <ItemGroup>
   <TrxFilePath Include="$(TestResultsRoot)\*.trx" />
 </ItemGroup>
 <!-- Namespace should be specific for MsTest version -->
 <XmlRead Prefix="n"
   Namespace="http://microsoft.com/schemas/VisualStudio/TeamTest/2010"
   XPath="/n:TestRun/n:ResultSummary/@outcome"
   XmlFileName="@(TrxFilePath)">
     <Output TaskParameter="Value" PropertyName="TestOutcome" />
   </XmlRead>
   <Error Text="Test outcome was $(TestOutcome)." Condition="'$(TestOutcome)' == 'Error' or '$(TestOutcome)' == 'Failed' or '$(TestOutcome)' == 'Timeout' or '$(TestOutcome)' == 'Aborted'" />
 </Target>

If you want to have such behavior  in all your builds the the simplest way is to create a file in $(MsBuildExtensionPath) (usually C:\Program Files\MSBuild\):

<?xml version="1.0" encoding="utf-8" ?>
<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
 <Import Project="$(MSBuildExtensionsPath)\Microsoft\VisualStudio\TeamBuild\Microsoft.TeamFoundation.Build.targets"
   Condition=" '$(TeamBuildVersion)' == '' "/>
 <Import Project="$(MSBuildExtensionsPath)\MSBuildCommunityTasks\MSBuild.Community.Tasks.Targets"
   Condition=" '$(MSBuildCommunityTasksLib)' == '' " />
 
 <PropertyGroup>
   <TestConfigurationDependsOn>
     ResetTestWorkspace;
     $(TestConfigurationDependsOn);
     AnalyzeTestResultsFile;
   </TestConfigurationDependsOn>
 </PropertyGroup>
 
 <Target Name="ResetTestWorkspace">
   <RemoveDir Directories="$(TestResultsRoot)" />    
   <PropertyGroup>
     <StopOnTestFailure>false</StopOnTestFailure>
   </PropertyGroup>
 </Target>
 
 <Target Name="AnalyzeTestResultsFile">
   <ItemGroup>
     <TrxFilePath Include="$(TestResultsRoot)\*.trx" />
   </ItemGroup>
   <!-- Namespace should be specific for MsTest version -->
   <XmlRead Prefix="n"
     Namespace="http://microsoft.com/schemas/VisualStudio/TeamTest/2010"
     XPath="/n:TestRun/n:ResultSummary/@outcome"
     XmlFileName="@(TrxFilePath)">
     <Output TaskParameter="Value" PropertyName="TestOutcome" />
   </XmlRead>
   <Error Text="Test outcome was $(TestOutcome)." Condition="'$(TestOutcome)' == 'Error' or '$(TestOutcome)' == 'Failed' or '$(TestOutcome)' == 'Timeout' or '$(TestOutcome)' == 'Aborted'" />
 </Target>
</Project>

I redefined property $(TestConfigurationDependsOn) so that target ResetTestWorkspace is automatically called right before test run is invoked. Target AnalyzeTestResultsFile is called after test run when trx file is already available.

With this TFS template all your builds will not fail when you have some tests inconclusive.

Tags: , ,
6 Comments »