[#QB-2439] QuickBuild seems to ignore the fact, that the two test cases, though named identically, are in different test classes.

QuickBuild

QuickBuild seems to ignore the fact, that the two test cases, though named identically, are in different test classes.

Created: 13/May/15 12:47 PM Updated: 03/Apr/16 07:17 AM

Component/s:

None

Affects Version/s:

6.0.16

Fix Version/s:

None

Original Estimate:	Unknown	Remaining Estimate:	Unknown	Time Spent:	Unknown
Environment:	Windows 2008R2 Server

Description

« Hide

In the supplied example project "com.rola.maven.test", there are two test classes "TestOne" and "TestTwo", each with a test case named "testHelloWorld". Both test classes are referenced in the test suite "AllTests".

Structure:
- AllTests
  - TestOne
    - testHelloWorld
  - TestTwo
    - testHelloWorld

When built with maven, only the test suite "AllTests.java" in run by the maven-surefire-plugin, resulting in a single XML file "target/surefire-reports/TEST-com.rola.maven.test.AllTests.xml".

In the surefire report created by maven or the JUnit Report created by ANT , the structure is recognized correctly, i.e. one test suite with two test cases, both named "testHelloWorld" from two different test classes, one succeeds, one fails.
In the QuickBuild JUnit Report, there is only one test case "testHelloWorld" displayed in the test suite. QuickBuild seems to ignore the fact, that the two test cases, though named identically, are in different test classes.

I am sending you an e-mail with the sample project and the associated backup QB

All

Comments

Work Log

Change History

Sort Order:

[ Permlink | « Hide ]

Steve Luo [14/May/15 03:46 AM]

Please don't run all of your test by using AllTest.java. In this way, the testsuite name in the output xml report is all changed to com.rola.maven.test.AllTest, QuickBuild will use this name as the suite name for two test cases.

Actually, you needn't use AllTest.java to aggregate the tests, QuickBuild will aggregate the tests automatically. So, just remove the configuration section below:

<execution>
            <id>com.rola.maven.test</id>
            <phase>integration-test</phase>
            <goals>
              <goal>test</goal>
            </goals>

           

</execution>

[ Permlink | « Hide ]

Georg Laschet [19/May/15 12:46 PM]

Since very recently we use both reports, the QB integrated and the surefire report.
We then noticed that the number of executed tests is different.
We cannot change the AllTests.java because we use hundreds of these suite classes to aggregate our tests in our framework. We don't want and cannot use the automatic test case scanning of maven surefire for several reasons.
These suite classes are a common JUnit-Feature both the ant JUnit report and the surefire report can handle the AllTest classes as we use them without problem.

[ Permlink | « Hide ]

Steve Luo [20/May/15 03:58 PM]

In QuickBuild, although you see just 1 test, you can find there is also a small label showing (2 runs, 1/1), means this test run 2 times, one passed and one failed. This is because we treat testHello as the test in suite com.rola.maven.test.AllTests, although the two testHello are in two different classes, they are in one suite, so we aggregate them together, that is why you see 2 runs.

We also compared the generated surefire html report, and don't find too much difference. Both QuickBuild and surefire use AllTests as the suite name and both ignore the fact that the tests are in two different test classes. You can find that there is no real package name and class name for two testHello in surefire report.

Actually, in this case, it is very hard to display like what QuickBuild current does as the test suite here is a super set of all other general test suite (test class).

[ Permlink | « Hide ]

Steve Luo [03/Apr/16 07:17 AM]

Need more information. Comment on this issue if you still want to discuss.