John Szakmeister [20/Feb/15 12:02 PM] I attached this screenshot as a file because the java plugin is having issues running in browsers with updated java plugins (complains about an invalid or expired certificate).

John Szakmeister [20/Feb/15 12:08 PM] Sample unittest file with name attributes set.

Steve Luo [21/Feb/15 05:08 AM] First, we have a plugin named xunit, the purpose of this plugin is to help the development for other xUnit alike test reports, just like what you have mentioned, cppunit, nose, busted, etc. there are lots of such kind of reports with slight difference, use JUnit plugin to publish them is not a good idea. So, we encourage you develop your own plugin with the help from xunit plugin by writing about 200 lines of code, below page is the reference on how to do this: http://wiki.pmease.com/display/QB60/Creating+a+Test+Report+Plugin And even more, you can look through the source code of built-in Boost Test plugin. Second, about test suite name, JUnit plugin aggregates the tests numbers from testcase based on there testsuite name, but in your report, test suite name calculated from classname attribute in testcase element is different from the testsuite name read from testsuite element. For example: <testsuite tests="200" errors="0" failures="0" skip="0" time="0.60" name="unittest-linux-64-debug">     <testcase time="0.04" name="path function path_full_dir_name returns the absolute directory name of a given relative one" classname="./test/unit/path_spec.lua:50"/> here, testsuite name for testsuite element is unittest-linux-64-debug, package name is empty, while in your testcase element, package name is ./test/unit/path_spec and test suite name is lua:50. I don't know how this report generated, but to me it is very confused, as classname in testcase doesn't match testsuite name attribute in testsuite element. So, I still think use different plugin to parse different report is more reasonable and stable instead of using JUnit/xUnit plugin to parse all of them.

John Szakmeister [21/Feb/15 09:10 AM] > First, we have a plugin named xunit, the purpose of this plugin is to help the development for other xUnit alike test reports, just like what you have mentioned, cppunit, nose, busted, etc. there are lots of such kind of reports with slight difference, use JUnit plugin to publish them is not a good idea. I have no idea how I was supposed to discover this on my own--I looked all over the place. :-( > I don't know how this report generated, but to me it is very confused, as classname in testcase doesn't match testsuite name attribute in testsuite element. It was generated by busted--a test runner for Lua. Neovim uses Lua and busted to drive all of its testing, and it's these sort of details that many test runners have (because it doesn't match exactly). Additionally, I don't see anything, anywhere saying that they have to match. > So, I still think use different plugin to parse different report is more reasonable and stable instead of using JUnit/xUnit plugin to parse all of them. A couple things here. First, I *love* QuickBuild, but I have a real hard time selling it to others. The first reason is it's not free--and I don't think it needs to be. It's an excellent tool at a reasonable price. But the second reason is because it doesn't support things that we, as in my customers and their organizations, need out-of-the-box. Writing plugins is painful for the circles I run in. I can do it, but most of the others I work with cannot. We don't run in Java circles. Most of what we do is low-level C and assembly language, with the occasional dip into Python. Out of the box, QuickBuild works with exactly *zero* of the test engines out there. It's a really hard sell to say "it's not free, and you'll need to write plugins to parse reports that are generated from common open source tools". Personally, I think it would be nice to have JUnit do this, and select something from a dropdown saying it was produced by cppunit, busted, etc. and still have the path open to writing your own reporter processor available. Hopefully, you can see the difficulty I face. I'd like to avoid using Jenkins--it's the wrong tool for the job, IMHO. But spending hours trying to customize QB to get a single report is more than many will bear. I've seen it first hand. :-( I hope all of that makes sense, and I hoping I'm not coming across as overbearing. I just want you guys to understand why I think it's a problem you don't support more out-of-the-box.

Steve Luo [21/Feb/15 03:24 PM] JUnit plugin is not a common plugin to parse other xUnit style plugin, that is why we don't mention the testsuite names have to match. It is expected to parse JUnit reports. QuickBuild is not free for commercial organisations and products, but QuickBuild is free for all open source projects. For your project, Neovim, we can provide free open-source license to you. We can't support everything out-of-the-box (I think no product can), that is why we use plugin system in QuickBuild so that people can develop their own plugin easily when they find QuickBuild cannot meet their requirements, actually, most of customers do have their own plugins. I can understand you, so I have opened a new issue QB-2337 for supporting Busted test reports. You can attach some Busted generated reports there to help us developing this plugin.

Steve Luo [10/Mar/15 03:22 AM] As we discussed before, please publish Lua Busted report instead of JUnit report. See QB-2337.