<< Back to previous view |
[QB-2315] Add support for tests that use the Test Anything Protocol
|
|
Status: | Resolved |
Project: | QuickBuild |
Component/s: | None |
Affects Version/s: | None |
Fix Version/s: | 6.0.9 |
Type: | New Feature | Priority: | Major |
Reporter: | John Szakmeister | Assigned To: | Unassigned |
Resolution: | Fixed | Votes: | 0 |
Remaining Estimate: | Unknown | Time Spent: | Unknown |
Original Estimate: | Unknown |
Description |
This has come up a few times at our place, and we've just worked around it some other way, but I imagine in would be useful to others too. The Test Anything Protocol (TAP) is used by a fair number of test engines. It'd be nice to have this supported from a test report perspective. You can find out more starting at the Wikipedia page for it: http://en.wikipedia.org/wiki/Test_Anything_Protocol
One thing to note: the number of tests (for example, 1..87) can either appear before or after the test run. Most emit it first, but there are some that do it the other way. Here's an example from Git's test suite *** t0000-basic.sh *** ok 1 - .git/objects should be empty after git init in an empty repo ok 2 - .git/objects should have 3 subdirectories ok 3 - success is reported like this ok 4 - pretend we have a fully passing test suite ok 5 - pretend we have a partially passing test suite ok 6 - pretend we have a known breakage ok 7 - pretend we have fixed a known breakage ok 8 - pretend we have fixed one of two known breakages (run in sub test-lib) ok 9 - pretend we have a pass, fail, and known breakage ok 10 - pretend we have a mix of all possible results ok 11 - test --verbose ok 12 - test --verbose-only ok 13 - GIT_SKIP_TESTS ok 14 - GIT_SKIP_TESTS several tests ok 15 - GIT_SKIP_TESTS sh pattern ok 16 - --run basic ok 17 - --run with a range ok 18 - --run with two ranges ok 19 - --run with a left open range ok 20 - --run with a right open range ok 21 - --run with basic negation ok 22 - --run with two negations ok 23 - --run a range and negation ok 24 - --run range negation ok 25 - --run include, exclude and include ok 26 - --run include, exclude and include, comma separated ok 27 - --run exclude and include ok 28 - --run empty selectors ok 29 - --run invalid range start ok 30 - --run invalid range end ok 31 - --run invalid selector ok 32 - test runs if prerequisite is satisfied ok 33 # skip unmet prerequisite causes test to be skipped (missing DONTHAVEIT) ok 34 - test runs if prerequisites are satisfied ok 35 # skip unmet prerequisites causes test to be skipped (missing DONTHAVEIT of HAVEIT,DONTHAVEIT) ok 36 # skip unmet prerequisites causes test to be skipped (missing DONTHAVEIT of DONTHAVEIT,HAVEIT) ok 37 - test runs if lazy prereq is satisfied ok 38 # skip missing lazy prereqs skip tests (missing !LAZY_TRUE) ok 39 - negative lazy prereqs checked ok 40 # skip missing negative lazy prereqs will skip (missing LAZY_FALSE) ok 41 - tests clean up after themselves ok 42 - tests clean up even on failures ok 43 - git update-index without --add should fail adding ok 44 - git update-index with --add should succeed ok 45 - writing tree out with git write-tree ok 46 - validate object ID of a known tree ok 47 - git update-index without --remove should fail removing ok 48 - git update-index with --remove should be able to remove ok 49 - git write-tree should be able to write an empty tree ok 50 - validate object ID of a known tree ok 51 - adding various types of objects with git update-index --add ok 52 - showing stage with git ls-files --stage ok 53 - validate git ls-files output for a known tree ok 54 - writing tree out with git write-tree ok 55 - validate object ID for a known tree ok 56 - showing tree with git ls-tree ok 57 - git ls-tree output for a known tree ok 58 - showing tree with git ls-tree -r ok 59 - git ls-tree -r output for a known tree ok 60 - showing tree with git ls-tree -r -t ok 61 - git ls-tree -r output for a known tree ok 62 - writing partial tree out with git write-tree --prefix ok 63 - validate object ID for a known tree ok 64 - writing partial tree out with git write-tree --prefix ok 65 - validate object ID for a known tree ok 66 - put invalid objects into the index ok 67 - writing this tree without --missing-ok ok 68 - writing this tree with --missing-ok ok 69 - git read-tree followed by write-tree should be idempotent ok 70 - validate git diff-files output for a know cache/work tree state ok 71 - git update-index --refresh should succeed ok 72 - no diff after checkout and git update-index --refresh ok 73 - git commit-tree records the correct tree in a commit ok 74 - git commit-tree records the correct parent in a commit ok 75 - git commit-tree omits duplicated parent in a commit ok 76 - update-index D/F conflict ok 77 - very long name in the index handled sanely # passed all 77 test(s) 1..77 A run containing a failed test might look like: *** t0000-basic.sh *** ok 1 - .git/objects should be empty after git init in an empty repo ok 2 - .git/objects should have 3 subdirectories not ok 3 - success is reported like this # # fail # ok 4 - pretend we have a fully passing test suite ok 5 - pretend we have a partially passing test suite ok 6 - pretend we have a known breakage ok 7 - pretend we have fixed a known breakage ok 8 - pretend we have fixed one of two known breakages (run in sub test-lib) ok 9 - pretend we have a pass, fail, and known breakage ok 10 - pretend we have a mix of all possible results ok 11 - test --verbose ok 12 - test --verbose-only ok 13 - GIT_SKIP_TESTS ok 14 - GIT_SKIP_TESTS several tests ok 15 - GIT_SKIP_TESTS sh pattern ok 16 - --run basic ok 17 - --run with a range ok 18 - --run with two ranges ok 19 - --run with a left open range ok 20 - --run with a right open range ok 21 - --run with basic negation ok 22 - --run with two negations ok 23 - --run a range and negation ok 24 - --run range negation ok 25 - --run include, exclude and include ok 26 - --run include, exclude and include, comma separated ok 27 - --run exclude and include ok 28 - --run empty selectors ok 29 - --run invalid range start ok 30 - --run invalid range end ok 31 - --run invalid selector ok 32 - test runs if prerequisite is satisfied ok 33 # skip unmet prerequisite causes test to be skipped (missing DONTHAVEIT) ok 34 - test runs if prerequisites are satisfied ok 35 # skip unmet prerequisites causes test to be skipped (missing DONTHAVEIT of HAVEIT,DONTHAVEIT) ok 36 # skip unmet prerequisites causes test to be skipped (missing DONTHAVEIT of DONTHAVEIT,HAVEIT) ok 37 - test runs if lazy prereq is satisfied ok 38 # skip missing lazy prereqs skip tests (missing !LAZY_TRUE) ok 39 - negative lazy prereqs checked ok 40 # skip missing negative lazy prereqs will skip (missing LAZY_FALSE) ok 41 - tests clean up after themselves ok 42 - tests clean up even on failures ok 43 - git update-index without --add should fail adding ok 44 - git update-index with --add should succeed ok 45 - writing tree out with git write-tree ok 46 - validate object ID of a known tree ok 47 - git update-index without --remove should fail removing ok 48 - git update-index with --remove should be able to remove ok 49 - git write-tree should be able to write an empty tree ok 50 - validate object ID of a known tree ok 51 - adding various types of objects with git update-index --add ok 52 - showing stage with git ls-files --stage ok 53 - validate git ls-files output for a known tree ok 54 - writing tree out with git write-tree ok 55 - validate object ID for a known tree ok 56 - showing tree with git ls-tree ok 57 - git ls-tree output for a known tree ok 58 - showing tree with git ls-tree -r ok 59 - git ls-tree -r output for a known tree ok 60 - showing tree with git ls-tree -r -t ok 61 - git ls-tree -r output for a known tree ok 62 - writing partial tree out with git write-tree --prefix ok 63 - validate object ID for a known tree ok 64 - writing partial tree out with git write-tree --prefix ok 65 - validate object ID for a known tree ok 66 - put invalid objects into the index ok 67 - writing this tree without --missing-ok ok 68 - writing this tree with --missing-ok ok 69 - git read-tree followed by write-tree should be idempotent ok 70 - validate git diff-files output for a know cache/work tree state ok 71 - git update-index --refresh should succeed ok 72 - no diff after checkout and git update-index --refresh ok 73 - git commit-tree records the correct tree in a commit ok 74 - git commit-tree records the correct parent in a commit ok 75 - git commit-tree omits duplicated parent in a commit ok 76 - update-index D/F conflict ok 77 - very long name in the index handled sanely # failed 1 among 77 test(s) 1..77 You can see that the diagnostic usually comes after the failed test. The numbers at the end (1..77) represents the number of tests in the suite and could preceed the run. Multiple suites may be executed as part of a single test run. IIRC, the "failed 1 among 77 test(s)" is not required, only the "ok" and "not ok" for each test. The standards for the test anything protocol can be found here: https://testanything.org/ |