A few months ago I started working on the Jackrabbit improvement issue JCR-325. Following a good practice, the first thing I did was create a test case for the missing functionality. However, this breaks another good practice of always passing 100% of the test suite. This and a few other know issue tests are currently causing the Jackrabbit test suite to fail even without any real problems, making it more difficult to check whether a recent change or an experimental patch breaks things.
To fix the situation I started wondering if there was a JUnit version of the TODO blocks in Perl’s Test::More. The problem is that JUnit can only report tests as successful or failing (or erroneous if they throw an exception), there is no way to easily mark test failures as TODOs. Googling around and asking the Jackrabbit mailing list produced some workarounds:
- Use a system property to determine whether to perform or skip the known issue test cases.
- Put the known issues tests in separate test case classes and exclude them from the test suite.
- Use a JUnit addon to ignore marked test cases as explained in an article that discusses this same issue.
- Use an alternative test framework like TestNG, that has this functionality built-in.
I didn’t want to start changing the entire test suite or even tweaking the build environment, so the last two options were out. I also wanted to make the setup easily configurable so a developer can selectively enable testing for a known issue, thus the first alternative of using a system property looks like the best solution. It seems that the Apache OJB project has reached the same solution.