Articles

JUnit tests for known issues

In Jackrabbit, Java on 2006-07-08 by Jukka Zitting

A few months ago I started working on the Jackrabbit improvement issue JCR-325. Following a good practice, the first thing I did was create a test case for the missing functionality. However, this breaks another good practice of always passing 100% of the test suite. This and a few other know issue tests are currently causing the Jackrabbit test suite to fail even without any real problems, making it more difficult to check whether a recent change or an experimental patch breaks things.

To fix the situation I started wondering if there was a JUnit version of the TODO blocks in Perl’s Test::More. The problem is that JUnit can only report tests as successful or failing (or erroneous if they throw an exception), there is no way to easily mark test failures as TODOs. Googling around and asking the Jackrabbit mailing list produced some workarounds:

  • Use a system property to determine whether to perform or skip the known issue test cases.
  • Put the known issues tests in separate test case classes and exclude them from the test suite.
  • Use a JUnit addon to ignore marked test cases as explained in an article that discusses this same issue.
  • Use an alternative test framework like TestNG, that has this functionality built-in.

I didn’t want to start changing the entire test suite or even tweaking the build environment, so the last two options were out. I also wanted to make the setup easily configurable so a developer can selectively enable testing for a known issue, thus the first alternative of using a system property looks like the best solution. It seems that the Apache OJB project has reached the same solution.

Advertisements

2 Responses to “JUnit tests for known issues”

  1. I really should do that when I’m doing things in Commons. Instead I tend to attach them to the JIRA issue until I figure out a fix. Previously I’d copied them to the side and not committed, but I lost too many that way.

    Reading your solutions; I’d be tempted to make JUnit test classes that end with Todo.java and not Test.java and have them ignored by the build. Then you’d need something custom to actually run them.

  2. Henri Yandell:
    > Reading your solutions; I’d be tempted to make JUnit test classes that end with Todo.java
    > and not Test.java and have them ignored by the build. Then you’d need something custom
    > to actually run them.

    Yes, that’s another solution. My requirement was to be able to selectively enable individual known issue tests directly from the Maven command line, so the system property approach suits my needs best. See my recent followup post on how I did it.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: