On JUnit's random order of test method execution

This is a rant about JUnit, or more precisely, a rant about JUnit's inability to execute test methods in natural method order. (The order in which they appear in the java source file.)

Up until and including Java 6, when enumerating the methods of a java class, the JVM would yield them in natural order. However, when Java 7 came out, Oracle changed something in the way classes are loaded, and so this operation started yielding methods in a seemingly random order. (Hash order probably, which is effectively random.)

Apparently JUnit was executing methods in the order in which the JVM was yielding them, so as a result of upgrading to Java 7, everybody's tests started running in random order. This caused considerable ruffling of feathers all over the world. 

People asked the creators of JUnit to provide a solution to this, but nothing was being done for a long time, allegedly because if You Do Unit Testing Properly™, you should not need to run your tests in any particular order, since there should be no dependencies among them. So, apparently, according to the creators of JUnit, if you want your tests to run in a particular order, it must be because you have tests that depend on other tests.

When the creators of JUnit finally did something to address the issue, (it did not take them long, only, oh, until Java 8 came out,) their solution was completely half-baked: the default mode of operation was still random method order, but with the introduction of a special annotation one could coerce JUnit to run test methods either in alphabetic order, (which is useless,) or in some other weird, ill-defined, so-called "fixed" order, which is not alphabetic, nor is it the natural order, but according to the JUnit documentation it guarantees that the methods will be executed in the same order from test run to test run. (And is also completely useless.) 

So, apparently, the creators of JUnit were willing to do anything except the right thing.

Amendment 2020: So, I tried JUnit 5, and even though the entire framework is said to have been re-written from scratch, the exact same problems persist.

No comments:

Post a Comment