On JUnit's random order of test method execution

This is a rant about JUnit, or more precisely, a rant about JUnit's inability to execute test methods in natural method order. 

Definition: Natural method order is the order in which methods appear in the source file.

What is the problem?

(Useful pre-reading: About these papers)

Up until and including Java 6, when enumerating the methods of a java class, the JVM would yield them in natural order. However, when Java 7 came out, Oracle changed something in the internals of the JVM, and this operation started yielding methods in random order.

Apparently, JUnit was executing methods in the order in which the JVM was yielding them, so as a result of upgrading to Java 7, everybody's tests started running in random order. This caused considerable ruffling of feathers all over the world.

Now, the creators of the Java language are presumably running unit tests just like everyone else, so they probably noticed that their own tests started running in random order before releasing Java 7 to the world, but apparently they did not care.

Luckily, the methods are still being stored in natural order in the class file, they only get garbled as they are being loaded by the class loader, so you can still discover the natural method order if you are willing to get just a little bit messy with bytecode. 

However, that's too much work, and it is especially frustrating since the class loader is in a much better position to correct this problem, but it doesn't.  (The class loader messes up the method order probably because it stores them in a HashMap, which yields its contents in Hash order, which is essentially random. So, fixing the problem would probably have been as simple as using a LinkedHashMap instead of a HashMap.)

People asked the creators of JUnit to provide a solution, but nothing was being done for a long time, allegedly because if You Do Unit Testing Properly™, you should not need to run your tests in any particular order, since there should be no dependencies among them. So, the creators of JUnit are under the incredibly short-sighted impression that if you want your tests to run in a particular order, it must be because you have tests that depend on other tests.

When the creators of JUnit finally did something to address the issue, (it did not take them long, only, oh, until Java 8 came out,) their solution was completely half-baked: the default mode of operation was still random method order, but with the introduction of a special annotation one could coerce JUnit to run test methods either in alphabetic order, (which is nearly useless,) or in some other weird, ill-defined, so-called "fixed" order, which is not alphabetic, nor is it the natural order, but according to them it guarantees that the methods will be executed in the same order from test run to test run. (And is completely useless.) 

So, apparently, the creators of JUnit were willing to do anything except the right thing, and even though JUnit 5 is said to have been re-written from scratch, the exact same problem persists.

Why is this a problem?

Well, let me tell you why running tests in natural method order is important:

We tend to test fundamental features of our software before we test features that depend upon them, so if a fundamental feature fails, we want that to be the very first error that will be reported. (Note: it is the features under test that depend upon each other, not the tests themselves!)

The test of a feature that relies upon a more fundamental feature whose test has already failed might as well be skipped, because it can be expected to fail, but if it does run, reporting that failure before the failure of the more fundamental feature is an act of sabotage against the developer: it is sending us looking for problems in places where there are no problems to be found, and it is making it more difficult to locate the real problem, which usually lies in the test that failed first in the source file.

To give an example, if I am writing a test for my awesome collection class, I will presumably first write a test for the insertion function, and further down I will write a test for the removal function. If the insertion test fails, the removal test does not even need to run, but if it does run, it is completely counter-productive to be shown the results of the removal test before I am shown the results of the insertion test. If the insertion test fails, it is game over. As they say in the far west, there is no point beating a dead horse. How hard is this to understand?

Another very simple, very straightforward, and very important reason for wanting the test methods to be executed in natural order is because seeing the test method names listed in any other order is brainfuck.