2018-04-09

GitHub project: mikenakis-testana

A command-line utility for running only those tests that actually need to run.


The mikenakis-testana logo, profile of a crash test dummy by Mike Nakis
Used under CC BY License.



License


This creative work is explicitly published under No License. This means that I remain the exclusive copyright holder of this creative work, and you may not do anything with it other than view its source code and admire it. More information here: michael.gr - Open Source but No License.

If you would like to do anything more with this creative work, please contact me.

What is mikenakis-testana? 


Testana is a utility for running JUnit-compatible tests in Maven-based Java projects, offering some amazing features, the most important of which being that every time you make some changes to your code and run Testana, it detects which tests need to run, and runs only those tests, saving you tons of time. 

Note: in this document the term project refers to the topmost (most-encompassing) organizational entity of your source code, while the term module refers to the smaller organizational entity that produces a single library or a single executable. This is IntelliJ IDEA terminology. In Eclipse the corresponding terms are workspace and project. In Visual Studio the corresponding terms are solution and project

What does mikenakis-testana do? 


You launch Testana each time you want to run any tests in your project.
  • In the general case you do not need to supply any arguments, you just have to run it on the root of your source folder tree, and it figures out the rest. So, you can just bind it to a key and have all your testing needs covered with the press of a button.
  • Testana figures out which tests need to run by:
    • Scanning your source tree for modules
    • Obtaining the output directories from the module description files (currently Maven is supported)
    • Checking the timestamps of the class files
    • Performing dependency analysis on the bytecode
    • Remembering the last successful run time of each test class. 
  • Testana then only runs the tests that need to run. This means no more waiting for all tests to run to completion; each time you run Testana, the only tests that run are those that actually need to run, if any.
  • Testana selects which tests to run from among all tests of all modules in your entire project, thus guaranteeing that any and all tests that need to run will run. This means that you do not have to guess which modules might need testing and which ones can safely be skipped, and you do not have to manually indicate those that do need testing. It does not hurt to consider all tests as candidates for running, because if it turns out that a test does not need to run, it will not run. 
  • Testana runs your test classes in order of dependency, meaning that classes that are most dependent upon will be tested first, and classes that depend upon those will be tested afterwards. This in turn means that in the event of test failures, the very first error message will usually be the most pertinent one, sparing you from the need to look any further down the logs. 
  • Testana runs the methods of a test class in their natural order, which is the order in which the methods appear in the source file. (Duh!) 
  • If you have a test class that inherits from another test class, Testana will run the test methods of the ancestor first

Why should I care about running only the tests that need to run? 


In workplaces with huge projects the usual situation is that tests take an unreasonably long time to run, so developers tend to take shortcuts in running them. One approach some developers follow is that they simply commit code without running any tests, relying on the continuous build to run the tests and notify them of any test failures. This has multiple disadvantages:
  • It causes repeated interruptions in the workflow, due to the slow turnaround of the continuous build, which is often several hours, sometimes overnight, and even in the fastest cases, always longer than a normal person's attention span. (That's by definition; if it was not, then there would be no problem with quickly running all tests locally each time before committing.) 
  • The failed tests require additional commits to fix, and each commit requires a meaningful commit message, which means that test failures often need to be registered and tracked as bugs, thus increasing the overall level of bureaucracy in the development process. 
  • The commit history becomes bloated with commits that were done in vain and that should never be checked out because they contain bugs that are fixed in later commits. 
  • Worst of all, untested commits that contain bugs are regularly being made to the repository. These bugs then stay there, while the continuous build takes its time doing its thing, eventually the tests run and they fail, the developers take notice, troubleshoot the test failure, come up with a theory as to what went wrong, come up with a fix, and commit the fix. This whole process takes quite a long time, during which other unsuspecting developers inevitably pull from the repository, thus receiving the bugs. Kind of like Continuous Infection
Testana solves all of the above problems by figuring out which tests need to run based on what has changed, and only running those tests. This cuts down the time it takes to run tests to a tiny fraction of what it is when blindly running all tests, which means that running the tests now becomes piece of cake and can usually be done real quick before committing, as it should.

Also, running tests real quick right after each pull from source control now becomes feasible, so a developer can avoid starting to work on source code on which the tests are failing.  (How often have you found yourself in a situation where you pull from source control, change something, run the tests, the tests fail, and you are wondering whether they fail due to changes you just made, or due to changes you pulled from the repository?)

Why should I care about selecting the tests to run from among all tests of all modules of the entire project? 


Another approach followed by some developers to save time is manually choosing a set of packages that they believe should be tested, and running only the tests of those packages, so as to manage to test at least something before committing.
  • One simple reason why this is problematic is that it requires cognitive effort to figure out which packages might need testing, and manual work to launch tests on each one of them individually; it is not as easy as pressing a single button that stands for "test whatever needs to be tested". 
  • A far bigger problem is that in manually selecting the tests to run, the developer is making assumptions about the dependencies of the code that they have modified and are about to commit. In complex systems, dependency graphs can be very difficult to grasp, and as the systems evolve, the dependencies keep changing. Unfortunately, unknown or not-fully-understood dependencies are a major source of bugs, and yet by hand-selecting what to test based on our assumptions about the dependencies, it is precisely the unknown and the not fully understood dependencies that are guaranteed to not be tested. This is a recipe for failure. 
Testana solves the above problems by always selecting the tests to run from among all tests of all modules of the entire project. It does not hurt to do that, because tests that do not need to run will not run anyway.

Why should I care about running test classes in order of dependency? 


Under JUnit, the order of execution of packages and classes within a package appears to be alphabetic. This is probably unintentional, and it is not configurable: As far as I know, JUnit does not offer any means of specifying the order in which packages should be tested, nor the order in which test classes within a package should be run, and it does not do anything in the direction of automatically figuring out by itself some order of execution that has any purpose or merit.

Alphabetic order of execution is not particularly useful. For example, in an alphabetic list of packages, `util` comes near the end, so it is usually tested last, and yet `util` tends to be a package that depends on no other packages, while most other packages depend on it, so if tests of other packages succeed, and yet tests of `util` fail, it can only be due to pure accident. It would be very nice to see `util` being tested first, so that if there is something wrong with it, then we know that we can stop testing: there is no point in testing packages that depend on a failing package.

Testana addresses this problem by executing test classes in order of dependency, which means that classes that do not depend on other classes will be tested first, and classes that depend upon them will be tested afterwards. This generally means that as soon as you see a test failure you can stop running the tests, because the most fundamental class with a defect has already been located. 

Why should I care about running test methods in natural order? 


By default, JUnit will run your test methods in random order, which is at best useless, and in my opinion actually outright treacherous. 

One reason for wanting the test methods to run in the order in which they appear in the source file is because we usually test fundamental features of our software before we test features that depend upon them. (Note: it is the features under test that depend upon each other, not the tests themselves that depend upon each other.) So if a fundamental feature fails, we want that to be the very first error that will be reported. Tests of features that rely upon a feature whose test has failed might as well be skipped, because they can be expected to all fail, and as a matter of fact, reporting these failures before the failure of the more fundamental feature (due to a messed up order of test method execution) is an act of sabotage against the developer: it is sending us looking for problems in places where there are no problems to be found, and it is making it more difficult to locate the real problem, which usually lies in the test that failed first in the source file.

To give an example, it is completely pointless to be told that my `search-for-item-in-collection` test failed, and only later to be told that my `insert-item-to-collection` test failed. If `insert-item-to-collection` fails, it is game over, there is no need to go any further, no need to try anything else with the collection, no point beating a dead horse. How hard is this to understand? 

Finally, another very simple, very straightforward, and very important reason for wanting the test methods to be executed in natural order is because seeing the test method names listed in any other order is brainfuck.

A related rant can be found here: michael.gr - On JUnit's random order of test method execution

Why should I care for running test methods of ancestors first? 


So, maybe you have never used inheritance in test classes, so this issue might be irrelevant to you, but I have, and I consider it very useful. I also consider `JUnit`'s behavior on this matter very annoying, because it does the exact opposite of what it should be doing. 

Inheritance in test classes can be very useful for achieving great code coverage while reducing the total amount of test code. So, for example, suppose you have a collection hierarchy to test: you have an `ArrayList` class and a `HashSet` class, and you also have their corresponding test classes, `ArrayListTest` and `HashSetTest`. Now, both `ArrayList` and `HashSet` inherit from `Collection`, which means that lots of tests are going to be identical between `ArrayListTest` and `HashSetTest`. One way to eliminate duplication is to have a `CollectionTest` abstract base class, which tests only `Collection` methods, and it does so without knowing what class is implementing them. Then, both `ArrayListTest` and `HashSetTest` can inherit from `CollectionTest` and provide specialized tests for functionality that is specific to `ArrayList` and `HashSet` respectively. Under such a scenario, when `ArrayListTest` or `HashSetTest` runs, we want the methods of `CollectionTest` to be executed first, because they are testing the fundamental (more general) functionality. 

To make the example more specific, `CollectionTest` is likely to add an item to the collection and then check whether the collection contains the item. If this test fails, then there is absolutely no point in proceeding with tests of `ArrayListTest` which will, for example, try adding multiple items to the collection and check to make sure that `IndexOf()` returns the right results. Again, `JUnit` handles this in exactly the opposite way one would expect: it executes the descendant (more specialized) methods first, and the ancestor (more general) methods last. 

Testana corrects it by executing ancestor methods first, descendant methods last. 

What are the limitations of Testana? 


  • Testana only works with Java. Support for more languages may be added in the future. 
  • Testana only understands Maven modules. Support for other module formats may be added in the future. 
  • Testana only understands the following JUnit annotations: `@Test`, `@Before`, `@After`, and `@Ignore`. Support for more JUnit annotations may be added in the future. 
  • Testana's dependency detection relies on strong typing. Dependencies that have been disavowed by being encoded in configuration files, (e.g. swagger files, spring configuration files,) denatured by encoding as strings, (stringly-typed,) obscured through hackery such as duck typing, or squandered by weak typing (euphemistically called dynamic typing) are not supported and there is no plan to support them. 
  • Testana only has a command-line interface. A GUI may be added in the future. 
  • Testana assumes that your local maven repository is under `~/.m2/repository`. It does not check `~/.m2/settings.xml` to see whether you have configured your local repository to reside elsewhere. This may be taken care of in the future. 
  • Testana currently does not consider resources when checking dependencies: a test may depend on a class which depends on a resource, and if the resource is modified, then the test should be re-run, but this does not currently happen. Adding support for this is quite high on my TODO list. 
  • Testana does not currently have a well defined strategy for handling dependency cycles. Nothing bad happens, but the order of test execution in those cases is kind of vague. This will be addressed in the future. 
  • Testana is still in Beta. There will be bugs. There will be cases not covered. There will be usage scenarios not considered. There will be change. (What else is new?) 

How can I see Testana in action? 


Without going into too much detail in this document, (because I want to have only one place to maintain, and that's the code,) here is roughly what you need to do: 
  • Check out the repository, and any repositories that it depends on. (Testana depends on a few other modules of mine which are also hosted on GitHub.) Testana and its dependencies are simple maven projects, so importing them into your IDE should be straightforward and problem-free. Once imported, Testana can simply be run from within the IDE, there is no need for `mvn install` or `mvn`-anything. 
  • Find the class that contains `public static void main` and run it, after making sure that the working directory has been configured to be the root directory of the source code of your project. 
  • If you run Testana without any arguments, its default behavior should be to do the right thing. 
  • On the first run, it should run all tests. 
  • On the second run, it should not run any tests, because nothing will have changed. 
  • If you touch one of your source files, recompile your project, and re-run Testana, you will notice that only tests that either directly or indirectly depend on the changed file will be run. 
  • If you run Testana with `--help` it will give you a rundown of all the options it supports. 
  • You might want to setup your IDE so that it builds your entire project before running Testana on it. That's because normally, prior to running Testana, your IDE will only build the modules that Testana depends on; however, Testana does not depend on any of the modules of your project, and yet Testana will discover them at runtime, and it will discover the test classes in them, and it will run them, so they all need to be up to date when Testana runs. The way to achieve this with IntelliJ IDEA is to edit the run/debug configuration of Testana and under `Before launch:` specify `Build Project` instead of the default, which is `Build`. 


No comments:

Post a Comment