michael.gr: Assertions and Testing

So, since we do software testing, we should quit placing assert statements in production code, right? Let me count the ways in which this is wrong:

(TL;DR: skip to the paragraph containing a red sentence and read only that.)

1. Assertions are optional.

Each programming language has its own mechanism for enabling or disabling assertions. In languages like C++ and C# there is a distinction between a release build and a debug build, and assertions are generally only enabled in the debug build. Java has a simpler mechanism: there is only one build, but assertions do not execute unless the -enableassertions (-ea for short) option is specified in the command line which started the virtual machine. Therefore, if someone absolutely cannot stand the idea that assertions may be executing in a production environment, they can simply refrain from supplying the -ea option; problem solved.

(Useful pre-reading: About these papers)

The mere fact that assertions are optional and not even enabled by default should be enough to quench any objections to their use. Now, in order to convince people to start actively using assertions instead of merely not minding if others do, I need to explain why assertions are awesome. This is what the rest of this document sets out to do.

2. Assertions check things that testing cannot (and should not.)

Testing treats (or should be treating) the production code as a black box, ensuring that given specific input, it produces expected results. Assertions, on the other hand, have a white box view of the code, (of course, since they live in it,) so they perform internal checks to make sure that everything is working as expected under the hood. Therefore, the domain of assertions is different from the domain of software testing, so there is a clear need for both.

If there is any uncertainty as to whether software testing should be taking a black box or a white box approach, let me briefly open up a parenthesis to clarify this one:

When tests are tied to implementation details of the production code, situations arise where the production code gets refactored or bugs are fixed in it, and as a result the tests break and have to be modified in order to continue passing. I would call this The Fragile Test Problem. To avoid this, tests should be written having in mind nothing but the operational requirements of the software system, so that they only need to be revised in the event of a change in the requirements. (Also see footnote 1.)
Testing against implementation details of the production code renders the tests non-reusable:

It should be possible to completely rewrite a piece of production code and then reuse the old tests to make sure that the new code works exactly as the old one did.
It should be possible to write a test once and have it test multiple different implementations of a system, created by independently working development teams taking different approaches to solving the same problem.

Just as users tend to test software in ways that the developer never thought of, (the well known "works for me but always breaks in the hands of the user" paradox,) software tests written by developers who maintain an agnostic stance about the inner workings of the production code are likely to test for things that were never considered by those who wrote the production code.
Yes, of course, black box testing cannot claim that it leaves nothing to chance, and that's precisely why you need assertions!

Close parenthesis.

So, each assertion can be seen as a little software test embedded within the software.

Various techniques that are applicable to software testing are also applicable to assertions. An example of such a technique is the "do not fix it unless there is a test for it" advice. When a malfunction is observed in production, which has obviously passed all existing tests, instead of theorizing as to what went wrong and implementing a fix according to the theory, we add a theory-agnostic test against that malfunction, and we observe it failing. Then, we fix the malfunction according to our theory, and we observe the test passing. If the test still does not pass, then our theory was wrong.

The exact same approach is valid with assertions, with the added benefit that the assertion is a one-liner instead of an entire test. The savings here can be huge, both in terms of work to be done by the programmer, and in terms of total tests execution time, because a test may have to do a lot of work to set up the right conditions for the malfunction to be observed.

3. Assertions are an excellent documentation tool.

Unfortunately there is a very prevalent bad habit among software engineers worldwide, the habit of documenting assumptions within comments. This is very bad because:

Every time the code gets revised, amended or refactored, someone must remember to also update the comments, but this does not always happen. As a result, comments tend to become out of date as the code evolves, their accuracy and relevance eventually deteriorating so much that they come in conflict with what the code actually does.
No comment ever was, or will ever be, as precise and unambiguous as a piece of code stipulating the same thing.
Even the most precisely and unambiguously expressed comment is, by its very nature of being a comment, not enforceable in any way.

So, if you require a certain condition to be met, or if you have a certain assumption which you believe to be true, put your code where your mouth is and back up your claim with an assertion statement. Since assertions are optional, there is no performance penalty, and even if the code is not even once run with assertions enabled, the assertion is still better documentation than a comment, because at the very least, it compiles.

4. Assertions can catch errors in the testing code.

Sometimes the person coding or maintaining the test code and the person coding or maintaining the production code might have a different understanding of what the operational requirements actually mean. Assertions within the production code help catch any such discrepancies at the earliest point possible, and they show that the production code is the way it is on purpose, and not by accident.

5. Assertions can be more pertinent than testing.

Sometimes the operational requirements are vague on issues on which the software design needs to make specific decisions. If the preferred way of solving a certain problem involves a division by something, then obviously, that something must not be zero, but if the operational requirements say nothing about zero, then the tests might not test for zero, and in any case they cannot be expected to test for zero. The implementation, however, knowing its own limitations, should assert against a zero. Ideally, such an issue of vagueness in the specification would be submitted back to the people responsible for it, and it should receive a definitive answer in the operational requirements document, which should then be translated into an additional test, but these things do not always happen in the real world.

6. Assertions pinpoint errors that testing only broadly hints at.

Consider this scenario: you have a TimeTable object which contains WorkShift objects. The shifts are stored in an array which is sorted by start time, and binary search is used to answer queries such as which employee is working at a certain time. Now, suppose that you have forgotten to sort the array after an insertion, so the binary search fails.

All that the failed test will tell you is that it scheduled John to work from 10:00 to 11:00, but a query for who works at 10:30 did not yield anyone. This is not very useful; the bug could be anywhere.

Proper use of assertions mandates that at the very least, immediately prior to performing a binary search on your array, you should ensure that it meets the requirement for it to be searchable via binary search, that is, to be sorted. So, voila! the assertion immediately discovers the nature of the bug.

Even better, at the end of each operation on your TimeTable object you can assert that the operation is leaving the object in a valid state, which includes the requirement that this array must be at all times sorted. Thus, you will have an assertion failure at the end of the method which inserted an item to the array but forgot to sort it, pinpointing the bug with great accuracy.

Now imagine the same happening in an immensely more complicated software system, where a maintenance programmer attempts to make a few small changes without comprehending exactly how the entire system works, and as a result the tests of this system start to fail without any indication as to where the problem might be. Sure, those few altered lines of code broke it, but how? What is wrong? Would it not be nice if the system could tell us what is wrong with it? Well, assertions help you achieve precisely that: software systems that can very often tell what is wrong with themselves.

7. Assertions reduce program complexity.

The time complexity and computational complexity of algorithms are subjects which have received extensive study, but most of the code being written on a daily basis all over the planet is not algorithms in an academic sense, so these notions are inapplicable to it. What is pertinent to most code that we regularly write is state complexity, which is a subject that has not received much study yet. I hope that an analytical state complexity algebra will be invented one day, allowing us to accurately calculate the state complexity of any given piece of code, but until then, nothing prevents us from theorizing about it in coarse terms. I hold it as self evident that if you add a variable to a system, the total state complexity of the system is compounded by something akin to the number of bits of that variable multiplied by the total number of statements throughout the system that make use of the value of that variable. If you add an if statement, total system state complexity is compounded by the number of bits of state altered by the body of the if statement, times two for the case that the body of the if statement does not get executed, and therefore the bits do not get altered. From this it should be evident that each time we add the tiniest little something to our program, we are exponentially increasing its state complexity.

There are only two constructs that I know of which actually reduce program state complexity instead of increasing it. One is the final keyword, and the other is the assert statement. The final keyword makes bits unalterable, thus excluding them from program state. The assert keyword limits the ranges of values that variables may have, thus also reducing the number of bits that participate in program state, and it even goes one step further, eliminating if statements and guaranteeing that certain paths of execution will never be followed.

So, think about it: every time you code an assertion statement you are actually making your program more simple instead of more complicated. I believe that this realization alone should be enough to convince anyone that assert, along with final, are literally the most useful constructs that any programmer could ever use.

8. Assertions help the compiler make better sense of your code.

Please do try this at home:

void foo( Object x ) { assert x != null; if( x == null ) { } }

If you have a reasonable number of warnings enabled, (as any decent programmer would,) your compiler should issue a warning telling you that the condition x == null is always false. What this demonstrates is that the preceding assertion gave the compiler knowledge of the fact that x cannot be null, and the compiler is now making use of this knowledge as it compiles the rest of your method, pointing out to you potential flaws in your reasoning. Furthermore, some compilers perform useful optimizations based on knowledge that they gather from assertion statements. (I do not know to what extent java compilers do that, but I know for a fact that the Microsoft Visual C++ compiler does it; see footnote 2.)

Conclusion

Test code should always be treating production code as a black box, and knowledge of the inner workings of the production code should be used at most as a hint for testing, never as an instrument for testing. Consequently, assertions are necessary for performing white box tests within the production code so as to ensure that absolutely nothing is left to chance.

Additionally, assertions help document the code, reduce its complexity, strengthen it against assumptions made by the testing code, and help build systems that, in the event of an error, can tell you what is wrong with themselves.

The fact that assertions are usually disabled on deployed systems means that their use can be thought of as incurring a zero performance penalty, which allows programmers to develop a maximalistic error-checking culture of having every single assumption always checked, never leaving anything to chance. By contrast, runtime checks always incur a performance penalty, and for this reason programmers tend to use them on a minimalistic, "only if necessary" basis. So, with runtime checks, programmers tend to constantly ask "should I check this?" while with assertions, they can develop the habit of asking "is there anything I forgot to check?" (If there is only one paragraph that you should take home from this paper, that was it.)

Addendum: How to use assertions

Assertions can be used in every single place where an unexpected exception would otherwise be thrown. This includes all instances of contract violations such as a null pointer exception or an illegal argument exception, and in general, all instances of errors which a) indicate a bug, (should never happen,) and b) cannot be handled in any meaningful way.

In java the assert keyword cannot throw any exception that you might wish it to throw; it only throws the AssertionError exception. Usually this is not a problem, because no piece of production code should ever try to catch an unexpected exception. (That is, after all, precisely why it is called unexpected; it is not that the guest may arrive unannounced, it is that he is not supposed to arrive at all.) Therefore, since it is never meant to be caught, AssertionError is, generally speaking, a suitable one-class-fits-all replacement of all unexpected exceptions.

However, when developing a library, which will be used with assertions presumably enabled at times, you will of course need to have tests which attempt to use your library in various wrong ways, ascertaining that every single one of those attempts gets asserted against, and the problem then with all errors being reported as AssertionErrors is that your tests cannot tell whether the error that was caught was the specific error that you were testing for, and not some unrelated coincidental error. For this reason, when developing libraries, a different approach is needed. One approach is the following:

assert n != 0 : new IllegalArgumentException( "n" );

This works nicely because according to the java language specification, the expression at the right hand side of the colon of an assertion statement does not have to be of any particular type, but if it happens to be of an exception type, then it will be treated as the 'cause' of the AssertionError exception. (In all other cases, its toString() will be used as the 'message' of the AssertionError exception.) Thus, your testing code can catch the assertion exception and examine its cause to ensure that it is indeed the expected exception.

So, your test suite might contain a utility method like the following:

public final <T extends Throwable> T expectException( Class<T> exceptionClass, Runnable runnable )
{
    try
    {
        runnable.run();
    }
    catch( Throwable throwable )
    {
        if( throwable instanceof AssertionError && throwable.getCause() != null )
            throwable = throwable.getCause();
        assert exceptionClass.isInstance( throwable ) : throwable; //exception of the wrong kind was thrown.
        assert throwable.getClass() == exceptionClass : throwable; //exception thrown was a subclass, but not the exact class, expected.
        @SuppressWarnings( "unchecked" )
        T result = (T)throwable;
        return result;
    }
    assert false; //expected exception was not thrown.
    return null; //to keep the compiler happy.
}

Which might be used as follows:

IllegalArgumentException e = expectException( IllegalArgumentException.class, () -> myObject.foo( 0 ) );
assert e.getMessage().equals( "n" );

There are times when you have the need to check whether assertions are enabled. For example, in the first thing I always do in my test suites is to make sure that assertions are enabled, because if someone forgot to pass the -ea switch, all tests may pass without checking for a single defect in the code. Here is how to test whether assertions are enabled in java:

    public static boolean isAssertEnabled()
    {
        //noinspection UnusedAssignment
        boolean assertEnabled = false;
        //noinspection AssertWithSideEffects,NestedAssignment,ConstantConditions
        assert assertEnabled = true;
        //noinspection ConstantConditions
        return assertEnabled;
    }

Note: the //noinspection comments are understood by IntelliJ IDEA; if the misfortune hath befallen thee of having to use some other IDE such as Eclipse, you will have to use some different warning suppression notation.

----------------------

Footnote 1

What makes the Fragile Test Problem especially bad is that the process of fixing tests to make them pass is often carried out under unfavorable conditions, resulting in a sloppy job:

The programmer needs to switch his frame of mind back and forth between his core work and the somewhat unrelated and usually less exciting context of tests;
The need for these fixes is usually unforeseen, so time for the fixes is rarely allocated in the schedule, which means that the programmer fixing broken tests is usually in a hurry;
It is not always clear whether the production code is right and the test is wrong, or whether the test is right and a dormant bug in the production code has been exposed; etc.

So, what tends to happen is that tests are quite often fixed sloppily, so over time they tend to evolve to "test around" (specifically pass) long-standing bugs.

In the preface of Roy Osherove's The Art of Unit Testing (Manning, 2009) the author admits to having participated in a project which failed to a large part due to the tremendous development burden imposed by badly designed unit tests which had to be maintained throughout the duration of the development effort.

Footnote 2

In Microsoft Visual C++ the ASSERT(x) macro does not expand to nothing when _DEBUG is undefined; instead, it expands to an assume(x) intrinsic directive which, even though it does not cause any code to be emitted, it allows the asserted expression to survive the preprocessing step and to be considered by the compiler, so that the corresponding optimizations can take place in the release build, too.

2014-09-21

Assertions and Testing