Software Testing with Fakes instead of Mocks

Abstract:

What are fakes, what are their benefits, and why they are incontestably preferable over mocks. Also, how to create fakes if needed.

Introduction

(Useful pre-reading: About these papers)

When testing a component it is often necessary to refrain from connecting it with the real collaborators that it would be connected with in a production environment, and instead to connect it with special substitutes of its collaborators, also known as test doubles, which are more suitable for testing than the real ones.

One book that names and describes various kinds of test doubles is xUnit Test Patterns: Refactoring Test Code by Gerard Meszaros, (xunitpatterns.com) though I first read about them from martinfowler.com - TestDouble, which refers to Meszaros as the original source.

There exist a few different kinds of test doubles; by far the most commonly used kind is mocks, which, as I explain elsewhere, are a very bad idea and should be avoided like COVID-19. (See michael.gr - If you are using mock objects you are doing it wrong.) Another kind of test double, which does not suffer from the disadvantages of mocks, is Fake Objects, or simply fakes.

What are fakes

In just one word, a fake is an emulator.

In a bit more detail, a fake is a component that fully implements the interface of the real component that it substitutes, or at any rate the subset of that interface that we have a use for; it maintains state which is equivalent to the state of the real component, and based on this state it provides the full functionality of the real component, or a very convincing illusion thereof; to achieve this, it makes some compromises which either do not matter during testing, or are actually desirable during testing. Examples of such compromises are:

• Having limited capacity.
• Not being scalable.
• Not being distributed.
• Not remembering any state from run to run.
• Pretending to interact, but not actually interacting, with the physical world.
  
• Generating fake data that would be unusable in a real production scenario.

A fake can be more suitable for testing than the real thing in the following ways:

• By performing much better than the real thing; for example:
• by keeping state in-memory instead of persisting to the file-system.
• by working locally instead of over the network.
• by pretending that the time has come for the next timer to fire instead of having to wait for that timer to fire.
• etc.
• By being deterministic; for example:
• by fabricating time-stamps instead of querying the system clock.
• by fabricating entities such as GUIDs, that would otherwise introduce randomness.
• by utilizing a single thread, or forcing threads to work in a lock-step fashion.
• etc.
• By avoiding undesirable interactions with the real world; for example:
• by pretending that a mass e-mail was sent instead of actually sending it.
• by pretending that an application-modal message box popped up, and that the user picked one of the available choices, instead of allowing an actual modal message box to block the running of tests on the developer's computer or, worse yet, on some continuous build server in some data center out there.
• by pretending that a robot made a movement, instead of causing an actual robot to move.
• etc.

A few examples of frequently used fakes:

• Various in-memory file-system libraries exist for various platforms, which can be used in place of the actual file-systems on those platforms.
• HSQLDB and H2 for Java, in-memory DbContext for DotNet EntityFramework, etc. are in-memory database systems that can be used in place of actual Relational Database Management Systems when testing.
• EmbeddedKafka can be used in place of an actual pair of Kafka + Zookeeper instances.
• A pseudo-random number generator seeded with a known constant value acts as a fake of the same pseudo-random number generator seeded with a practically random value such as the current time coordinate.

To recap:

• Fakes refrain from performing the actual operations that the real thing would perform, (e.g. when a file is created while using an in-memory file-system, no file gets created on disk,) but
• They do go through all the motions, (e.g. attempting to create a file using an invalid filename will cause an error just as in a real file-system,) and
• They do maintain the same state, (e.g. reading a file from an in-memory file-system will yield the exact same data that were previously written to it,) so
• They do fully behave as if the operations were actually performed as far as the component-under-test is concerned, while
• The compromises that they make in order to achieve this are inconsequential or even desirable when testing. (e.g. during a test run it does not matter if files created during a previous test run do not exist anymore, and as a matter of fact it is better if they do not exist.)

Note that the terminology is a bit unfortunate: fakes are not nearly as fake as mocks.

• Mocks are the ultimate in fakery because:
• They only respond to invocations that we prescribe in each test, based on our assumptions as to how the component-under-test would invoke the real thing.
• They maintain no state.
• They contain no functionality.
• They only return results that we prefabricate in each test, based on our assumptions as to how the real thing would respond.
• Fakes are not quite as fake as their name suggests, because:
• They expose the same interface as the real thing.
• They maintain an equivalent state as the real thing.
• They implement equivalent functionality as the real thing.
• They return the exact same results as the real thing.

Benefits of fakes

• By using a fake instead of the real thing:
• We achieve better performance, so that our tests run quickly.
• We avoid non-determinism during testing, so our tests are repeatable.
• We avoid undesirable interactions with the real world, so nobody gets hurt.
• We have less code to write, since a fake is usually simpler to set up than the real thing.
• By using a fake instead of a mock:
• We save ourselves from having to write complicated mocking code in each test.
• We do not need to claim any knowledge as to how the component under test invokes its collaborators.
• We do not have to make assumptions about the state in which the collaborators are at any given moment.
• We do not have to make assumptions as to what results would be returned by each collaborator in each invocation.
• In both cases:
• We are incorporating in our tests a collaborator which has already been tested and can be reasonably assumed to be free of defects. Thus, in the event of a test failure we can be fairly confident that the defect lies in the component-under-test, (or in the test itself,) but not in one of the collaborators, so we achieve defect localization, which is the aim of Unit Testing.

Creating fakes of our own components

In some cases we may want to create a fake ourselves, as a substitute of one of our own components. Not only will this allow other components to start their testing as early as possible without the need for mocks, but also, a non-negligible part of the effort invested in the creation of the fake will be reusable in the creation of the real thing, while the process of creating the fake is likely to yield valuable lessons which can guide the creation of the real thing. Thus, any effort that goes into creating a fake of a certain component represents a much better investment than the effort of creating a multitude of throw-away mocks for various isolated operations on that component.

One might argue that keeping a fake side-by-side with the real thing may represent a considerable additional maintenance overhead, but in my experience the overhead of doing so is nowhere near the overhead of maintaining a proliferation of mocks for the real thing.

• Each time the implementation of the real thing changes without any change to its specification, (such as, for example, when applying some refactoring, or a bug fix,) some mocks must be modified, some must even be rewritten, while the fake usually does not have to be touched at all.
• When the specification of the real thing changes, the mocks have to be rewritten, and the fake has to be modified, but the beauty of the fake is that it is a self-contained module which implements a known abstraction, so it is easy to maintain, whereas every single snippet of mocking code is nothing but incidental complexity, and thus hard to maintain.
• In either case, a single change in the real thing will generally require a single corresponding change in the fake, whereas if we are using mocks we invariably have to go changing an arbitrary number of mocking snippets scattered throughout the tests.

Furthermore, the use of fakes instead of mocks promotes the creation of black-box tests instead of white-box tests. Once we get into the habit of writing all of our tests as black-box tests, new possibilities open up which greatly ease the development of fakes: we can now write a test for a certain module, and then reuse that test in order to test its fake. The test can be reused because it is a black-box test, so it does not care how the module works internally, therefore it can test the real thing just as well as the fake of the real thing. Once we run the test on the real thing, we run the same test on the fake, and if both pass, then from that moment on we can continue using the fake in place of the real thing in all other tests.

The tests that exercise the real thing will be slow, but the real thing does not change very often, (if ever,) so here is where a testing tool like Testana shines: by using Testana we ensure that the tests exercising the real thing will only run in the rare event that the real thing actually changes. For more information about Testana, see michael.gr - Testana: A better way of running tests.

Creating fakes of external components

If we are using an external component for which no fake is available, we may wish to create a fake for it ourselves. First, we write a test suite which exercises the external component, not really looking for defects in it, but instead using its behavior as reference for writing the tests. Once we have built our test suite to specifically pass the behavior of the external component, we can reuse it against the fake, and if it also passes, then we have sufficient reasons to believe that the behavior of the fake matches the behavior of the external component. A similar technique is described by Martin Fowler in his Contract Test post.

In an ideal world where everyone would be practicing Black-Box testing, we should even be able to obtain from the creators of the external component the test suite that they have already built for testing their creation, and use it to test our fake.

In an even more ideal world, anyone who develops a component for others to use would be shipping it together with its fake, so that nobody needs to get dirty with its test suite.

Conclusion

Despite widespread practices in the industry, fakes are the preferred alternative to mocks. Even though they might at first seem laborious, they are actually very convenient to use, and on the long run far less expensive than mocks.

---

Cover image: fake moustache by michael.gr based on art by Claire Jones from the Noun Project.