Software Testing with Fakes instead of Mocks

What are fakes, what are their benefits, and why they are uncontestably preferable over mocks. Also, how to create fakes if needed.

Introduction

(Useful pre-reading: About these papers)

When testing a component it is often necessary to refrain from connecting it with the real collaborators that it would be connected with in a production environment, and instead to connect it with special substitutes of its collaborators, also known as test doubles, which are more suitable for testing than the real ones.

One book that names and describes various kinds of test doubles is xUnit Test Patterns: Refactoring Test Code by Gerard Meszaros, (xunitpatterns.com) though I first read about them from martinfowler.com - TestDouble, which refers to Meszaros as the original source.

There exist a few different kinds of test doubles; by far the most commonly used kind is mocks, which, as I explain elsewhere, are a very bad idea and should be avoided like COVID-19. (See michael.gr - On Mock Objects and Mocking.) Another kind of test double, which does not suffer from the disadvantages of mocks, is fakes.

What are fakes

In just one word, a fake is an emulator.

(I would have very much preferred the industry to be calling them "emulators" instead of "fakes".)

In a bit more detail:

A fake is a component that fully implements the interface of the real component that it substitutes, or at any rate the subset of that interface that we have a use for; it maintains state which is equivalent to the state of the real component, and based on this state it provides the full functionality of the real component; to achieve this, it makes some compromises which either do not matter during testing, or are actually desirable during testing, such as:

• Having limited capacity.
• Not being scalable.
• Not being distributed.
• Not remembering any state from run to run.
• Generating fake data that would be unusable in a real production scenario.

A fake can be more suitable for testing than the real thing in the following ways:

• By performing much better than the real thing; for example:
• by keeping state in-memory instead of persisting to the file-system.
• by working locally instead of over the network.
• by pretending that the time has come for the next timer to fire instead of having to wait for a timer to fire.
• etc.
• By being deterministic; for example:
• by fabricating time-stamps instead of querying the system clock.
• by fabricating entities such as GUIDs, that would otherwise introduce randomness.
• by utilizing a single thread, or forcing its threads to work in a lock-step fashion.
• etc.
• By avoiding undesirable interactions with the real world; for example:
• by pretending that a mass e-mail was sent instead of actually sending it.
• by pretending that an application-modal message box popped up, and that the user dismissed it, instead of allowing an actual modal message box to block the operation of some CI/CD server in some data center out there.
• by emulating the movements of a factory robot instead of causing an actual factory robot to move about the factory floor.
• etc.

A few examples of frequently used fakes:

• Various in-memory file-system libraries exist for various platforms, which can be used in place of the actual file-systems on those platforms.
• In Java, HSQLDB and H2 are in-memory databases that can be used in place of an actual relational database.
• In DotNet, EntityFramework allows the creation of an in-memory DbContext.
• EmbeddedKafka can be used in place of an actual pair of Kafka + Zookeeper instances.
• A pseudo-random number generator seeded with a constant value is a fake of the same pseudo-random number generator seeded with the current time coordinate.

To recap: 

• Fakes refrain from performing the actual operations that the real thing would perform, (e.g. when a file is created while using an in-memory file-system, no actual file gets created on disk,) but
• They do go through all the motions, (e.g. attempting to create a file using an invalid filename will cause an error just as in a real file-system,) and 
• They do maintain the same state, (e.g. reading a file from an in-memory file-system will yield the exact same data that were previously written to it,) so 
• They do fully behave as if the operations were actually performed, while 
• The compromises that they make in order to achieve this are inconsequential or even desirable when testing. (e.g. during a test run it does not matter if files created during a previous test run do not still exist, and as a matter of fact it is better if they do not still exist.)

Note that the terminology is a bit unfortunate: fakes are not nearly as fake as mocks.

• Mocks are the ultimate in fakery because they contain no functionality and no state; they only respond to specific invocations that we assume will be made, and they return results that are pre-fabricated by us, based on our assumptions about the state that the real thing should be in.
• Fakes do actually implement the functionality of the real thing, so they are capable of responding to any invocation regardless of our assumptions, and they return the results that the real thing would return, without us having to guess what they should be.

Benefits of fakes

• By using a fake instead of the real thing:
• We achieve better performance, so that our tests run quickly.
• We avoid non-determinism during testing, so our tests are repeatable.
• We avoid undesirable interactions with the real world, so nobody gets hurt.
• We have less code to write, since a fake is usually a lot simpler to initialize and clean-up than the real thing.
• By using a fake instead of a mock: 
• We save ourselves from having to write copious amounts of complicated mocking code in each test.
• We do not need to claim any knowledge as to how the component under test invokes its collaborators.
• We do not have to make assumptions about the state in which the collaborators are at any given moment.
• We do not have to guess what results would be returned by each collaborator in each invocation.
• In both cases: 
• We are incorporating in our tests a collaborator which has already been tested by its creators and can be reasonably assumed to be free of defects. Thus, in the event of a test failure we can be fairly confident that the defect lies in the component-under-test, (or in the test itself,) but not in one of the collaborators, so we achieve defect localization, which is the main aim of Unit Testing.

Creating fakes of our own components

In some cases we may want to create a fake ourselves, as a substitute of one of our own components. Not only will this allow other components to start their testing as early as possible without the need for mocks, but also, a non-negligible part of the effort invested in the creation of the fake will be reusable in the creation of the real thing, while the process of creating the fake is likely to yield valuable lessons which can guide the creation of the real thing. Thus, any effort that goes into creating a fake of a certain component represents a much better investment than the effort of creating a multitude of throw-away mocks for various isolated operations on that component.

One might argue that keeping a fake side-by-side with the real thing may represent a considerable additional maintenance overhead, but in my experience the overhead of doing so is nowhere near the overhead of maintaining a proliferation of mocks for the real thing.

• Each time the implementation of the real thing changes without any change to its specification, (such as, for example, when applying a bug fix,) some mocks must be modified, some must even be rewritten, while the fake usually does not have to be touched at all.
• When the specification of the real thing changes, both the mocks have to be rewritten, and the fake has to be modified, but the beauty of the fake is that it is a self-contained module which implements a known abstraction, so it is easy to maintain, whereas every single snippet of mocking code is nothing but incidental complexity, and thus hard to maintain.
• In either case, a single change in the real thing will require at most a single corresponding change in the fake, whereas if we are using mocks we invariably have to go changing a large number of mock snippets scattered throughout the test suites.

Furthermore, the use of fakes instead of mocks promotes the creation of black-box tests instead of white-box tests. Once we get into the habit of writing all of our tests as black-box tests, new possibilities open up which greatly ease the development of fakes: it is now possible to write a test for a certain module, and then reuse that test in order to test its fake. The test can be reused because it is a black-box test, so does not care how the module works internally, therefore it can test the real thing just as well as the fake of the real thing. Once we run the test on the real thing, we run the same test on the fake, and if both pass, then from that moment on we can continue using the fake in place of the real thing when testing anything that uses it.

Creating fakes of external components

If we are using an external component for which no fake is available, we may wish to create a fake for it ourselves. First, we write a test suite which exercises the external component, not really looking for defects in it, but instead using its behavior as reference for writing the tests. Once we have built our test suite to specifically pass the behavior of the external component, we can reuse it against the fake, and if it also passes, then we have sufficient reasons to believe that the behavior of the fake matches the behavior of the external component.

In an ideal world where everyone would be practicing Black-Box testing, we should even be able to obtain from the creators of the external component the test suite that they have already built for testing their creation, and use it to test our fake.

In an even more ideal world, anyone who develops a component for others to use would be shipping it together with its fake, so that nobody needs to get dirty with its test suite.

Conclusion

Despite widespread practices in the industry, fakes are the preferred alternative to mocks. Even though they might at first seem laborious, they are actually very convenient to use, and on the long run far less expensive than mocks.

---

Cover image: fake moustache by michael.gr based on art by Claire Jones from the Noun Project.