On messages and message-passing

Over the decades, numerous software system architectures have emerged which require invocations across subsystems to be done via message-passing instead of programmatic interface method calls. Such architectures are so common that many programmers have come to regard message-passing as an end in and of itself, oblivious of the fact that it is nothing but a (poor) technical mechanism for accomplishing a certain architectural goal.

(Useful pre-reading: About these papers)

The architectural goal is to be able to perform general-purpose operations on invocations, for example routing the invocations according to configuration, or queuing the invocations for delivery on a different thread. In order to be able to do things like that, the invocations must first be expressed in a general-purpose form.

Message-passing is simply the only general-purpose form that could be imagined by the pioneers who built the first asynchronous event-driven systems, or perhaps the only form that could readily be implemented using the programming languages available back then. However, in succeeding decades our thinking and our tools have advanced considerably, to the point where we now have much better ways of achieving things technically, so it might be worth taking a moment to re-examine the concept of message-passing.

Here is a list of problems with message passing:

• Custom message classes have to be written and maintained, usually in large numbers, constituting nothing but incidental complexity which steers focus away from the class hierarchy of the problem domain, and towards the class hierarchy of the overelaborate inter-module communication apparatus.
• For each invocation, a message class needs to be instantiated, filled, and submitted, requiring several lines of custom-written code. This is also nothing but incidental complexity, diverting the attention of programmers from solving the problem at hand to negotiating the trifling technicalities of placing invocations.
• On the receiving end, each message must be examined in order to determine what kind of message it is, usually by means of an unwieldy switch statement, and its contents have to be extracted before any useful work can be done. Again, this is all incidental complexity, contributing nothing towards the end-goal of the software system; its sole purpose is to serve the message-passing bureaucracy.
• In order to reduce the total number of different message classes that need to be defined, programmers often reuse message classes for different purposes, filling different parts according to each purpose. This habit further increases the total amount of incidental complexity both at the sending and at the receiving end, and very often leads to bugs due to wrongly packed or wrongly unpacked messages.

So, message-passing exists for the sole purpose of expressing invocations in a general-purpose form, but as it turns out, its use is laborious, and it tends to flood systems with debilitating amounts of incidental complexity.

The most natural, simple, convenient, straightforward, robust, maintainable, and self-documenting paradigm for making and receiving invocations, which facilitates problem-solving instead of hindering it, is programmatic interface method calls. Unfortunately, interfaces are not general-purpose in and of themselves, because each interface constitutes a unique type, requiring custom-written code to place calls to it and custom-written code to receive calls for it, thus preventing us from applying general-purpose operations on it.  So, we have two separate and seemingly conflicting concerns:

• How to express invocations in the most convenient way
• How to perform general-purpose operations on the invocations

Ideally, separate concerns should not be mixed; the need to somehow apply general-purpose operations on invocations should not be dictating how we write code, and it should certainly not be making our job harder. Therein lies perhaps the biggest objection to message-passing: they are an onerous contrivance that programmers by themselves would never opt to use out of their own free will, but usually gets imposed on them by software architects who do not actually have to write code using this contrivance.

Message-passing has enjoyed widespread use mainly due to the historical inability of application programmers to think in terms of abstractions: it is always possible, even in systems that require message-passing, to write all application code so that it never deals with any messages at all, and uses nothing but application-specific programmatic interfaces instead; the trick is to create packaging and unpackaging adaptors, where on the sending side we are simply invoking a programmatic interface which is implemented by a packaging adaptor that creates messages, packs them, and sends them off to be enqueued, while on the receiving side a corresponding unpackaging adaptor is fed with messages from the queue, unpacks them, and calls the corresponding implementation of the interface. Alas, this arrangement requires a modicum of abstract thinking, and application programmers are generally not into that sort of thing.

Furthermore, if we bother creating such packaging and unpackaging adaptors, the realization quickly starts to sink-in that all the message classes are irrelevant; there is no need to define a special message class containing a separate field for each parameter of each method, because the only code that would ever deal with such a class would be the corresponding pair of packaging and unpackaging adaptors; so, the adaptors might as well use a single universal message class which simply stores all parameters in an array of object, and voila, the entire menagerie of message classes becomes entirely unnecessary.

Thus, it becomes evident that what we are really after is not message-passing per se; it is some general-purpose form of expressing invocations, so that general-purpose operations can be performed on them, and some mechanism for converting back and forth between this general-purpose form and the natural form, which is programmatic interface method calls, so that we can write code naturally. Ideally, the conversion mechanism would be automatic and transparent, so that we do not even have to write those adaptors. Messages have only existed due to the historical absence of such an automatic and transparent mechanism.

Fortunately, with modern reflecting, intermediate-code-based, just-in-time compiled programming languages, today we have at our disposal all that is necessary to build such mechanisms. For more information see michael.gr - Intertwine.

Mandatory grumpy cat meme