The Deployable Design Document

Abstract

A need is identified and a solution is proposed for a novel set of software tools to facilitate the visual composition of technical software design documents as schematic diagrams consisting of predefined software components, and the automatic deployment of runnable software systems from such design documents.

(Useful pre-reading: About these papers)

Note: this post has been superseded by michael.gr - Towards Authoritative Technical Software Designs

The problem, in summary

The conventional means for creating technical design documents for software systems today are paper, whiteboard, or in the best case some general-purpose box-and-arrow drawing application.

Designs created by such means tend to have arbitrary notation, arbitrary content, arbitrary semantics, and even arbitrary levels of abstraction, making them works of art rather than works of engineering: they are mere suggestive sketches of the vague wishes of the architects, bearing no necessary relationship to reality.

In order to bridge the gap between what the design describes and what can actually be built and deployed, software engineers and operations engineers tend to engage in various degrees of improvisation, resulting in software systems that are described not by the design, but instead by the source code, and various scripts and configuration files scattered in various places.

Ideally, a technical design document should be the single authoritative source of truth for the actual structure, composition, and topology of the system that it describes.

Prior Art

The promise of visual tools for software engineering has existed since the dawn of our discipline.

• A plethora of Integrated Development Environments (IDEs) (wikipedia) and Rapid Application Development (RAD) (wikipedia) environments have been created, aiming to ease development by means of visual tools, but these tools are usually restricted to a Graphical User Interface Builder (Forms Designer) and perhaps a few ways of illustrating, but not defining, various aspects of the application being developed, such as class diagrams, dependency diagrams, control flows, data dependencies, etc.
• In the Microsoft world there have been products (e.g. Visio) embodying a latent promise for visual software design tools, and products (e.g. Microsoft Visual C++) with deceitful titles alluding to such tools, but so far, no product that actually delivers any such tools. (See michael.gr - On Microsoft "Visual" products.)
• Some Visual Programming Languages (wikipedia) have been created, such as Snap!, Scratch, EduBlocks, Blockly, etc., which do produce runnable programs, but they are exactly equivalent to program code, so they express implementations rather than designs. (See michael.gr - On Visual Programming Languages.)
• A plethora of Specification Languages (wikipedia), Modelling Languages (wikipedia), and Architecture Description Languages (wikipedia) have been created, most of which are virtually unknown outside the circles that invented them. Of those that are somewhat known, most see very little practical application in the real world. A few that are noteworthy are:
• The Unified Modelling Language (UML) (wikipedia) was an attempt to standardize at least the notation of diagrams, but it is cumbersome to work with, and there are not many tools that actually do something meaningful with these diagrams, so it is largely in disuse. (See michael.gr - On UML.)
• The Specification and Description Language (SDL)(wikipedia) is seeing some use in process control and real-time applications. It is suitable for describing implementations rather than designs.
• The Business Process Modeling Notation (BPMN) (wikipedia) is seeing some use in describing business processes within software systems, but not the software systems themselves.
• Various reverse-engineering tools have been developed, such as Lucidscale (lucidscale.com) which tend to be limited to specific realms, such as the cloud environments of particular vendors, and are invariably restricted to the visualization, exploration, and documentation of existing systems, rather than the design and deployment of new systems, or even the modification of existing systems.
• Various programming paradigms have been proposed upon which visual tools can be built, such as Flow-based Programming (FBP) (wikipedia) suffering from problems such as restricting the means by which components within the system may communicate (e.g. only via asynchronous message-passing.) Furthermore, implementations of such paradigms suffer even further from being tied to specific programming languages, for example NoFlo (noflojs.org, kickstarter) is tied to JavaScript.
• C4 Model (wikipedia) -- Work in progress.
• Model-Driven Engineering (wikipedia) -- Work in progress.
• Low Code Development Platforms (wikipedia) -- Work in progress.
• Various infrastructure definition tools like Terraform (wikipedia) -- Work in progress.

The problem, in detail

Conventional means of software design suffer from the following shortcomings:

• Designs often include elements that are not well-defined in engineering terms.

You see this with design documents containing supposedly technical but actually quite nebulous entities such as a data store here, a messaging backbone there, or a remote server over there. None of these entities is concrete enough and unambiguous enough to be suitable for inclusion in a technical design document.

• Designs often include elements that are outside the realm of engineering.

You see this with design documents containing little human figures representing users, pictures of money representing payments, etc. The presence of such items in a software design usually indicates a confusion between what is a technical design and what is a functional specification.

• Designs often include elements from wrong levels of abstraction.

You see this with designs that mix software components with flowcharts. Notwithstanding the fact that flowcharts are also boxes connected with arrows, they represent decision-making logic, which is an implementation detail of the component that provides it, and as such they have no place in a technical design document.

You also see this with designs that confuse interfaces with other concepts, such as ownership, containment, inheritance, information flows, etc. thus prescribing connections between components that are unattainable, inapplicable, or irrelevant.

• Designs are distanced from the engineering entities they deal with.

Conventional means of software design provide no means of establishing or enforcing a correspondence between a box as it appears on the design, and the actual runnable software module that it represents. As such, designs are not informed by what modules are actually available for incorporation, nor about the ways in which these modules can actually be interconnected. Thus, designs tend to be based on hypotheses, assumptions, and wishes, rather than fact.

• Designs are often expressed at an unworkably high level of abstraction.

The level of abstraction most commonly chosen by software architects is that of a block diagram, which might be suitable for abstract architectural work, but does not contain enough detail to guarantee the feasibility of the technical design. The level of abstraction necessary in order to guarantee feasibility is that of the schematic diagram, which is one step lower.

Unfortunately, since designs are distanced from the engineering entities they deal with, they do not have enough factual information at their disposal to be able to delve into such a level of detail as necessary for a schematic diagram.

Furthermore, conventional design tools tend to be two-dimensional, lacking support for hierarchy; thus, they do not facilitate the decomposition of a schematic diagram into nested schematic diagrams expressing finer levels of integration. Producing an entire technical design at a single level of integration would be unwieldy, so it is avoided. Breaking down a conventional design into subsets expressing different levels of integration suffers from consistency problems, since there exist no technical means of establishing and enforcing a relationship between a certain subset of the design and the box which represents that subset within some other subset.

• Designs often prescribe impossible combinations of elements.

The ways in which a design intends to interconnect components do not necessarily match the ways in which the components can actually be interconnected.

• A design may assume that a certain component exposes a particular interface while in fact the component does not expose such an interface.
• A design may prescribe a connection between two components on a certain interface, while in fact the components cannot be connected because the type and nature of the interface that one component exposes is not the same as the type and nature of the interface that the other component consumes.

• Designs are often incomplete.

Even if a design manages to stick to entities that are well-defined in engineering terms, suitable for inclusion in a software design, and of the right level of abstraction, (such as predefined software components on a schematic diagram,) the design may still:

• Incorporate a component which needs to invoke a certain interface but omit incorporating a component which implements that interface.
• Incorporate a component which exposes a certain interface which must be invoked for the system to function properly but omit incorporating a component which invokes that interface.

In such cases, the software system cannot be deployed as designed, and yet the architects are free to proclaim the design as complete.

The above list of problems results from the lack of design tools capable of informing the design with what is available and restricting it to what is possible. Correspondingly, the process of building and deploying software systems is also not informed, nor restricted, by the design. This causes the following severe problems:

• Software systems do not necessarily match their designs.

Even if the technical design happens to describe a software system that can actually be built as described, (which is rarely the case,) there are no technological safeguards to guarantee that it will, so the software engineers and the operations engineers are free to build and deploy a system that bears little or no relationship to the design. Neither the architects, nor the management, have any way of knowing.

• Software systems diverge from their designs over time.

Even if the technical design initially matches the deployed software system, (which, again, is almost never the case,) the system is bound to evolve. The design should ideally evolve in tandem, but it rarely does, again because there are no technological safeguards to enforce this; the engineers are free to modify and redeploy the system without updating the design document, and in fact quite often they do, because they can. (It is the path of least resistance.) Thus, sooner or later the design does not correspond to reality anymore.

If, due to the above reasons, you suspect that your technical design document does not correspond to reality, and you would like to know exactly what it is that you have actually deployed and running out there, you have to begin by asking questions to the software engineers and the operations engineers.

In order to answer your questions, they will in turn have to examine source code, version control histories, build scripts, configuration files, server provisioning scripts, and launch scripts, because the truth is scattered in all those places. In some cases they might even have to try and remember commands that were once issued to bring the system to life. If this sounds a bit like it is held together by shoestrings, it is because it is indeed held together by shoestrings.

Thus, the information that you will receive will hardly be usable, and even if you manage to collect it all, make sense out of it, and update the design document with it, by the time you are done, the deployed system may have already changed.

As a result, it is generally impossible at any given time to know the actual technical design of a deployed software system with any degree of certainty.

This is a very sorry state of affairs for the entire software industry to be in.

The Solution, in summary

In order to ensure that the technical designs of software systems describe fact and not mere intention, we need:

• A new type of design document which is innately deployable.
• A toolset which facilitates:
• Creation of such documents graphically.
• Deployment of systems according to their designs, with the press of a button.

By deploying a system from its own design document, we guarantee that it exactly matches its design.

For the design document to be deployable, the following must be true:

• The design must consist of well-defined components.
• Components must be interconnectable in well-defined ways.
• Components must be capable of deploying instances of themselves.
• All components necessary for the deployed system to function must be present in the design.

A component is well-defined if it implements a certain design protocol, which:

• Describes a set of characteristics of a component
• Can be requested to deploy a runnable instance of the component according to information present in the design.

Characteristics of a component include:

• The type-name of the component,
• A list of design-time properties that the component exposes,
• A list of inputs and outputs that the component supports.

Inputs and outputs are collectively known as ports.

• An input of a component represents functionality that the component is making available for other components to use.
• An output of a component represents functionality which must be made available by other components for the component to use.

Essentially, an output is nothing but a reference to some input somewhere else in the design.

Well-defined interconnectivity is achieved by establishing a strict set of rules that govern how connections can be made between ports:

• Each port is of a certain type which uniquely identifies the functionality that is being made available via this port. For example, ports of type acme-weather-forecast-service can be used for exchanging weather forecast information in a particular way which has been specified by Acme, Inc.
• Each port is of a certain nature, which identifies the technical means by which invocations are made via this port. For example, acme-weather-forecast-service might be available in java, in which case the corresponding ports will be of java-interface nature, while the same service might also be available as a web service, in which case the corresponding ports will be of rest-over-http nature.
• An input can be connected to an output only if they are both of the same nature and same type.

The distinction between natures and types exists in order to facilitate adaptors. An adaptor maps one nature to another while retaining the type. For example, acme-weather-forecast-service might be implemented in Java, thus being represented as an input of java-interface nature. To make it available as a web service we will need a rest-over-http-from-java-interface adaptor which exposes an output of the same nature and type so that it can invoke the existing input, and exposes an input which is still of the same type but of rest-over-http nature. It is anticipated that with the help of reflection, many adaptors can be implemented as general-purpose components accomplishing their task without requiring any application-specific code to be written.

The requirement for completeness can be enforced with the following rules:

• Every output must be connected to an input.
• Every input must have at least one output connected to it.

Some software systems are so complex that expressing them in a single technical design may be inconvenient to the point of being unworkable. To solve this issue, we facilitate Hierarchical System Composition by introducing containers.

• A container is a component which encapsulates an entire separately-editable design, and exposes some of the inputs and outputs of the nested design as inputs and outputs of its own.

Thus, containers are capable of abstracting away entire sections of a design, and they can be nested indefinitely, so they facilitate designs of any scale and complexity.

Any conventional pre-existing or yet-to-be-conceived runtime module can be turned into a well-defined component which is suitable for incorporation into a technical design by creating, and registering with the design toolset, a special design companion module which implements the design protocol on behalf of the runtime module.

The solution, in detail

Work in progress.