2022-08-22

The Deployable Design Document

blueprint-technical-drawing-4056027 by xresch, in the public domain,
from https://www.allaboutlean.com/cost-of-complexity/blueprint

Abstract

A need is identified and a solution is proposed for a novel set of software tools to facilitate the visual composition of technical software design documents as schematic diagrams consisting of predefined software components, and the automatic deployment of running software systems from such design documents.

The problem, in summary

The conventional means for creating technical design documents for software systems today are paper, whiteboard, or in the best case some general-purpose box-and-arrow drawing application.

Designs created by such means tend to have arbitrary notation, arbitrary content, arbitrary semantics, and even arbitrary levels of abstraction, making them works of art rather than works of engineering: they are mere suggestive sketches of the vague wishes of the designers, bearing no necessary relationship to reality.

In order to bridge the gap between what the design describes and what can actually be built and deployed, software engineers and operations engineers tend to engage in various degrees of improvisation, resulting in software systems that are described not by the design, but instead by various source code files, scripts, and configuration files scattered in various places.

Ideally, a technical design document should be the single authoritative source of truth for the actual structure, composition, and topology of the system that it describes, but currently there exists no technological mechanism to either facilitate or enforce this.

The problem, in detail

The promise of visual tools for software engineering has existed since the dawn of our discipline; there have been products embodying a latent promise for such tools, and products whose titles alluded to such tools, but never a product that actually delivered such tools. (See michael.gr - On Microsoft "Visual" products.)

Some products like Lucidscale come close to offering the desired visual tools, but they are generally limited to high levels of abstraction within specific realms, (cloud environments of particular vendors,) and are generally restricted to visualization, exploration, and documentation of existing systems rather than creation and deployment of new systems, or even modification of existing systems.

Some Visual Programming Languages have been created, which do produce runnable programs, but their level of abstraction is too low: they are exactly equivalent to program code, so they express technical implementations rather than technical designs. (See michael.gr - On Visual Programming Languages.)

UML was an attempt to standardize at least the notation of software designs, but it is in disuse. (See michael.gr - On UML.) Even if designers could agree to use some standard notation, that would solve the least of our problems, because conventional means of software design suffer from far greater shortcomings:

  • Designs often include elements that are not well-defined in engineering terms.

You see this with supposedly technical but actually quite nebulous entities such as a 'data store' here, a 'messaging backbone' there, or a 'remote server' over there. None of these entities is concrete enough and unambiguous enough to be suitable for inclusion in a technical design document.

  • Designs often include elements that are outside the realm of engineering.

    You see this with little human figures representing users, pictures of money representing payments, etc. The presence of such items in a software design usually indicates a confusion between what is a technical design and what is a functional specification.

    • Designs often include elements from wrong levels of abstraction.

    You see this with designs that mix software components with flowcharts. Notwithstanding the fact that flowcharts are also boxes connected with arrows, they represent decision-making logic, which is an implementation detail of the component that provides it, and as such they have no place in a technical design document.

    You also see this with designs that confuse interfaces with other concepts, such as ownership, containment, inheritance, information flows, etc. thus prescribing connections that are unattainable, inapplicable, or irrelevant.

    • Designs are distanced from the engineering entities they deal with.

    Conventional means of software design provide no means of establishing or enforcing a correspondence between a box as it appears on the design, and the actual runnable software module that it represents. As such, designs are not informed by what modules are actually available for incorporation, nor about the ways in which these modules can actually be interconnected. Thus, designs tend to be based on hypotheses, assumptions, and sometimes even wishes, rather than fact.

    • Designs are often expressed at an unworkably high level of abstraction.

    The level of abstraction most commonly chosen by software designers is that of a block diagram, which might be suitable for abstract architectural work, but does not contain enough detail to guarantee the feasibility of the technical design. The level of abstraction necessary in order to guarantee feasibility is that of the schematic diagram, which is one step lower.

    Unfortunately, since designs are distanced from the engineering entities they deal with, they do not have enough factual information at their disposal to be able to delve into such a level of detail as necessary for a schematic diagram.

    Furthermore, conventional design tools tend to be two-dimensional, lacking support for hierarchy; thus, they do not facilitate the decomposition of a schematic diagram into nested schematic diagrams expressing finer levels of integration. Producing an entire technical design at a single level of integration would be unwieldy, so it is avoided. Breaking down a conventional design into subsets expressing different levels of integration suffers from consistency problems, since there exist no technical means of establishing and enforcing a relationship between a certain subset of the design and the box which represents that subset within some other subset.

    • Designs are often incomplete.

    Even if a design manages to stick to entities that are well-defined in engineering terms, suitable for inclusion in a software design, and of the right level of abstraction, (such as predefined software components on a schematic diagram,) the design may still:

    • Incorporate a supposedly existing component while in fact no such component exists.
    • Incorporate a component which needs to invoke a certain interface but omit incorporating a component which implements that interface.
    • Incorporate a component which exposes a certain mandatory interface (an interface which must be invoked for the system to function properly) but omit incorporating a component which invokes that interface.

    In such cases, the software system cannot be deployed as designed, and yet the designers are free to proclaim the design as complete.

    • Designs often prescribe impossible combinations of elements.
    The ways in which a design intends to interconnect components do not necessarily match the ways in which the components can actually be interconnected.
      • A design may assume that a certain component exposes a particular interface while in fact the component does not expose such an interface.
      • A design may prescribe a connection between two components on a certain interface, while in fact the components cannot be connected because the type of the interface that one component exposes is not the same as the type of the interface that the other component consumes.
    The above list of problems results from the lack of design tools capable of informing the design with what is available and restricting it to what is possible. Correspondingly, the process of building and deploying software systems is also not informed, nor restricted, by the design. This causes the following severe problems:
    • Software systems do not necessarily match their designs.
    Even if the technical design happens to describe a software system that can actually be built as described, (which is rarely the case,) there are no technological safeguards to guarantee that it will, so the software engineers and the operations engineers are free to build and deploy a system that bears little or no relationship to the design. Neither the designers, nor the management, have any way of knowing.
    • Software systems diverge from their designs over time.
    Even if the technical design initially matches the deployed software system, (which, again, is almost never the case,) the system is bound to evolve. Ideally, the design should evolve in tandem, but it rarely does, again because there are no technological safeguards to guarantee that it does; the engineers are free to modify and redeploy the system without updating the design document, and in fact quite often they do, because they can. (It is the path of least resistance.) Thus, sooner or later the design does not correspond to reality anymore.

    If, due to the above reasons, you suspect that your technical design document does not correspond to reality, and you would like to know exactly what it is that you have actually deployed and running out there, you have to begin by asking questions to the software engineers and the operations engineers.

    In order to answer your questions, they will in turn have to examine source code, version control histories, build scripts, configuration files, server provisioning scripts, and launch scripts, because the truth is scattered in all those places. In some cases they might even have to try and remember commands that were once issued to bring the system to life. (If this sounds a bit like it is held together by shoestrings, it is because it is indeed held together by shoestrings.)

    Thus, the information that you will receive will hardly be usable, and even if you manage to collect it all, make sense out of it, and update the design document with it, by the time you are done, the deployed system may have already changed.

    As a result, it is generally impossible at any given time to know the actual technical design of a deployed software system with any degree of certainty.

    This is a very sorry state of affairs for the entire software industry to be in.

    The Solution, in summary

    In order to ensure that the technical designs of software systems describe fact and not mere intention, we need:

    • A new type of design document which is innately deployable.
    • A toolset which facilitates:
      • Creation of such documents graphically.
      • Deployment of systems according to their designs, with the press of a button.

    By deploying a system from its own design document, we guarantee that it exactly matches its design.

    For the design document to be deployable, the following must be true:

    • The design must consist of well-defined components.
    • Components must be interconnectable in well-defined ways.
    • Components must be capable of deploying instances of themselves.
    • All components necessary for the deployed system to function must be present in the design.

    A component is well-defined if it implements a certain design protocol. The design protocol describes a set of characteristics of a component, and is capable of deploying a runnable instance of the component according to information present in the design. Characteristics of a component include the type-name of the component, a list of design-time properties that it may have, and a list of connections that it supports.

    Well-defined interconnectivity is achieved by means of identifying connections as either inputs or outputs, and establishing a strict set of rules that govern how connections can be made.

    • An input of a component makes some functionality of the component available for other components to use. Inputs can be of various different natures; for example, an input can be a programmatic interface in a particular programming language, or a custom REST API served over HTTP.
    • An output of a component represents a reference to an input somewhere else in the design, so that the component can use the functionality provided by that input.
    • Each connection is of a certain type. In the case of programmatic interfaces, the type of the connection is the type of the interface. In the case of REST APIs, the type of the connection is the specification (schema) of that particular REST API, or more likely, the name by which the schema is known.
    • An input can be connected to an output only if they are both of the same nature and same type.
    Any pre-existing or yet-to-be-built conventional runtime module can be turned into a well-defined component which is suitable for incorporation into a technical design by creating (and registering with the design toolset) a special design companion module which implements the design protocol on behalf of the runtime module.

    The requirement for all necessary components to be present in the design can be satisfied by requiring that every output must be connected to an input, and every input must have at least one output connected to it. 

    Some software systems are so complex that expressing them in a single technical design may be inconvenient to the point of being unworkable. To solve this issue, we introduce containers. A container encapsulates an entire separately-editable design, and exposes some of the inputs and outputs of the nested design as inputs and outputs of its own. Thus, containers are capable of abstracting away entire sections of a design, and they can be nested indefinitely, so they facilitate designs of any scale and complexity.

    The solution, in detail

    Work in progress.



    No comments:

    Post a Comment