Towards Authoritative Software Design

Abstract

This paper examines the long-standing need within the software engineering discipline for technical design documents that are authoritative. A design document is authoritative if there exist technical means of materializing it as a working system, thus guaranteeing that the end result is indeed as described by the design. We look at existing solutions in other engineering disciplines, we notice the scarcity and inadequacy of solutions for software, we look into the difficulties of solving the problem for software, and we conclude with realizations on what it would take to come up with a solution that works. 

(Useful pre-reading: About these papers)

Design in other engineering disciplines

In long-established engineering disciplines such as mechanical, electrical, civil, etc., for several decades now, design work has been facilitated by Computer-Aided Design (CAD) tools (W) and Computer-Aided Engineering (CAE) tools (W). 

Civil engineering has been transformed by CAD tools, which enable the efficient design of complicated three-dimensional structures with high precision. Detailed information about materials, dimensions, and tolerances can be specified, based on which the tools perform various forms of analysis to detect potential problems and to ensure that specifications and safety regulations are met. For example, the tools can perform simulations to verify the structural integrity of a building under static loads, winds, earthquakes, etc. Based on the results, the engineers can (again very efficiently) edit the design to optimize it, and repeat the analysis as necessary. Finally, the design is sent to the builders, and it is detailed enough so that it can be built exactly as specified. (And it better be built exactly as specified, or else the safety regulations may not be met.)

In electronic engineering, which is the discipline from which most parallels can be drawn to software engineering, virtually all design work since the 1980s is being done using Electronic Design Automation (EDA) / Electronic Computer-Aided Design (ECAD) tools (W). These tools have revolutionized the discipline by using a standardized notation to design, modify, and optimize products, as well as to manufacture them.

• The standardized notation used in electronic schematic diagrams is understood by all electronic engineers: a new hire begins their first day at work by studying the schematics, and before the end of the day they are often able to pick up the soldering iron and start doing productive work. Contrast this with software engineering, where a new hire usually cannot be productive before spending weeks studying source code and documentation, and having numerous knowledge transfer meetings with senior engineers who already know the system.
• Most importantly, ECAD tools minimize human error by helping to bridge the gap from the physical world to the design, and from the design back to the physical world. The tools have built-in knowledge of electronic components available for inclusion in a design, and electronic manufacturing has long ago advanced to the point where an electronic design document can be turned into a functioning circuit board with nearly zero human intervention. Thus, electronic design documents today are authoritative: the end products are accurately described by their designs.

Unfortunately, thus far, the software engineering discipline has been following a different path, where design documents are scarce, and authoritative design documents are completely non-existent. This situation has been allowed to go on for so long, largely because in software engineering we already have a certain other kind of document which is authoritative, and this is the source code.

However, source code is an implementation, or at best a detailed technical description, but not a design. To say that the design of a several-million-source-code-line software system is the several million lines of source code that make up that system is equivalent to saying that the design of a jet airliner is a list of the several million parts that make up that jet airliner, down to the individual nut and bolt. A design is supposed to list operative components, and to show how they are interconnected, but not to delve past the level of detail of the component. Unfortunately, we do not have that in software engineering, at least not in an authoritative form.

Conventional software design

The usual means by which we describe the design of software systems today are:

• Pen and paper
• Whiteboard
• General-purpose shape-drawing applications

or, in the best case,

• Box-and-arrow drawing applications that are smart enough to keep the arrows connected as we drag the boxes around the canvas.

Unfortunately, even the box-and-arrow applications have no notion of what the boxes and the arrows stand for. Thus, conventional means of software design lack technical means of accomplishing any of the following:

• Informing the design with software components available for inclusion.
• Restricting the design to only valid ways of interconnecting the components.
• Materializing the design into a running software system.

The above limitations have the following serious consequences:

• Designs created via conventional means tend to have arbitrary notation, arbitrary levels of abstraction, and even arbitrary semantics, making them works of art rather than works of engineering: they are nothing but mere suggestive sketches of the vague wishes of the architects, bearing no necessary relationship to reality.

• Software engineers and operations engineers do somehow manage to build working systems out of unworkable designs, but in doing so they engage in improvisation, thus creating systems whose structure is defined not by the design document, but instead by the source code, and various scripts and configuration files scattered all over the place.

Ideally, the technical design document should be the single authoritative source of truth for the structure of the system that it describes, but conventional means of software design do not accomplish this. Thus, software design as conventionally practiced is non-authoritative. 

It is a great paradox of our times that the software engineering discipline is virtually bereft of authoritative design tools, when such tools are the bread and butter of the long-established engineering disciplines. 

Prior art

Through the decades, plenty of tools and methodologies have been developed with the aim of aiding the software architecture and design process. A common pattern among them is that they try to make some aspect of development more visual rather than textual. However, none of them comes anywhere even close to being a useful solution for authoritative design. They invariably fall into one of the following categories: 

• Implementation tools (For example: Visual Programming Languages (W) like Snap!, Scratch, EduBlocks, Blockly, etc.,) - They are indeed visual, and they do indeed produce runnable software, but their structure is equivalent to code, so they express implementations rather than designs.  
• Reverse-engineering tools (For example: class diagrams, dependency diagrams, call trees, etc) - They are restricted to the visualization, exploration, and documentation, but not the editing of existing software, and certainly not the design of new software.
• Niche tools (For example: Web Services Description Language (WSDL) (W), Business Process Execution Language (BPEL) (W), Specification and Description Language (SDL) (W), etc.) - They are exclusively focused on specific domains such as web services, business processes, distributed systems, etc.
• User-friendly whiteboards  (For example: Microsoft Visio (W), Modelling Languages (W) such as Unified Modeling Language (UML) (W), The C4 model (W), various Specification Languages (W), etc.) - They are restricted to modelling, so they bear no necessary relationship to reality. Some of them impose some limitations on what is supposed to be included in a design, but these limitations exist only in theory, because they are not enforced by any technical means.
• Look ma, no code tools (For example: Rapid Application Development (RAD) tools (W),  No-Code Development Platforms (NCDPs) (W), and Low-Code Development Platforms (LCDPs) (W)) - They impose the use of a massive vendor-specific platform; they impose limitations on what you can do; they do not scale; they are aimed at non-programmers, allowing easy creation of simple user-interface-centric applications to quickly (and usually haphazardly) meet specific narrow business needs.

For a more detailed look at prior art, see michael.gr - The state of affairs in computer-aided software design.

A detailed look at the problem

Software design as conventionally practiced suffers from the following shortcomings:

• Designs often include elements that are not well-defined in engineering terms.

You see this with design documents containing supposedly technical but actually quite nebulous entities such as a data store here, a messaging backbone there, or a remote server over there. None of these entities is concrete enough and unambiguous enough to be suitable for inclusion in a technical design document.

• Designs often include elements that are completely outside the realm of engineering.

You see this with design documents containing human figures representing users, pictures of money representing payments, etc. The presence of such items in a software design usually indicates a confusion between what is a technical design and what is a functional specification.

• Designs often include elements from wrong levels of abstraction.

You see this with designs that mix software components with flowcharts, state diagrams, etc. Notwithstanding the fact that these are also boxes connected with arrows, they represent decision-making logic, which is an implementation detail of the component that contains that logic, and as such they have no place in a software design document.

You also see this with designs that confuse interfaces with other concepts, such as ownership, containment, inheritance, data flow, etc.

• Designs are often expressed at an unworkably high level of abstraction.

The level of abstraction most commonly chosen by software architects is that of a block diagram, which might be suitable for abstract architectural work, but it does not contain enough detail to guarantee the technical feasibility of the proposed design.

The level of abstraction necessary in order to guarantee feasibility is that of the component diagram, which shows individual components, and connections between them on specific interfaces.

Unfortunately, since designs are distanced from the engineering entities that they are dealing with, they do not have enough factual information at their disposal to be able to delve into such a level of detail as necessary for a component diagram.

• Designs are not informed with what elements are available for incorporation.

The medium on which conventional designs are expressed provides no means of establishing or enforcing a correspondence between a box as it appears in the design, and the actual provisionable, instantiatable, and runnable software module that it represents. This can be okay in the case of modules that have not been developed yet, but more often than not a design intends to incorporate existing, ready-made, reusable modules. In the absence of any technical means for informing the design about existing modules, the design inescapably represents hypotheses and assumptions rather than fact.

• Designs often prescribe invalid combinations of elements.

The ways in which conventional designs intend to interconnect components do not necessarily match the ways in which the components can actually be interconnected.

• A conventional design may assume that a certain component exposes a particular interface while in fact the component does not expose such an interface.
• A conventional design may prescribe a connection between two components on a certain interface, while in fact the components cannot be connected because the type of the interface exposed by one component is not a valid match for the type of the interface required by the other component.

• Designs fail to capture dynamic aspects of software systems.

Conventional means of software design often lack the ability to express dynamic constructs such as:

• Plurality: Multiple instantiation of a certain module, where the number of instances is decided at runtime.
• Polymorphism: Fulfilling a certain role by instantiating one of several modules capable of fulfilling that role, where the choice of which module to instantiate is made at runtime.

• Designs are often incomplete.

A design may incorporate a component which needs to invoke a certain interface in order to get its job done, but omit incorporating a component implementing that interface. In such cases, the software system cannot be deployed as designed, and yet there is nothing to prevent the architects from proclaiming the design as complete.

The above list of problems results from the lack of technical means of informing the design with what is available and restricting it to what is possible. Correspondingly, the process of building and deploying software systems is also not informed, nor restricted, via any technical means, by the design. The end-result of all this is the following:

• Software systems do not match their designs.

Even if the technical design happens to describe a software system that can actually be built as described, (which is virtually never the case,) there are no technological safeguards to guarantee that it will, so the software engineers and the operations engineers are free to build and deploy a system that bears little or no relationship to the design. Neither the architects, nor the management, have any way of knowing.

• Software systems diverge from their designs over time.

Even if the technical design initially matches the deployed software system, (which, again, is virtually never the case,) the system is bound to evolve. The design should ideally evolve in tandem, but it rarely does, again because there are no technological safeguards to enforce this: the engineers are free to modify and redeploy the system without updating the design document, and in fact they almost always do, because they see no point in double book-keeping. Thus, over time, the design bears even less relationship to reality.

If, due to the above reasons, you suspect that your technical design document is counterfactual, and you would like to know exactly what it is that you have actually deployed and running out there, you have to begin by asking questions to the software engineers and the operations engineers.

In order to answer your questions, the engineers will in turn have to examine source code, version control histories, build scripts, configuration files, server provisioning scripts, and launch scripts, because the truth is scattered in all those places. In some cases they might even have to try and remember specific commands that were once typed on a terminal to bring the system to life. If this sounds a bit like it is held together by shoestrings, it is because it is in fact held together by shoestrings.

Thus, the information that you will receive will hardly be usable, and even if you manage to collect it all, make sense out of it, and update the design document with it, by the time you are done, the deployed system may have already changed, which means that your design document is already obsolete.

As a result, it is generally impossible at any given moment to know the actual technical design of virtually any non-trivial software system in existence.

This is a very sorry state of affairs for the entire software industry to be in.

Towards a solution

If we consider all the previously listed problems that plague software design as conventionally practiced, and if we look at how the corresponding problems have been solved in long-established engineering disciplines, we inescapably arrive at the following realization:

The design of a software system can only be said to accurately describe the actual running system if there exist technical means of creating the system directly from the design, with no human intervention.

In order to automatically create a software system from its design, the design must be semantically valid. This brings us to a second realization:

The semantic validity of a software design can only be guaranteed if there exist technical means of informing the design with components available for incorporation and restricting the design to only valid ways of interconnecting them.

The above statements define a software design as authoritative.

Any attempt to introduce authoritative design documents in the software engineering discipline would necessarily have to borrow concepts from the electronic engineering discipline. This means that the solution must lie within the realm of Component-Based Software Engineering (CBSE) (W), where systems consist of well-defined components, connectable via specific interfaces, using well-defined connectivity rules.

What we need is a toolset that implements such a paradigm for software. The toolset must have knowledge of available components, knowledge of the interfaces exposed by each component, and rules specifying valid ways of connecting those interfaces. The toolset must then be capable of materializing the design into a running software system.

The toolset must not repeat the mistakes and suffer from the drawbacks of previous attempts at component-based software engineering. Thus, the toolset must meet the following goals:

• Facilitate any programming language.

By this we do not mean that it should be possible to freely mix C++ components with Java components; what we mean is that it should be possible to express in one place a C++ subsystem consisting of C++ components interconnected via C++ interfaces, and in another place a Java subsystem consisting of Java components interconnected via Java interfaces, and at a higher scope to have each of these subsystems represented as an opaque component, where connections between the two components are made via language-agnostic interfaces (e.g. REST) or cross-language interfaces (e.g. JNI, JNA, etc.)

• Facilitate any scale, from embedded systems to network clouds.

This means that the nature of a component and the nature of an interface must not be restricted, so that they can be realized in different ways at different levels of scale. For example, at the embedded/C++ level of scale, a component might be a single C++ class exposing C++ interfaces, whereas at the internet level of scale a component is likely to be a (physical or virtualized) network host exposing TCP interfaces.

• Guarantee type-safety at any scale.

Type safety can be carried across different levels of scale by means of parametric polymorphism (generic interfaces.) For example, a type-safe interface between a client and a server can be described with a construct like Tcp<Rest<AcmeShopping>> which stands for a TCP connection through which we are exchanging REST transactions according to a schema which corresponds to a programmatic interface called "AcmeShopping".

• Require no extra baggage.

Components should not be required to include a lot of extra overhead to facilitate their inclusion in a design. Especially at the embedded level, components should ideally include zero overhead.

This means that a C++ class which accepts as constructor parameters interfaces to invoke and exposes interfaces for invocation by virtue of simply implementing them should ideally be usable in a design as-is.

The extra functionality necessary for representing the component within a design, provisioning a target environment with it, instantiating it, and wiring it should be provided by a separate companion module, which acts as a plugin to the design toolset, and exists only during design-time and deployment-time, but not during run-time.

• Support automatic deployment.

The toolset must support components representing various different kinds of environments such as network hosts, isolated devices, operating systems, virtual machines, etc. Each of these components must be configurable with everything necessary in order to provision a certain environment with a corresponding part of the design. Essentially, the toolset must be capable of deploying a software system of arbitrary complexity to a production environment of arbitrary complexity, and it must be capable of doing so with no human intervention other than the pressing of a "Deploy" button.

• Support automatic wiring.

Once an execution environment has been provisioned with software components, the components must be wired together in order to start running. Traditionally, the wiring of freshly instantiated components is done by carefully hand-crafted code, to account for circular dependency issues between components. If we are to have fully automated deployment, the wiring cannot be done by hand-crafted code anymore; it must be automated, therefore it must be standardized. This in turn means that certain connectivity rules are necessary in order to guarantee that software designs do not suffer from arbitrary dependency issues that require custom handling. For more on this, see michael.gr - Call Graph Acyclicity.

• Support iterative development.

Once a system has been designed, coded, and deployed, it is a fact of life that it will keep evolving. The design toolset must seamlessly allow modifying the code, or the design, or both, and re-deploying.

• Facilitate incremental adoption.

It should be possible to express, via an authoritative design document, the structure of a small subsystem within a larger system whose structure has not (yet) been expressed authoritatively.

• In systems of medium scale and above, this may be handled by making the core deployment and wiring engine of the toolset available on demand, during runtime, to quickly materialize a small subsystem within the larger system.
• In embedded-scale systems, it should be possible to utilize code generation to do the instantiation and the wiring, so as to avoid having the core engine present at runtime.

• Utilize a text-based document format.

In software we make heavy use of version control systems, which work best with text files, so the design documents must be text-based. The text format would essentially be a system description language, so it must be programmer-friendly in order to facilitate editing using a text editor or an IDE. A graphical design tool would read text of this language into data structures, allow the visual editing of such data structures, and save them back as text.

• Facilitate Hierarchical System Composition.

Some systems are so complex that expressing them in a single diagram may be inconvenient to the point of being unworkable. To solve this issue, container components are necessary, which encapsulate entire separately-editable diagrams, and expose some of the interfaces of the components in the nested diagram as interfaces of their own. Thus, containers can be used to abstract away entire sub-designs into opaque black-boxes within greater designs. It goes without saying that container components must allow unlimited nesting.

• Facilitate dynamic software systems.

Every non-trivial system has the ability to vary, at runtime, the number of instances of some components in response to changing computation needs, and to choose to instantiate different types of components to handle different needs. Therefore, a toolset aiming to be capable of expressing any kind of design must be capable of expressing, at the very minimum, the following dynamic constructs:

• Plurality: Multiple instantiation of a certain component, where the number of instances is decided at runtime.

• Polymorphism: Fulfilling a certain role by instantiating one of several different types of components capable of fulfilling that role, where the choice of which component type to instantiate is made at runtime.

• Polymorphic plurality: A combination of the previous two: A runtime-variable array of components where each component can be of a different, runtime-decidable type.

• Facilitate multiple alternative configurations.

In virtually every software development endeavor there is a core system design which is materialized in a number of variations to cover different needs. For example, a debug build vs. a release build; a testing build vs. a production build; a build with or without instrumentation; a build with hardware emulation vs. a build targeting the actual hardware; etc. Therefore, the toolset must facilitate the expression of alternative configurations so that each configuration can be defined authoritatively.

• Be accessible and attractive.

The extent and speed by which a new software development technology is adopted greatly depends on how accessible and attractive the technology is. To this end:

• The core toolset must be free and open source software. (Profit may be made from additional, optional tools, such as a visual editor.) This also means that the toolset must be a cross-platform, installable software package rather than a cloud offering.
• A clear distance must be kept from unattractive technologies like UML, XML, etc.
• The literature around the toolset must avoid wooden language and alienating terms such as "enterprise architecture", "standards committee", "industry specifications consortium", etc.

Scratch (please ignore)

How to Become a Great Software Architect • Eberhard Wolff • GOTO 2019

Ivory tower architecture, detachment from reality, no impact

Relationship between architect and self-organizing agile team

Hopelessness of enforcing the architecture

Architects who also code vs. who do not code

Software Architecture vs Code • Simon Brown • GOTO 2014

UML is dead

Multitude of examples of preposterous approaches to architecture diagrams

Mismatch between coding and architecture

Need for a programming language that has components as a first-class thing

Definition of component: a bunch of related stuff with a nice clean interface (something you can substitute out)

The C4 model (System Context, Containers, Components, Classes)

"I am working on a tool to [shoot?] through the code base, extract components or services or microservices, and then automatically draw the picture based on relationships and dependencies". (This is reverse engineering, not design.)

Organ Transplants

Need to address John D. Cook's concern that software reuse is more like an organ transplant than snapping together lego blocks. If the interfaces of a module are compatible with existing interfaces in our system, then the module can snap into our system like a lego block. If not, then it is first and foremost an architectural gamble whether it can be shoe-horned into our system; if it can, then the point of software design is to show the module in the schematic diagram, and to also show the adaptor components necessary for interfacing between it and the rest of the system. In this sense, yes, it is like an organ transplant, and yes, authoritative software design accommodates organ transplants.

Metadata

Need to allow the addition of arbitrary metadata to the design, and then to either have different metadata-driven views (viewpoints, perspectives) of the design, or to allow separate applications to read the design document, allow editing of the meta-data, and present the design in different ways. This can facilitate, for example:

• Storing Architectural Decision Records (ADRs) and other kinds of documentation of interest to different stakeholders. For more on ADRs see https://thinkrelevance.com/blog/2011/11/15/documenting-architecture-decisions -> link is dead, new link: https://cognitect.com/blog/2011/11/15/documenting-architecture-decisions by Michael Nygard. For ADRs as a vehicle of engagement between architects and developers instead of documentation, see Mark Richards - The Intersection of Architecture and Implementation - DDD Europe.
• Recording various metrics, either manually or automatically.
• Manually:
• Estimates about technical debt per component
• Threat modelling
• Compliance points of interest
• Automatically:
• Test code coverage percentages
• Performance statistics
• Frequency of change statistics (commits)
• Keeping track of Team Architecture
• Specifying and visualizing which development teams are responsible for building and/or maintaining different parts of the architecture. (See the concept of "Team Architecture" in "Practical (a.k.a. Actually Useful) Architecture" by Stefan Tilkov, GOTO 2023, section 2, "Explicitly architect your team setup") -- Also related term: Team Topologies

Generative Artificial Intelligence

The role of Authoritative Software Design in the age of GenAI-assisted programming

Interface Language

The toolset requires a common interface definition language. Interface methods have arguments, which are of specific data types, so this implies a data type definition language.

Postmortem architecture

It has been a recurring pattern in my career that someone would pay me to create something, and once I was done creating it, they would ask me to produce an architecture document showing the design of what I had just created. I have also seen prominent software architects admitting in conferences that they are sometimes brought in by companies to document already existing systems in order to have a basis upon which to further build. (For example, in "Practical (a.k.a. Actually Useful) Architecture" by Stefan Tilkov, GOTO 2023, section 7, "Balance prescriptive vs. descriptive architecture".) This is deplorable.

Useless architecture

"Enterprise architects in large companies who build these giant diagram walls that nobody ever looks at". This leads to the false conclusion that up-front architecture is useless, but this inescapably implies that only postmortem architecture is viable. See "Practical (a.k.a. Actually Useful) Architecture" by Stefan Tilkov, GOTO 2023, section 7, "Balance prescriptive vs. descriptive architecture".) This also creates the ugly "ivory tower architecture" impression to non-architects.

Layers (Configurations)

In software engineering we often express different configurations of our systems to suit different needs; for example, we may have a configuration for production, and one or more different configurations for testing. The bulk of the components and the wires between them exist in all configurations, but some configurations have additional sets of components and slightly different wiring.

To facilitate this, the toolset must support design layers, similar to drawing layers found in drawing applications like Photoshop. Design layers are unrelated to architectural layers that comprise layered architectures, although people will, quite likely, find ways to represent architectural layers using design layers.

The details of how layers are going to work in order to support configurations are to be decided, but one preliminary idea is to have one or more base layers where the bulk of the components are laid out, and a few mutually exclusive configuration layers on top of them. A configuration layer combines with one or more base layers to form a complete system, and is deployable, whereas base layers do not describe complete systems and are therefore not deployable by themselves.

Views (Viewpoints / Perspectives)

In a detailed, authoritative technical software design, the key components of a system do not necessarily correspond to the top-level components of the system. For example, an entire RDBMS might be hidden deep inside some "Persistent Storage" component, or an entire LLM might be hidden inside some "Property Assessment" component. Thus, the key components might be difficult to see at first glance, due to being lost in the nooks and crannies of the diagrams, and the connections between them might be obscured as they pass through non-key components such as adaptors, containers, etc. To address this problem, a key component overview of a system may at times be necessary.

Such a key component overview might correspond to what Dave Farley calls the Whiteboard model or the Tourist Map View of a system, and is related to what Simon Brown calls the Context View of a system. (See Modern Software Engineering: Software Architecture Principles From 5 Leading Experts)

It is entirely possible, of course, to set aside for a moment the authoritative technical software design of a system, and create a key component overview of that system on an actual whiteboard. (Duh!) It would be nice, however, to have the option of automatically obtaining the key component overview directly from the technical design, since we have already done all the detailed work, and since the detailed work is authoritative.

How (and if) this is going to be accomplished is still to be decided, but one preliminary idea is to introduce a concept of Views, which are similar to layers but allow us to re-position components in them. In other words, whereas a component exists only on a specific layer and has a fixed position within that layer, the same component may also be added to a view, where it can be given a different position, specific to that view. Then, we need to be able to add pseudo-wires to the view, showing how the key components are connected, and the toolset should be able to validate those pseudo-wires based on whether there exist corresponding wires in the underlying detailed design, allowing for the fact that wires in the detailed design may pass through components that have been omitted in the key component overview. 

With views it may even be possible to enrich a diagram with various pseudo-components such as human figures, to satisfy the need of some people to see human figures in software designs. Such pseudo-components would exist only for visualization purposes and would not not correspond to any actual software elements.

Another possible use of views / viewpoints / perspectives is to visualize data flow. Every interface can, in principle, be associated with a direction of data flow with respect to the direction of invocation: when invoked, some interfaces only pull data, some only push data, and some perform bi-directional exchange of data. By annotating every interface with its data flow direction, we can request the tooling to show us connections between components in terms of data flow instead of control flow.

Reducing clutter

Viae

Wires traveling long distances within a diagram may cause clutter, especially if these wires represent ubiquitous cross-cutting concerns. For this reason, the concept of "via" will be borrowed from electronic design. A via is a named circle into which a wire may terminate and thus vanish from view; all viae with the same name are implicitly connected without having to show the wires between them. A via is strongly typed like any pin; when the first pin is wired to a via, the via implicitly takes the type of that pin.

Ribbons

Sometimes there may be multiple parallel wires that travel over long distances on a diagram. Some of them might even go in opposite directions. To reduce clutter, it will be possible to group such wires together in ribbons, which will be drawn as two parallel hairlines with a slanted hash between them. Ribbons run between connectors. Connectors are pseudo-components, meaning that they only exist in the design and have no counterpart in code. Connectors break down the ribbon into individual wires. The shape of connectors is to be determined, but it will probably be borrowed from electronic design. Ribbons can also be routed in and out of viae. Ribbon viae will be bigger than single-wire viae.

Styling rules and conventions.

To ensure that every engineer can easily read any diagram created by any other engineer, the toolset will aim to standardize the notation used in software design documents, the same way that electronic schematic diagrams follow a standard notation which is universally understood by all electronic engineers.

The details are to be decided, but some preliminary ideas are as follows:

Diagrams shall be drawn using nothing but monochrome (black & white) shapes.

The use of color and semi-transparency in diagrams leaves too many possibilities open for distraction and for representing things in non-standard ways. Therefore, color and transparency will be reserved for signifying selection and different layers in the visual editor.

The majority of software developers are male, and a significant percentage of the male population is color-blind; therefore color may only be used for aesthetics; color may not be used to differentiate between technical concepts that matter.

Nonetheless, it will probably be possible to present a design in a colorful way by placing different groups of components in different layers, choosing a different color for each layer, and requesting the tool to display all layers simultaneously.

The default and recommended shape for every component type is a simple rectangle, but some component types may have a non-rectangular shape if there is merit in this. The name and type of each component will be visible in the middle of the shape of the component, and for that to work component shapes will be required to be convex. Some conventions may emerge; for example, converters may look like triangles, adapters may look like AND gates, persistence stores may look like cylinders, and various other types of components may have shapes that bear some remote resemblance to some flowchart symbols. (Even though software designs are most definitely not flowcharts.)

Outputs will look like triangular arrows pointing out of a component, inputs will look like triangular arrows pointing into a component. By default, the name and type of each pin will be visible. Names and types of pins will be drawn outside the shape of the component.

(Need to show an illustration here.)

Design conventions

The preferred placement of pins on the perimeter of a component shall be:

• Inputs on the left and top edges
• Outputs on the right and bottom edges

The general intent of pin placement shall be:

• General-purpose and cross-cutting concern interfaces:
• inputs along the left edge
• outputs along the right edge.
• Application-logic interfaces:
• inputs along the top edge.
• outputs along the bottom edge.

(This is reversely analogous to electronic design, where signals flow from left to right and voltages from top to bottom.)

Wires between inputs and outputs will be hairlines.

Wires may bend only in right angles. When two wires cross, this means that they are completely separate from each other. When multiple outputs converge into a singe input, a thick dot will indicate that the wires are connected.

At various points along a wire there may be tiny skinny arrows to remind us of the direction of the wire (always from the output to the input.)

Since wires for cross-cutting concerns will usually be horizontal, and since they will often be drawn from pins straight into viae, diagrams are likely to grow more vertically than horizontally; in other words, diagrams will tend to be tall and narrow, and this is in line with our prevalent habit of scrolling through content vertically.

Technical software design is NOT architecture

• Technical software design is not concerned with discovering, recording, and analyzing requirements, quality attributes, constraints, risks, trade-offs.
• It is not about analyzing the "-ilities" of the system
• It will not help you make fundamental decisions about technologies.
• It will not prevent you from jumping head-first into ephemeral technical fashions, engaging in C.V.-driven development, buzzword bingo, etc.
• Technical software design does not tell you how to come up with a strategy for meeting the requirements. That is pure architectural work. Once the architect has decided how the requirements will be met, technical software design allows the architect to describe the structure of the software system that will meet the requirements.
• At most, technical software design allows the architect to document the decisions that were made by means of Architectural Decision Records.
• "The diagrams show you the outcome of the decision making process". -- Simon Brown (See Simon Brown - The lost art of software design - Devoxx Greece 2024)

Minimum Viable Architecture, Evolutionary Architecture, Continuous Architecture

"Continuous Architecture" book by Murat Erder, Pierre Pureur

Continuous Architecture • Murat Erder • GOTO 2016

Big architectural decisions made too early lead to brittle software designs. If we are to start with a minimum viable architecture, then obviously, we will be extending it as we go along. The Unit of Work of an Architect is an Architectural Decision.

Continuous Architecture in Practice Part 1/2 • Eoin Woods & Simon Brown • GOTO 2021

Continuous Architecture in Practice Part 2/2 • Eoin Woods & Simon Brown • GOTO 2021

The drive to do architectural work in almost every sprint.

The role of the chief architect is not to make all architectural decisions, but to empower other engineers to make good architectural decisions. This necessitates collaborative design work, while the design document remains authoritative at every step along the way.

The toolset should ideally be capable of integrating with git so as to look at the revision history of the design document and help visualize changes introduced by individual commits over time.

The "Waterfall" approach, introduced in the 70's, called for "big design up front". The Agile movement reconsidered this, along with many other things, and decided to do away with big design up front. The Agile manifesto did not call for absolutely no design up front, but that is how people interpreted it. This had detrimental consequences. (See Simon Brown • Software Architecture for Developers • YOW! 2017)

"Architecture represents the significant design decisions that shape a system, where significant is measured by cost of change." -- Grady Brooch. In other words, architecture deals with the kind of stuff that it is painful to change later. This definition notwithstanding, Dave Farley adds that architecture has to be an evolutionary process (see What Software Architecture Should Look Like • GOTO 2022) and that Architecture is a moving snapshot of our current understanding of the system. It allows us to play "what if" games in our minds or on the whiteboard. Also: "Architecture is the stuff that we wish we got right at the start of the project." --Ralph Johnson. Therefore, architectural change is inevitable, which means that pain is inevitable. We need tools that ease the pain as much as possible. "Don't aim for perfection - aim for evolvability" -- Stefan Tilkov. (Also see Michael Nygard: "Architecture without an end state", mentioned by Stefan Tilkov.) "Big up-front design is dumb, no up-front design is dumber" -- Dave Thomas

Authoritative design vs. documentation

As a new engineer you are told to read documents, but the documents are not necessarily all in one place, some of them are out of date and therefore misleading, some are even completely deprecated but nobody told you to skip them, etc. Authoritative software design keeps all the design documentation in one place, it keeps it brief and to the point, and it guarantees that it is all pertinent because the end-system is actually built from the design.

Authoritative design vs. modelling

The critique most often heard about modelling methodologies and tools is that they represent loathsome double book-keeping, and thus constitute another layer of red tape which is preventing developers from getting things done. Authoritative software design eliminates this problem because the design document contains information which is necessary for building the system and is not encoded anywhere else: which components to instantiate, where to instantiate them, and exactly how to wire them together. This is all information that the programmers do not have to put down in code. Thus, the design document becomes an essential engineering instrument, subject to the same single book-keeping as the rest of the source code.

Authoritative design vs. ivory tower architecture

Authoritative software design democratizes the design process by empowering developers to modify the design on an as-needed basis. The design document is just another source file, kept under source control like all other source files. Any developer can check it out, modify it to suit their needs, and create a merge request for it. Architects can keep an eye on how the design evolves, and perhaps weigh-in on some decisions, by participating in merge requests. So, architects now become engineers like everyone else, instead of artisans.

Architecture Artifacts

Eoin Woods says that architecture artifacts must have an audience and a purpose, and must be minimal, usable, significant, indexed, and checked. (See Democratising Software Architecture • Eoin Woods • GOTO 2023) >

Audience: all software engineers / deployment teams

Purpose: to communicate the design, allow the continuous collaborative editing of the design, and even to build the system.

Minimal, Usable, Significant: these are guaranteed by the fact that the system is actually built from the design.

Indexed: Everything is in the design document.

Checked: The toolset guarantees the type safety of all interfaces between all components of the system at all levels of scale, and also guarantees that there are no unconnected pins.

Agility

Recognizing all the shortcomings of ivory tower Software Architecture, the Agile movement attempted to downplay the role of architecture. Developers following the agile advice tend to have brief architecture meetings in which they produce nothing more sophisticated than whiteboard drawings. Then, they usually forget about them. Even if they do take a photo of the whiteboard before erasing it, they usually never look at it again. (Often, they cannot even find it.) (See Simon Brown & Stefan Tilkov • Part 1/2 • GOTO 2021 • 12:38) (See Simon Brown • The Lost Art of Software Design • YOW! 2019, the entire talk, ignoring the small part about C4, or Simon Brown - The lost art of software design - Devoxx Greece 2024 which is free from C4 stuff.)

Authoritative software design takes Software Architecture out of the realm of ivory towers and back into the realm of software engineering. Authoritative software design ensures architecture is relevant because the design document now becomes the technical means by which the software system is materialized.

Why architecture is important

"If you are going to try and move really fast, change your system all the time, have loads of parallel activity going from all your different service teams, pushing stuff into production every couple of days, you better have a pretty good architecture to support that." -- Eoin Woods (See Continuous Architecture in Practice Part 2/2 • Eoin Woods & Simon Brown • GOTO 2021)

Guidance

Simon Brown says that the biggest thing he does not like about UML is that it gives you no assistance. (See Simon Brown & Stefan Tilkov • GOTO 2021 • Part 2/2 • 12:37) In other words, you fire up a UML tool and a blank canvas appears, with nothing to guide you as to what you should do next, or even what you could do next. This leaves you with nothing but writer's block. With authoritative software design, you would begin with the topmost scope of your system; so, if you are building a system that spans the internet, you would drag the "Internet Scope" component into the canvas, and open it up to see what it has to offer, which would be various kinds of network hosts that you can place on the canvas, configure, and flesh-out; if you are building a Microsoft Windows Desktop application, you would drag the "Microsoft Windows Environment" scope into the canvas, and open it up to see what kinds of components you can add inside it, and so on.

Reverse engineering

Everyone who has ever tried to use reverse-engineering tools to create a design diagram from their code base has realized the major problem that all these tools suffer from: the diagram is a complete mess, even though you know that neither the code base, nor the architecture that it embodies, is a mess. Components are randomly scattered around the canvas with no logic or intent, and a myriad lines randomly crisscross all over the place giving an appearance of complete chaos. (See Simon Brown & Stefan Tilkov • GOTO 2021 • Part 2/2 15:29)

This is happening because the reverse-engineering tools have no concept of the intended structure of the system; they do not know which components are entry points, which components are adapters, which components play related roles and should therefore be grouped together, which components implement cross-cutting concerns, etc. Essentially, the tools do not know which components should conceptually be at the top of the diagram, which ones should conceptually be at the bottom, which ones are conceptually nested within others, and which ones are not even worth showing.

Therefore, reverse-engineering is not the answer. Software Architecture and by extension software design must be a forward-engineering process: We start with the design document, we implement any custom components as dictated by the diagram, and then we materialize the diagram into a working system.

Modelling

Most developers find modelling abhorrent because every single instance of modelling represents double book-keeping. Authoritative software design abolishes modelling and replaces it with engineering.

Collaboration

Operations folk do not see it as their job to do architecture, and architects do not see it as their job to write deployment scripts. Programmers do not see it as their job to do either. Authoritative software design allows the design document to be edited by multiple different people, by making it a text document, so that it can be versioned like a source file. Thus, architects can do their job, operations folk can do their job, and programmers can do their job.

Architectural Alignment

Mark Richards describes three kinds of integrity that is necessary between architecture and implementation: operational, structural, and constraints. (See Mark Richards - The Intersection of Architecture and Implementation - DDD Europe)

• Operational alignment refers to the software achieving metrics required by the architecture. Authoritative software design has nothing to say about this, and it is up to the developers to implement architecture fitness functions to monitor alignment and issue notifications if it is not achieved. He calls this process governance.
• Structural alignment refers to how well entities described by the design match corresponding implementation entities. Authoritative software design guarantees structural alignment, because it requires every component in the diagram to correspond to an actual runnable software component. A design consists only of components listed in the design, and only those components are deployed to the running system.
• Constraints alignment refers to how well the implementation entities follow various constraints prescribed by the design between architectural entities. Again, authoritative software design guarantees constraints alignment because it describes the interfaces through which components may communicate with each other and precludes any kind of communication that is not visible in the design document.

Note that both for structural alignment and constraints alignment, authoritative software design makes governance unnecessary: integrity is practically guaranteed by the toolset, so we never get to build the wrong thing and only later receive a notification that we did something wrong.

Mark Richards attributes misalignment to a great chasm between architecture and implementation. Authoritative software design largely eliminates that chasm.

Toolset complexity

The toolset is bound to be quite complex. To keep it feasible, the toolset must be architected as a relatively small core accepting plugins that add support for different environments and programming languages. The toolset should come with just a few such plugins, and it should be left up to other organizations to write additional plugins in order to support environments and programming languages that are of interest to them.

Software design has fallen out of grace in recent years. People do not see much value in it. This is because software design has been falsely associated with the "big design up front" dogma. One of the goals of the agile movement was to bring an end to that dogma, and rightfully so; however, a sizable portion of the industry threw out the baby with the bathwater, by going all the way to the other extreme and deciding to have no design at all. Software design is not about big design up front; sane engineering rules still apply: begin with a minimum viable design, start implementing, and from that moment on keep doing continuous design. The industry missed this approach because the tooling available was not suitable for continuous design. Authoritative software design aims to fix this.

"You cannot decide to not have an architecture; if you don't actively create it, be prepared to deal with the one that emerges" -- Stefan Tilkov (See Stefan Tilkov • "Good Enough" Architecture • GOTO 2019)

"There are two ways of getting it right -- the one is hiding the complexity, the other one is exposing some of the complexity but making it easy to deal with the complexity. I am in that camp very much because I have seen so many bad illusions where people try to hide things. I said no -- let's not hide this but let's make it easy to deal with." -- Gregor Hohpe (See Gregor Hohpe • Build Abstractions Not Illusions • YOW! 2023)

In Code is your partner in thought (Global Software Architecture Summit 2019), George Fairbanks talks about ways of interacting with code so that the code itself helps us do our job better. He says we are thinking using code, and communicating using code, and he gives ideas and examples on how thinking and communication can be improved merely by choosing more wisely how we write the code. This is true at the level of code, but what about the greater picture? What about the design? In the same talk, George Fairbanks talks about good vs. bad external representations, and how they facilitate or hinder thinking. The fact of the matter is that code itself is not a good external representation for the design of a system; for that, we need the design expressed as a diagram, not as code.