2017-07-17

On Full Stack Development

Q: How does a full stack developer sleep?
 A: They don't sleep, they REST.



Almost every single company delivering some kind of product on the web since the early 2010s seems to be looking to hire full-stack developers.  Full stack development is all the rage.  If you are an employer, you can't possibly be doing it right unless you are hiring full-stack developers to do the job.  If you are a programmer, you are not particularly employable unless you can work with the full stack.

It is almost as if software architectures requiring individual programmers to work on every single layer of the entire stack are a good thing. It is almost as if the the job market is full of programmers who are actually capable of working on every layer of the stack.  And it is almost as if they could possibly be any good at that. Or any part of it.

How did it all go so wrong?


The technological difficulties we are facing on the internet nowadays have to do with how to deliver a decent user experience on lame technologies like web browsers running javascript, how to write our server software in ways that allow us to massively scale the delivery of our services to huge numbers of concurrent users, and how to keep our systems running securely and reliably.  So far, there exists no product that you can buy from the shelf, add your own business logic to it, press a button, and have it deploy your services to the masses out there, satisfying the aforementioned requirements. There is a lot of work you have to do in-house, and this work involves a lot of experimentation, as well as the use of lots of other technologies that are in turn --some more, some less-- experiments.

The unfortunate combination of market forces and technological limitations results in the invention of unorthodox techniques out of sheer necessity, in order to dodge the roadblocks and keep delivering at least as much market value as the shop next door. They are not necessarily the best techniques, they do not even have to be particularly good, they just happen to have achieved some results somewhere, somehow, (high profile, cash-rich cases in the USA usually,) and then everyone over the entire world rushes to copy them without much thought, without even first examining whether they are necessary, or even applicable to their case. These techniques often involve some patently absurd choices, with immediately obvious drawbacks, so those who evangelize them resort to giving them nice and positive-sounding names in an attempt to downplay their deficiencies.

As Tertullian wrote, "prorsus credibile est, quia ineptum est." ("It is certainly believable, because it is absurd.")

The term "Full-Stack Developer" brings to mind programmers with a can-do attitude on their faces, who know how to do everything, shy away from nothing, are on top of their game, right? It is a very positive picture. The only flaw with the picture is that it flies in the face of Specialization and Division of Labor, which are some fairly old and tried inventions that have been the driving forces behind the Scientific Revolution and the Industrial Revolution. Pretty much the shape of the world as we know it today is owed to these two concepts, but modern techno-hipsters think they know better.

Full stack development requires work to be divided vertically rather than horizontally: each development team works on a separate feature of the product, which spans multiple software layers, instead of having different teams working on different layers. That's a very unfortunate direct consequence of the choice of the technology stack, where the modern trend is to have to put application code on the client, application code on the interface between client and server, and application code on (multiple layers of) the server. This usually means application programmers having to deal even with the transport protocol that the client and the server use to communicate with each other.  Adding insult to injury, the protocol of choice is REST, which is a particularly capricious protocol, requiring client and server code to be built with specific knowledge of the fact that they are communicating using REST, they cannot just use some abstraction where messages go into one end of a pipe, and come out of the other end of the pipe, or even better, an object is changed here, and the changes are automatically reflected there.


Due to these reasons, full stack development means that in order to add the tiniest new feature in the product, or to modify the slightest existing behavior of the product, you have to touch everything, from the abyssal depths of the persistence layer on the server to the wuthering heights of the cascading style sheets on the client, as well as every. single. layer. in-between. The job cannot be localized on a single layer, because the chosen technology stack employs no abstractions: it is application software all the way down.  And the job cannot be handed out to different developers with different specializations, because everything is so inextricably complexed together, that only someone with the entire picture in their mind can do the job.

Essentially, full stack development means "we have no frameworks and no abstractions; we are all goalkeepers, and every single one of us is playing all over the entire soccer field."

One might argue that this is not a problem, all you need to do is to hire lots and lots of bright people who will manage all this complexity. Unfortunately, this comes at a huge cost. That which used to require one team of developers to build, now requires five or ten teams. This in turn means lots of product management, which means that you have non-programmers prioritizing the programmers' work and telling programmers what to do and what not to do. This is how you get overwhelmed development departments trying to manage huge code bases that despite their efforts are turning into trash due to accumulated technical debt because management sees no customer value in fixing technical debt. From the user's perspective, this is how you get standing annoyances and bugs that remain on the product for months under the excuse that they are not blockers. When each team works on the same layers as every other team, this also means that each team is re-inventing every single wheel on its own. Surely any healthy company encourages communication between development teams, but communication is also tantamount to overhead, and there is still a lot that is not communicated, because nobody can communicate the entirety of their work experience every day.

And then there is the much more severe problem of security. Have you noticed how many high profile hacks are surfacing in the news every few days lately? Ever wondered why this is happening? It is happening because it is so easy!  When the technology stack requires that every single feature must have a client footprint, this means that each development team gets to work on their part of the client side, which in turn means that each team has to be experts in security. How familiar are your developers with the dozens upon dozens of well known attacks that are available to the bad guys on the client side? (https://www.owasp.org/index.php/Category:Attack) How familiar are they with just the top 10 of those attacks? (https://www.owasp.org/index.php/Top_10_2017-Top_10) Once a vulnerability is found, how easy is it to fix it?  When every single product feature has a footprint on the client, how many separate pieces of code need to be audited and fixed each time a potential threat is discovered?

The number one difficulty in our discipline is the management of complexity. Full stack development extends the scope of the complexity that we have to deal with in our daily work to include in it the complexity of multiple layers of the technology stack, and multiplies all that by the extra dimension of managing how these layers interact with one another. Asking developers to be full stack developers is underestimating the complexity of working on a single layer, let alone half a dozen layers. That's what happens when you throw away all of the accumulated wisdom of the discipline and think you are re-inventing everything from scratch.



No comments:

Post a Comment