This post is mirrored on my blog.

I started my morning off with Alan Kay's talk Programming and Scaling. There are several things I agree with and disagree with in that talk. It got me thinking about software as a whole, and my part to play in it.

Background

I have long been a fan of Jonathan Blow and Casey Muratori. Two talks in particular have been especially influential for me:

My other programming hero, John Carmack, has a different opinion of software, where things are generally pretty good. From his Twitter:

Just last night I finished a book (How to Take Over the World: Practical Schemes and Scientific Solutions for the Aspiring Supervillain) with a related data point that made me a bit sad. I remain a strident optimist, and there is a ton of objective data to back that position, but I do feel that broad pessimism is an unfortunate brake on progress.

Fiery indignation about a flaw, when directly engaged in building a solution is a fine path to progress, but "pessimism cheerleading" and self-flagellation have few upsides. Being an optimist, I feel that progress is going to win anyway, but I would rather win faster!

I can't find a direct quote about software, but I believe he would agree that he is optimistic about software as well. I have a hard time reconciling this, because he has created really amazing things and knows way more than I do about programming, but I still strongly feel we are not on the right track as an industry.

Conway's law

Casey Muratori's talk The Only Unbreakable Law is a must-watch. He talks about Conway's law, which is basically that organizations can only create things which replicate the organizational structure.

This is something I've had personal experience fighting in the game industry. Game studios frequently organize based on roles, which means there is a design team, a gameplay programming team, an art team, etc. Communication is naturally more expensive between teams, which leads to team power struggles, out-of-sync feature development, poor product coherency, and other issues.

For example, Design requests a feature which doesn't get implemented for weeks until Gameplay gets around to it. If the designer instead sat next to the gameplay engineer and brainstormed with them, from the very inception of the idea the engineer has been involved. This gives the engineer opportunities to suggest designs which uniquely leverage the technology, and gives the designer opportunities to have a more realistic picture of the feasibility of there ideas. (I have heard of design throwing out ideas thinking that they would be too hard to implement, but are actually technically simple, for example.) With the two people sitting together, the chances are high that the feature will in by the end of the day.

Methodologies like Scrum try to combat siloization by having cross-disciplinary teams--e.g. a designer, engineer, and artist all working closely together. Note, however, that you are still splitting a project into separate teams. If you have a Quests team and a Combat team, chances are lower that quests and combat will be as coherent with each other than if you had simply a "gameplay" team with no artificial divisions.

Open source development

The story gets even more complicated when you enter the free software world. Now, there aren't even teams, but frequently individual contributors, who both

  • don't have as close an understanding as the project's original developer
  • likely have a lower-bandwidth communication with the project's owners.

The FOSS world still naturally creates divisions here: there's the trusted core development team, and everyone else. This is a natural consequence of there being differing levels of quality, experience, and understanding between contributors.

At a more macro scale, you have teams of contributors working on single libraries or programs. There is rarely ever coherence between two programs, for example GIMP and Emacs. There's no one leading the overall system's direction. This has an advantage in the freedom it provides, but the sacrifice is increased system complexity, redundant work, and overall lower quality.

If you have N developers working on M different photo editors, you likely will end with M photo editors at 80% completion; if you have them all work on one editor, it seems you would end up with one 100% photo editor. However, even as I say this I know it's a fallacy, because having more people work on something does not imply higher quality--see both Brooks' Mythical Man Month and Conway's law again. I think this shows how natural that line of thinking is.

One could instead imagine the M different editors being different attempts to make something great, and then let natural selection take over. This seems like a reasonable away to approach innovation, but has a few problems:

  • People will make things without understanding the state of the art and the previous attempts, causing them to waste time retracing steps.
  • People can copy-cat programs and make things with only trivial innovations, causing programs to get stuck in local maxima. (Though evolutionarily, this would be the equivalent to mutation, which can end up a winning strategy in the very long term. Ideally, we would be able to do things faster thanks to our ability to think and react, but it's possible at the macro scale we can't beat out evolution. I'm out of my depth here.)
  • End-users likely don't care about there being ten different innovative attempts at editing photos. They just want one program that does everything they want. I don't believe this is a good counter-argument, however. People have different ways of thinking about and doing things, which necessitates different programs--there will never be a program that can please everybody. In fact, the more people you try to please, you either have to simplify the program (to appeal to beginner users; this is largely the strategy SaaS seems to take) or bloat the program with every feature you can think of.

A system created by ten people sitting in a room together is guaranteed to be more conceptually sound than one created by thousands. This doesn't necessarily mean the ten-person program will end up better, but the chances of all 10 people being on the same page are vastly higher than all those thousands.

I do not want my argument to be used as one against free software; I only want to point out that small, tight teams are likely to produce more coherent programs. I don't think the proprietary vs. free applies here, except that it seems easier to convince ten people to sit in the same room for a few years if you're paying them.

Local maxima

Systems can grow to the point where making a necessary change takes so long and is so costly that the change cannot happen without large rework. In order to escape local maxima, we must be willing to start over.

Over time, software seems to increase the amount of dependencies it has. For example, an abstraction library like SDL will add code for X11, then Wayland, then Win32, then Apple's Metal, etc. New technology is invented, which is a good thing, but the question becomes, well, what do we do with all the old stuff?

How Microsoft Lost the API War discusses exactly this with regard to Microsoft. He reaches the conclusion that the Web is the ultimate solution, which I refuse to accept, and which I believe end-users also dislike. We still install "apps", and we still care about latency, even if we don't know it; things are so bad it's easy to forget what good software feels like.

Maybe the existence of successful app markets is evidence that web developers have done a really terrible job, that no UI standardization severely limits app quality, or that the foundational web technology can never result in as good of experiences for users. Hey, maybe all these are true.

While I'm on Joel on Software, I have to reference his argument against ever re-writing code. I disagree with this essay when applied to the industry at large. It might make sense if you are a startup with a limited runway or a for-profit company, but this attitude in regard to overall industry health and technological innovation is extremely harmful. We must not only be willing to rewrite things, but encourage it! We should be telling every new graduate, "study the mistakes of the past, and build the new future!", not "don't ever start from scratch, just keep piling on".[^1] We need new ideas and we need to try new approaches if we are ever going to evolve.

Software engineering vs. engineering engineering

In that Alan Kay talk he references the construction of the Empire State Building. This got me thinking how programming is fundamentally different from construction engineering. At the core it's about the difficulty of changing the system: a building is immensely difficult to change after it is finished, whereas software can be relatively trivial to modify.

This is something we should embrace. It is miraculous that we have a technology that can so easily be molded into different shapes. However, it comes with some interesting side-effects at many levels.

For example, organizations feel more comfortable taking risks with software. If they can fix the software after it has shipped, they are more likely to take a "more, buggy features" approach than a "fewer, higher quality features" approach and fix the bugs after. This is one reason why technical debt grows even before a piece of software is released--you're already operating under the assumption that you'll fix it "later", because it's relatively easy to fix it then.

The lean startup methodology brought continuous deployment, which meant that software gets shipped without even being tested, because even if the end-users have a bad experience, the change can come so fast that it will be fixed the very next day. I'm not endorsing this as a good practice, just indicating it's something unique to software.

Ideally, we embrace the ease of changing to allow users to customize the software and to allow software to be ported easily to new platforms. We should also try to be aware of the negative impact frequent change has on user experiences when we ship software that might frustrate them, ruin their data, or put them in danger.

Burn it all to the ground

Project Oberon is "a design for a complete desktop computer system from scratch. Its simplicity and clarity enables a single person to know and implement the whole system, while still providing enough power to make it useful and usable in a production environment."

Alan Kay and several others formed Viewpoints Research Institute partially to answer a similar question: If we were to start from scratch, how much code would it take to get us what we have?

Another great paper to read is Rob Pike's "polemic" Systems Software Research is Irrelevant.

Many programmers thinking of these issues fantasize about doing these kinds of rewrite projects. It takes both confidence and, pessimistically, arrogance to believe that you can re-invent the software world better than those who came before. However, we should not discourage people from doing this, for all the reasons I have stated previously:

  • The system could be much stronger with fewer minds designing it (Conway's law)
  • The system could escape the local maxima the previous system was trapped in (Accretion of complexity)

Problems with starting over

The hardest pill to swallow is the upfront cost of catching up. This cost is largely levied by the use of existing technology. Complexity seeps in from every level:

  • Users are frequently quirky and understand things in different ways. This is the domain of user interface/user experience design.
  • People disagree, but must come together and create standards of communication to utilize the physical infrastructure optimally. These standards bring with them significant complexity. They get revised every so often, thereby obsoleting existing software and requiring new work be done to support the new standards. Backwards compatibility can be a way to reduce this impact, but it can also reduce innovation and impose a large cost on implementing a standard.
  • Programmers want to save time by layering on abstractions, each of which adds complexity and possibility of bugs.
  • Computer hardware is complicated, frequently in order to offer higher performance (which I believe is frequently worth it because it enables new possibilities).
  • The physics of the universe itself make things difficult. Computers get hot, bits must be transferred between computers via some medium, there's EM noise and physical vibration, etc.

At a nitty-gritty software level, one might try to mitigate some of these complexities by re-using software. This is made more difficult by the explosion of dependencies between software. In the worst case I've seen, a single JavaScript file to rasterize a font to an image resulted in downloading code from three different languages (Python, C, and JavaScript), took over 3 minutes to download on my 20 Mebibyte/second connection, and failed to run due to missing pip dependencies! Software is a disaster.

Please, don't let this discourage you from trying to start something from scratch! We need people who are willing to do the hard work to advance the field.

Virtualize everything

A common head-in-the-sand approach to attacking complexity is to create your own little virtual machine, then program against it.

This brings along with it numerous problems:

  • You have to implement that virtual machine, which means you still have to deal with that complexity; it's not a way to actually eliminate the complexity at its source.
  • Performance always degrades. Always. At the core of the counter-argument to this is the "sufficiently smart compiler" that will magically make the cost of virtualization/high abstraction go away. It's not here yet, and I wouldn't bank on it being here for a while. JIT compilers bring significant complexity and runtime cost.
  • Hardware matters. The industry has largely been profiting off of orders-of-magnitude improvements in hardware. Virtualizing away what makes one machine special or faster than another is counter-productive.
  • It's another layer which has overhead, has to be learned, can break, people must agree on, etc. It's a liability. It's more. We want less, not more.

A similar approach is to have strict abstraction layers which separate your code from the "other stuff". Interestingly, this approach suffers the same ills predicted by Conway's law. If I'm writing a game in C using SDL as my abstraction layer, I am less likely to do things that the operating system would allow, but SDL does not. It's possible for me to modify SDL to expose the functionality, but there's more friction in doing so, which means I'm less likely to do so.

I'm not trying to rag on using libraries, because it seems like a much more sustainable approach than using gigantic game engines or frameworks. It's a danger we still need to be aware of in all of these approaches.

What's the solution?

The only solution I can think of to these problems is strong culture, community, and education.

We cannot focus on a single piece of technology that will somehow save us--we have to focus on the people. Smart, aligned people can make new technology and tackle new problems.

Culture

People need to care about these problems. If no one believes we even have problems[^2], then we of course will never efficiently solve them.

We need to encourage innovation. When someone starts writing a new operating system, programming language, game engine, etc., we should be cheering them on.

If someone isn't yet experienced enough to practically undertake the project, we can advise them as much while still encouraging them by saying e.g. "I don't think you are ready to undertake that scale of project, but you will get there in time if you continue learning, especially by studying X, Y, and Z."

In comparison, if you ever say "it's stupid to create an X, we already have Y", think really hard about what positive influence you actually have. Think about your Y, and what came before it. Would the creators of that Y have been told the same thing, for their era? Wouldn't you rather have a new, potentially better X in ten years, rather than aborting it at its inception?

Similarly, we need to encourage digging deeper. We should encourage web developers to learn about cache lines and GPU architectures. Game devs should know how the internet works. Don't encourage people to only learn the minimum amount necessary to complete their next task. Learning as little as possible is not a virtue. Take joy in exploring all the fascinating innovations. Take note of all the cruft you find on the way so you can learn what not to do.

Community

You need people willing to do work to fix these problems. Those people likely need to talk to other people to get advice or increase their understanding.

This is one of the shining traits of free software. Knowledge gained from studying proprietary software is restricted to that company and the people who bring knowledge from other companies. Clearly an open industry is superior; artificial restriction of knowledge can be nothing but harmful to innovation.

As a small PSA, please don't start your community on a closed platform like Discord. These platforms are not good for long-term archival and search. The data aren't available to the public. You pour more fuel on a closed, proprietary future rather than supporting an open and free discourse, of which we have multiple free and open protocols to choose from.

Education

New people are constantly being born, growing up, and entering the field. We need systems to educate people on how to combat these problems.

If you think universities instill any good software development practice, well, you haven't interviewed many new grads. This isn't necessarily the fault of universities. They rarely have a stated goal to create good programmers, and instead focus on the science of the field. There's definitely value to that; the danger is assuming that someone with a CS degree knows anything about writing good software. Don't believe me? Next time you read a CS paper, actually check out the code in their GitHub link. Again, writing good code is not really a goal of papers, nor the university.

Technical schools are similarly suspicious. Coding bootcamps frequently hire their own graduates to be instructors, meaning the instructors never gain real industry experience or write significant programs before parroting what they were taught. It's effectively playing the game of Telephone with programming knowledge.

Education in the game industry

The game industry has GDC, a conference where developers gather to share insights gained in the field. However, talks frequently cover the "what we did" rather than the "how we arrived here" nor "what was tried but ended up failing". Most GDC talks are pay-walled in GDC Vault. Talks frequently are thinly veiled recruiting advertisements rather than honest attempts at sharing knowledge. They are often mile-high views without revealing any deep technical details.

The game industry does not have an open source software practice. id Software, while under John Carmack, is one of the only AAA game studios to release the entirety of their game code under free software licenses. Since Carmack left, they no longer follow this practice. Indie developers are no better on this front, despite the drastically lower legal oversight that might cause a corporation more friction in releasing their code.

University courses in games are becoming more common, but courses which teach game tech fundamentals (as opposed to specifically Unity or Unreal) are becoming increasingly rare. People are graduating with degrees that certify they know Unity, not how game engines actually work. These practices do not bode well for innovation in engine technology in the long term.

Conclusion

I am consistently frustrated with my experience writing software. What is a fundamentally simple task is often made incredibly hard to implement due to accidental complexity and cruft. I feel powerless to fix everything, because I have limited time and limited willpower.

However, I continue to learn and grow, and hope that I can positively impact the field by contributing to and participating in these solutions I've proposed. At the end of the day, the magic of software keeps me going, and I hope I can somehow spread that magic and amplify it with my work.

If you have thoughts, I encourage you to share them with me at [email protected].

[^1]: For those feeling motivated to jump right in, I will temper this advice by saying that you should really know the actual problems and state of the art before trying to solve them. This can take a few years of study and experience before you will actually be able to know where the problems and the frontiers are.

[^2]: I have raged against the Unity- and Unreal-ification of the game industry time and time again. It seems the majority opinion at both the indie and AAA level is that moving to the big U engines is a good, perfect thing with no drawbacks. The short-sightedness of the engine commoditization is the second biggest problem in the game industry in my opinion, the first being the rising the cost per byte (Raph Koster - Industry Lifecycles). These two are intimately related--the verdict on whether moving to big-U's is better rests on the answer to the following question: are our games getting better (higher quality, and not just graphically), easier to make, and faster to deliver? If the answer is not "yes" to all of these, there's something lost here.