Maybe I’m just not good enough at paying attention, but for me it seems like you have to actually run into problems over and over and figure out how to avoid the problems. Then you end up being able to mentally simulate what problems you will run into, and design is basically all about avoiding future problems of various kinds (and balancing tradeoffs about which future problems to avoid and how much effort to put into each, whether you can solve multiple with one design play, etc).
I have this too, I have never been able to "do exercises" or "study a codebase". I need to be making something that I am excited about, then I'll learn, from examples, from wanting to be thorough and correct.
But sometimes I think I'm just not their yet, if I become able to read code like a book and really understand what happens, which often I don't, then perhaps I'll enjoy the process more.
That's how I learn as well. I've found that reading code is a completely different thing from learning. Not even tangentially related.
I can read a codebase and run simulations in my head. But this almost never results in learning anything. I can study code and see how a particular task is done, but I don't learn it until I put it into practice. Just about the only thing I get from reading other people's code is the most egregious ways to not do something.
This is a pretty common thing, a plurality of humans learn this way. We just have different standards for software, for some reason. We like to pretend books teach you something more "real" than just getting your hands dirty and writing some real fucked up code. But the reality is that figuring out why your disaster code doesn't work, and then fixing it is one of the most educational experiences a programmer can have.
> for me it seems like you have to actually run into problems over and over and figure out how to avoid the problems
This shows how immature the field of software engineering is. Imagine bridges or houses were built like that. Or your surgeon was trained like that.
Over time, we hopefully develop estblished norms, but at the moment, things are too much in flux. Put 5 sw engineers in a room, pose a problem and you will get not just 5 different solution proposals, but there will likely be strong disagreements on which approach is a good one.
"I recognize a good solution when I see it" is just not good enough for a serious engineering discipline.
I generally agree with your point on ease of experimentation, but if we insist on calling it software engineering, then maybe the field needs to adhere to engineering principles, as the GP highlighted.
I believe part of engineering isn’t over-engineering for the task at hand as well. If the costs of a “failure” are low/zero then the right thing can be to move quickly expecting some problems.
I think the field could get better at knowing when costs are low (eg sometimes scalability, cheaper to change a database choice than rebuild a bridge) and where the costs are sometimes very high (eg security).
When design is a process to build something pleasing (like in music or painting), then yes, it's an art, and you can rightfully have different opinions about outcomes, sometimes diametrically opposed opinions, which is all not just fine and accepted but inherent in the field and a required ingredient.
When design is about building something functional, then it's part of engineering. You can objectively measure whether it does satisfy the requirements. That's where serious engineering disciplines are and where software engineering needs to aim for if it wants to keep engineering in its title without getting laughed at. As long as programmers consider themselves part artists, that's hard to achieve, but as the field matures, there is hope.
Note that some fields are trying to be both. For example architecture. There it's solved by making a more or less clear distinction between the pleasing art part (that's what the architect does) and the functional engineering part (structural engineer). Both roles are not being conflated. In software, we still do that.
Bridge building is a lot more conservative when it comes to taking risk in the construction, but that is how we build bridges and lots of bridges collapse because of similar causes:
- Design Deficiencies
- Construction Mistakes
- Maintenance Issues
- etc.
An average of 128 bridges collapse annually in the United States. More than 17,000 bridges in America are considered "fracture critical" (vulnerable to collapse from a single impact).
Engineering doesn't mean all processes are flawless and surely not that all humans participating are perfect.
The collapsing bridges in the US are mainly caused by insufficient maintenance, which is not an engineering problem but a political one. And the vast majority of collapses happens with bridges that are known to be unstable and already blocked for traffic. The engineering part did that, as if it screams "told you so" at the politics which doesn't allocate sufficient funds for maintenance.
If they could afford experimenting and have a few bridges collapse before they get it right with no significant negative consequences IMHO it wouldn’t be the worst way to learn.
Maybe even more so for surgeons, being able to experiment and fail in a risk free environment seems like a good thing.
> This shows how immature the field of software engineering is. Imagine bridges or houses were built like that. Or your surgeon was trained like that.
It's not that software engineering is immature, it's just more dynamic.
We are not the surgeon, we write the surgeon. We write a surgeon to fix a broken leg. Once that is done, we don't have to fix another leg. Now we need to reattach a finger. Once that is solved, maybe replace a kidney.
You cannot repetitively train or have strict rules for that, because every time it's something new. You need to have broad knowledge and experience to be able to fight the next unknown challenge. It's unknown because it's never been done before, or it has been done but your competitor will not reveal the details.
Building bridges or being a surgeon sounds very boring to me, since it's always the same (maybe some minor variants). Building software? Very much not the same.
> This shows how immature the field of software engineering is. Imagine bridges or houses were built like that.
You're forgetting a key difference between software and the physical world.
The blessing and curse of software is that there are few constraints on the solution so it can take an enormous number of different forms and still be valid. There isn't gravity, friction or the various other things that constrain physical solutions.
And each of those almost limitless solutions has different trade-offs related to a large number of different variables. It takes many many years of seeing the different impacts of those design decisions across the different variables within the constraints of different domains and contexts.
It's truly a combinatorial problem with so many dimensions that no human is smart enough to just simulate the impact of design decisions in new and unfamiliar domains+context.
You could argue that there could be standard ways of doing things within specific domains+context, but the context is so varied between companies, and business priorities have so many differences that I don't think it's a solvable problem.
The best we can do is have higher level patterns and approaches to specific types of problems that can guide people based on others experiences, but all of that still needs to be mixed and matched for the specific problem space.
I agree with your assessment and didn't say it's easy. If it was then we'd already have solved it.
But that doesn't mean it can't be solved. I'm sure 150 years ago people said something similar about flight. Not a great analogy, but the point stands that if we keep working on it and have a goal then in a few decades we could get closer to actual engineering than the mess we are in today.
For me, software design is more comparable to business system and organization design. It’s just that instead of humans executing the processes you design, it’s machines doing it.
I don’t think business system and organization design is approached like bridge design either, is it?
Also, bridges, houses and surgeons can physically kill people if something goes wrong. Software that can physically kill people, such as that in airplanes or missiles, is actually treated quite differently from most software, I think? I don’t have experience in those industries those so I can’t comment on the specifics of how is it different, but my impression is that things are a lot more rigid. Business organization design also can’t directly kill people.
In general, I think that there is a fundamental tension between looseness and flexibility of operations and innovation. If you are super rigorous and have set in stone best practices, it is going to be harder to find new ways of doing things that work better.
I’m not really bothered if people don’t consider software to be a serious engineering discipline. I’m not sure I do either. If someone wants that kind of thing, I’d recommend they go into a different engineering discipline, rather than trying to make software like that.
Tell me you don't know the first thing about engineering without telling me...
This is the engineering process. If you put five engineers of any discipline in a room, you will get five different answers. Every contractor and architect has their own ideas about how things should be done.
Furthermore, we do build houses this way, even in the modern age with building codes. The builder is going to do whatever they can get away with and this is a universal truth. The only reason bridges are held to such high standards is because of the monetary cost of a collapse.
The "norms" are not what you think they are. They're tradition, they're "we've always done it this way". What you're talking about are laws written for public safety. Those laws only exist because we tried and failed to do things a certain way and the cost in money or lives was untenable.
> "I recognize a good solution when I see it" is just not good enough for a serious engineering discipline.
You're conflating engineering with science. A scientist will rigorously test and validate, but a huge part of engineering is just recognizing a good solution. Of course there's a lot of testing and validation in engineering as well, but not with the kind of rigor you're implying.
That's kinda like saying you can learn to drive by just getting into a car, crashing then thinking about how not to crash it next time.
In reality both things are necessary. The car analogy doesn't hold for road driving because we drive well within the limits, but for racing it really is necessary to know exactly where the limits are. I don't think we should really be treating our profession like a race, though.
But if you don't read it's going to be an incredibly long slow process and a lot of car crashes and mangled gearboxes etc. So I say read, read, and read some more. Even if you don't see the point of it right now your experience will later find a place for it and you won't end up descending a hill for the first time not knowing to shift to a lower gear.
>That's kinda like saying you can learn to drive by just getting into a car, crashing then thinking about how not to crash it next time.
That would be a perfectly valid way of learning to drive if crashing had no danger or destruction and you could instantly reset the car every time. Software is a special case of engineering where the cost of failure is extremely low, so trial and error is generally the fastest way to get going with actually doing something.
It would be an incredibly inefficient way of learning anything. You might as well say you could just rediscover all of human knowledge by just observing the world and doing your own experiments. Turns out standing on the shoulders of giants tends to get you further.
you mention a key point. profession. programmer doesnt really imply professional programmer.
I'd aay if you do it for a living, certain tedious chores must be learned. the best programmers i know (professional) can all read code. they spent many junior years learning to read it, being on code auditing desk.... nowadays idk how the landscape looks, but for all of them they had to review and read code to find bugs before they were allowed to produce code (they all worked at same company ofc... so my view is limited!)
i do feel such discipline is needed. they can always poke holes on my code no matter how many holes i plug :) - i am semi professional. i write code for work, but not production code. (experimental). i never learned to audit code and feel that makes it impossible for me to truly create production grade code
This is pretty much how I've learned up to this point (and will of course continue). Trying to learn from real world code will be a new experience for me. Not sure how valuable it will be but should be fun either way.
My immediate reaction to this question is: "your team's". Nothing will teach you more about how to design software then really understanding why good and bad solutions were adapted to solve a certain real problem.
Software exists precisely because there is still a messy layer connecting user requirements to actions on a computer. If there was not messiness then we could just automate it all. Approaching software from some sort of Platonic ideal of what software should be will frequently lead to bad decisions on it's own.
When you start to see how certain pressures lead to certain paths you learn to recognize the wrong decisions that are often good at the time, and avoid them. At the same time, you need to learn to develop methods that work quickly and effectively. By far the biggest real challenge in real world software is time constraints. This is almost never discussed in theoretical views of software, but the truth is you're always going to be writing code under pressure to ship. You will come across situations where you do not have time to do what you want to do or think is best.
Good software is software that runs and solves the user need, but you will come to realize that there are design solutions that will make successfully running happen more often. The best way to find these is to study the real software you're writing.
Then it still matters a lot what kind of codebase he is working on. A web project is differently structured and done than the codebase for a embedded operating system. There are differend standards and practices in the industry.
I learned a lot by just stepping through the code with the debugger of libaries I used. That brought more practical insight while learning about design patterns etc. In the end, it is all about patterns. Finding the right pattern for a given problem.
I would be interested in recommendations for those who have a poor example or no examples
In my case, I'm a junior engineer that has recently been given more responsibility designing aspects of our product. I'm just trying to learn all I can so my designs will be good!
I haven't considered time to implement as a metric for evaluating design, but it makes a lot of sense.
I definitely learn so much from my team's codebase. Most of what I learn is either from the good designs I see in there or from my googling trying to fix the not so good parts.
These are several years old at this point, but many open source project leaders contributed to the series "The Architecture of Open Source Applications", which is free to read online: https://aosabook.org/en/index.html
My experience (30 years in software, 25 in practicing architecture, MIT system architecture masters) tells me there is no such thing as abstractly “good” design. There are designs with negative consequences for sure, but “good” depends on the context: what are you building, safety/security requirements etc. Probably most importantly on the implementation team and it’s structure. A team of juniors will butcher your intricate design and Conway’s Law makes your software reflect the team.
Yeah, as I've been learning more about software design, it's become pretty clear there is no silver bullet. It would be nice though. I find it a bit overwhelming to try to find the best solution for a problem when there are so many different architectures and programming paradigms.
That being said, taking into account the requirements does eliminate quite a few of the options. Right now, I work on safety-critical embedded systems which requires us to make some decisions that would most likely be way different in other environments.
It's about having most compromises be Not Totally Stupid and have the worst ones only be locally stupid. There is a big universe out there of compromises that fulfill those criteria. As you imply, a lot of the time it's more about constantly fighting for Less-Bad-and-avoiding-Dumb than about architecture as such.
The codebase is not where the design usually lives. It's where the implementation lives. You could imagine a rewrite into another programming language which would preserve the design but completely replace the implementation.
You should practice writing design docs. Don't worry about what the doc is supposed to look like, and definitely don't work off of a template. The most important thing about the doc is that another human could do the implementation if you gave it to them.
The doc can also function as a "proof of consideration". If you choose to do something one way, but there are other possible ways to do it, you can acknowledge the other possible ways, and say why they are worse. By preemptively acknowledging an alternative, you have proved to readers that you considered it.
All a "good" system designer is doing is considering a larger design space than most, and consistently finding good points in the space. Pick a problem, sample points in the design space, tell me why some points are better than others, and write it all down.
Trying to learn software design by looking at code is kind of like trying to learn about architecture looking at the bricks that make up a building.
To learn about design you need a wider perspective. You can theoretically learn it from code but it won’t be most effective. Look at great documentation and literature about design instead.
Yeah, I guess without the context I wouldn't really understand why it was designed that way. Or know what situations a design like that would be good for.
Is there any documentation or literature that has helped you?
Once upon a time there was an email MTA (email server) called qmail. It had an architecture that was surprising and Daniel Bernstein documented his philosophy. It left an impact on me by showing me how we can rely on the OS-supplied facilities to accomplish a lot. I don't think qmail is some shining star that will teach you everything you need to know about software design. It is an interesting case study to add to your repertoire.
I think there is value in groking entire code bases. It's not just about whether or not they are well designed though. It is an important skill to be able to analyze and see the big picture of how things work together in large systems. For me it often involves drawing diagrams (sometimes UML) to map things out. Being able to view systems in this way is a pre-requisite to intentionally designing your own systems at this level. And yes, once you can work at this level you can learn from good designs, but also see problems with bad ones.
EDIT: and to answer your question, if you're working on something that is "like X but different" then read the source code for X. You could also look at source code for software that you use from day to day: software where you already know what it does. For example, if you're in web, maybe the web framework, or web server, if you write python, maybe a core library that you use, or maybe the python interpreter, or if you use vscode ..., if you use android ..., you get the idea. At the start I would suggest smaller programs, and programs where you already know the domain (e.g. cpython might not be the best place to start if you never implemented an interpreter before, you may spend more time learning about interpreters than the design of this one, still a good thing to learn of course.)
Well maybe I'll take some time to learn about email servers then. I use email everyday so I suppose it'll be good to know a little more about how it works.
* "well designed": What was the objectives and ideas...
* "codebases": How well that was implemented
They are a lot of lofty claims saying how this or that is "fast, secure, etc" but don't end like that in the actual implementation.
But most of the time, that could be seen in the "design claims" already! Good design is not just full of adjectives and nice sounding goals, but the concrete considerations, what was the trade-offs, false-starts, and reasons behind the decisions.
You can see some examples reading about the design of Erlang, early pascal, most RDBMS, etc.
So, you first mid/long term goal is to learn to distinguish what good design actual look like.
then, in relation with codebases then to be kinda easier: It actually follow the design?
A good example is the 'std' library of Rust. It has a lot of lofty claims about security and such things that could sound alarms, but then you dive in the code of it and see is there A LOT of care about it, and a lot of docs comments discussing this stuff and then the code match.
P.D: The "std" or equivalent of the lang is one of the most important codebases you need to learn and study, and the MAJOR way to judge how truly good is it.
Start with the question: what is the problem that you want to solve? Next, find codebases that solve that problem and study how they do it.
Good design is so deeply tied to the domain details. Wonham's Internal Model Principle applies to code.
Example: I wanted to solve the problem of unit testing for embedded targets. I found open source projects that do this and read the code critically to see how and why it is written. As I build my own approach, I revisit theirs to learn more as my understanding of the domain deepens.
While studying well-designed codebases is incredibly valuable, there's an important "tip of the iceberg" effect to consider: much of good software design lives in the "negative space" - what's deliberately not there.
The decisions to exclude complexity, avoid premature abstractions, or reject certain patterns are often just as valuable as the code you can see. But when you're studying a codebase, you're essentially seeing the final edit without the editor's notes - all the architectural reasoning that shaped those choices is invisible.
This is why I've started maintaining Architectural Decision Records (ADRs) in my projects. These document the "why" behind significant technical choices, including the alternatives we considered and rejected. They're like technical blog posts explaining the complex decisions that led to the clean, simple code you see.
ADRs serve as pointers not just for future human maintainers, but also for AI tools when you're using them to help with coding. They provide readable context about architectural constraints and compromises - "we've agreed not to do X because of Y, so please adhere to Z instead." This makes AI assistance much more effective at respecting your design decisions rather than suggesting patterns you've deliberately avoided.
When studying codebases for design patterns, I'd recommend looking for projects that also maintain ADRs, design docs, or similar decision artifacts. The combination of clean code plus the architectural reasoning behind it - especially the restraint decisions - provides a much richer learning experience.
Some projects with good documentation of their design decisions include Rust's RFCs, Python's PEPs, or any project following the ADR pattern. Often the reasoning about what not to build is more instructive than the implementation itself.
Oooh I like that idea. I may steal it. I'll be on the lookout for documents like that. It'll be interesting to see what patterns/designs were avoided. There are so many ways to accomplish the same thing it might be nice to set some limits.
One thing to keep in mind is that what was well-designed 30, 20 or 10 years ago may not be considered such now. Hardware changes and so do the design decisions involving performance.
For example, if you are looking at C++ networking libraries, learning from ACE or even Asio may not be the best idea - better look at "thread per core, share nothing" Seastar[0].
Another thing is that it may be better to read design docs, not the code. For example, the rationale for the mold linker design[1].
The MIT course for Operating Systems has a teaching OS called xv6. There's some nifty implementations in there. I'm not sure I'd comment on the codebase itself, but I think more importantly it teaches you good abstractions which Unix is all about: https://pdos.csail.mit.edu/6.1810/2024/xv6.html
Proceed with extreme caution. This is highly subjective and often results in a blind leading the blind scenario. For example how do you determine which code bases are well designed if you yourself are not comfortable determining such?
Instead, heed the guidance of some of the other comments and just start building things. Plan on failing and solving for the failure. That’s called practice. With enough practice you will know what works well for you. If you are measuring things along the way you will also know what’s slow.
The codebases I learned from the most are Git, Postgres, CPython. Not saying they are perfect designs, but they are well maintained, solve hard problems, have seen many years of evolution, and are very easy to get your hands on.
Top 5 codebases for changing my mind about things:
Wietse Venema's Postfix mail server. Taught me tons about security posture, the architecture i'd describe as microservices before microservices was a thing, but contrary to the modern take on microservices (it's mostly a tool for decomposing work across large semi-isolated groups) this was primarily about security and simplicity.
Spring framework - this opened my eyes to ways of working that i hadn't really thought enough about before, the developers on that project have a culture of deeply considering the needs of their users (who are java developers often in an enterprise environment).
Git - the thing i like about the git code base is that once you've covered the objects database (e.g. blobs, trees and commits) and the implementation of refs, everything else just feels like additional incremental features. With those core concepts, everything else is kinda harmoniously built on top.
Varnish by Poul Henning-Kamp is another one - feels like he went to great lengths to make that code base a teaching tool despite the fact it's also a top tier reverse proxy.
Last one isn't a code base - but it will help with software design in the large; studying how the lieutenants model works in the linux kernel.
Thinking about my answers, i think i've highlighted something subtly different than "well designed codebases" it's more a list of codebases that left a notable long lasting impression on me because of design decisions they made.
Are there any books or websites you'd recommend I read to learn more about design instead? I 100% need to learn by doing, but wouldn't it also be good to learn what has already been discovered so I don't have to reinvent the wheel? Or is it best to discover good design on your own?
Your own. Everyone else's is trash. Never understand someone else's code-- the whole point is that there should be just a handful of functions of yours I can call, and shouldn't ever care to see how the sausage is made. Otherwise, your abstraction sucks
I agree that the internals of my code should be abstracted away from my users. But don't I still want the internals to be well designed so they are maintainable and extendable?
study modern code bases of language you want to learn, and look first at their history to see if they are mature, came from a company, academics etc.
you should not only study 'good' code... how will you know what is bad code?
study code that does similar things to what u want (client/server/game/ai/datacrunching etc.)
and study lot of it..different qualities, ages, and sources
I'm definitely going to do that. I asked because I wanted to start off with some that I'd knew would be well designed. However, it would probably be more interesting to see a wide variety of quality than just the good ones.
Or maybe open source projects are all usually well designed. I haven't looked at many in depth. The only one I've really looked into was Clang to try to figure out why clang format ignored my style rules when styling files of certain names. (Turns out there is a list of file names that automatically are considered Objective-C rather than C++)
One of the best is actually the implementation of the Tcl language. It's highly readable C code, very well structured (it was initially by Ousterhout, after all).
Maybe I’m just not good enough at paying attention, but for me it seems like you have to actually run into problems over and over and figure out how to avoid the problems. Then you end up being able to mentally simulate what problems you will run into, and design is basically all about avoiding future problems of various kinds (and balancing tradeoffs about which future problems to avoid and how much effort to put into each, whether you can solve multiple with one design play, etc).
I have this too, I have never been able to "do exercises" or "study a codebase". I need to be making something that I am excited about, then I'll learn, from examples, from wanting to be thorough and correct.
But sometimes I think I'm just not their yet, if I become able to read code like a book and really understand what happens, which often I don't, then perhaps I'll enjoy the process more.
That's how I learn as well. I've found that reading code is a completely different thing from learning. Not even tangentially related.
I can read a codebase and run simulations in my head. But this almost never results in learning anything. I can study code and see how a particular task is done, but I don't learn it until I put it into practice. Just about the only thing I get from reading other people's code is the most egregious ways to not do something.
This is a pretty common thing, a plurality of humans learn this way. We just have different standards for software, for some reason. We like to pretend books teach you something more "real" than just getting your hands dirty and writing some real fucked up code. But the reality is that figuring out why your disaster code doesn't work, and then fixing it is one of the most educational experiences a programmer can have.
> for me it seems like you have to actually run into problems over and over and figure out how to avoid the problems
This shows how immature the field of software engineering is. Imagine bridges or houses were built like that. Or your surgeon was trained like that.
Over time, we hopefully develop estblished norms, but at the moment, things are too much in flux. Put 5 sw engineers in a room, pose a problem and you will get not just 5 different solution proposals, but there will likely be strong disagreements on which approach is a good one.
"I recognize a good solution when I see it" is just not good enough for a serious engineering discipline.
> Imagine bridges or houses were built like that. Or your surgeon was trained like that.
While I don't disagree with you in general, this does feel a bit off.
By that logic you can call the field of music immature, and all of the arts. I think the difference is that its easy to experiment without high costs.
I genuinely think that if building bridges was cheap and quick, the fastest way to learn was to try...
I generally agree with your point on ease of experimentation, but if we insist on calling it software engineering, then maybe the field needs to adhere to engineering principles, as the GP highlighted.
I believe part of engineering isn’t over-engineering for the task at hand as well. If the costs of a “failure” are low/zero then the right thing can be to move quickly expecting some problems.
I think the field could get better at knowing when costs are low (eg sometimes scalability, cheaper to change a database choice than rebuild a bridge) and where the costs are sometimes very high (eg security).
Notice OP says "software design". Design is still an art, even in mechanical engineering.
Engineering applies to looking at a design and proving something about it.
The term "design" is ambiguous here.
When design is a process to build something pleasing (like in music or painting), then yes, it's an art, and you can rightfully have different opinions about outcomes, sometimes diametrically opposed opinions, which is all not just fine and accepted but inherent in the field and a required ingredient.
When design is about building something functional, then it's part of engineering. You can objectively measure whether it does satisfy the requirements. That's where serious engineering disciplines are and where software engineering needs to aim for if it wants to keep engineering in its title without getting laughed at. As long as programmers consider themselves part artists, that's hard to achieve, but as the field matures, there is hope.
Note that some fields are trying to be both. For example architecture. There it's solved by making a more or less clear distinction between the pleasing art part (that's what the architect does) and the functional engineering part (structural engineer). Both roles are not being conflated. In software, we still do that.
> Imagine bridges or houses were built like that.
Bridge building is a lot more conservative when it comes to taking risk in the construction, but that is how we build bridges and lots of bridges collapse because of similar causes:
An average of 128 bridges collapse annually in the United States. More than 17,000 bridges in America are considered "fracture critical" (vulnerable to collapse from a single impact).Engineering doesn't mean all processes are flawless and surely not that all humans participating are perfect.
The collapsing bridges in the US are mainly caused by insufficient maintenance, which is not an engineering problem but a political one. And the vast majority of collapses happens with bridges that are known to be unstable and already blocked for traffic. The engineering part did that, as if it screams "told you so" at the politics which doesn't allocate sufficient funds for maintenance.
> Imagine bridges or houses were built like that
If they could afford experimenting and have a few bridges collapse before they get it right with no significant negative consequences IMHO it wouldn’t be the worst way to learn.
Maybe even more so for surgeons, being able to experiment and fail in a risk free environment seems like a good thing.
> This shows how immature the field of software engineering is. Imagine bridges or houses were built like that. Or your surgeon was trained like that.
It's not that software engineering is immature, it's just more dynamic.
We are not the surgeon, we write the surgeon. We write a surgeon to fix a broken leg. Once that is done, we don't have to fix another leg. Now we need to reattach a finger. Once that is solved, maybe replace a kidney.
You cannot repetitively train or have strict rules for that, because every time it's something new. You need to have broad knowledge and experience to be able to fight the next unknown challenge. It's unknown because it's never been done before, or it has been done but your competitor will not reveal the details.
Building bridges or being a surgeon sounds very boring to me, since it's always the same (maybe some minor variants). Building software? Very much not the same.
Most swe is just crud
> This shows how immature the field of software engineering is. Imagine bridges or houses were built like that.
You're forgetting a key difference between software and the physical world.
The blessing and curse of software is that there are few constraints on the solution so it can take an enormous number of different forms and still be valid. There isn't gravity, friction or the various other things that constrain physical solutions.
And each of those almost limitless solutions has different trade-offs related to a large number of different variables. It takes many many years of seeing the different impacts of those design decisions across the different variables within the constraints of different domains and contexts.
It's truly a combinatorial problem with so many dimensions that no human is smart enough to just simulate the impact of design decisions in new and unfamiliar domains+context.
You could argue that there could be standard ways of doing things within specific domains+context, but the context is so varied between companies, and business priorities have so many differences that I don't think it's a solvable problem.
The best we can do is have higher level patterns and approaches to specific types of problems that can guide people based on others experiences, but all of that still needs to be mixed and matched for the specific problem space.
I agree with your assessment and didn't say it's easy. If it was then we'd already have solved it.
But that doesn't mean it can't be solved. I'm sure 150 years ago people said something similar about flight. Not a great analogy, but the point stands that if we keep working on it and have a goal then in a few decades we could get closer to actual engineering than the mess we are in today.
For me, software design is more comparable to business system and organization design. It’s just that instead of humans executing the processes you design, it’s machines doing it.
I don’t think business system and organization design is approached like bridge design either, is it?
Also, bridges, houses and surgeons can physically kill people if something goes wrong. Software that can physically kill people, such as that in airplanes or missiles, is actually treated quite differently from most software, I think? I don’t have experience in those industries those so I can’t comment on the specifics of how is it different, but my impression is that things are a lot more rigid. Business organization design also can’t directly kill people.
In general, I think that there is a fundamental tension between looseness and flexibility of operations and innovation. If you are super rigorous and have set in stone best practices, it is going to be harder to find new ways of doing things that work better.
I’m not really bothered if people don’t consider software to be a serious engineering discipline. I’m not sure I do either. If someone wants that kind of thing, I’d recommend they go into a different engineering discipline, rather than trying to make software like that.
Tell me you don't know the first thing about engineering without telling me...
This is the engineering process. If you put five engineers of any discipline in a room, you will get five different answers. Every contractor and architect has their own ideas about how things should be done.
Furthermore, we do build houses this way, even in the modern age with building codes. The builder is going to do whatever they can get away with and this is a universal truth. The only reason bridges are held to such high standards is because of the monetary cost of a collapse.
The "norms" are not what you think they are. They're tradition, they're "we've always done it this way". What you're talking about are laws written for public safety. Those laws only exist because we tried and failed to do things a certain way and the cost in money or lives was untenable.
> "I recognize a good solution when I see it" is just not good enough for a serious engineering discipline.
You're conflating engineering with science. A scientist will rigorously test and validate, but a huge part of engineering is just recognizing a good solution. Of course there's a lot of testing and validation in engineering as well, but not with the kind of rigor you're implying.
That's kinda like saying you can learn to drive by just getting into a car, crashing then thinking about how not to crash it next time.
In reality both things are necessary. The car analogy doesn't hold for road driving because we drive well within the limits, but for racing it really is necessary to know exactly where the limits are. I don't think we should really be treating our profession like a race, though.
But if you don't read it's going to be an incredibly long slow process and a lot of car crashes and mangled gearboxes etc. So I say read, read, and read some more. Even if you don't see the point of it right now your experience will later find a place for it and you won't end up descending a hill for the first time not knowing to shift to a lower gear.
>That's kinda like saying you can learn to drive by just getting into a car, crashing then thinking about how not to crash it next time.
That would be a perfectly valid way of learning to drive if crashing had no danger or destruction and you could instantly reset the car every time. Software is a special case of engineering where the cost of failure is extremely low, so trial and error is generally the fastest way to get going with actually doing something.
It would be an incredibly inefficient way of learning anything. You might as well say you could just rediscover all of human knowledge by just observing the world and doing your own experiments. Turns out standing on the shoulders of giants tends to get you further.
you mention a key point. profession. programmer doesnt really imply professional programmer.
I'd aay if you do it for a living, certain tedious chores must be learned. the best programmers i know (professional) can all read code. they spent many junior years learning to read it, being on code auditing desk.... nowadays idk how the landscape looks, but for all of them they had to review and read code to find bugs before they were allowed to produce code (they all worked at same company ofc... so my view is limited!)
i do feel such discipline is needed. they can always poke holes on my code no matter how many holes i plug :) - i am semi professional. i write code for work, but not production code. (experimental). i never learned to audit code and feel that makes it impossible for me to truly create production grade code
This is pretty much how I've learned up to this point (and will of course continue). Trying to learn from real world code will be a new experience for me. Not sure how valuable it will be but should be fun either way.
This is it
My immediate reaction to this question is: "your team's". Nothing will teach you more about how to design software then really understanding why good and bad solutions were adapted to solve a certain real problem.
Software exists precisely because there is still a messy layer connecting user requirements to actions on a computer. If there was not messiness then we could just automate it all. Approaching software from some sort of Platonic ideal of what software should be will frequently lead to bad decisions on it's own.
When you start to see how certain pressures lead to certain paths you learn to recognize the wrong decisions that are often good at the time, and avoid them. At the same time, you need to learn to develop methods that work quickly and effectively. By far the biggest real challenge in real world software is time constraints. This is almost never discussed in theoretical views of software, but the truth is you're always going to be writing code under pressure to ship. You will come across situations where you do not have time to do what you want to do or think is best.
Good software is software that runs and solves the user need, but you will come to realize that there are design solutions that will make successfully running happen more often. The best way to find these is to study the real software you're writing.
What if the question is born out of the insight that his team's software is poorly designed, motivating the desire to get outside input?
What if the question is asked by a college student?
Then it still matters a lot what kind of codebase he is working on. A web project is differently structured and done than the codebase for a embedded operating system. There are differend standards and practices in the industry.
I learned a lot by just stepping through the code with the debugger of libaries I used. That brought more practical insight while learning about design patterns etc. In the end, it is all about patterns. Finding the right pattern for a given problem.
I would be interested in recommendations for those who have a poor example or no examples
In my case, I'm a junior engineer that has recently been given more responsibility designing aspects of our product. I'm just trying to learn all I can so my designs will be good!
On forums like this people have a frustrating inability to answer the question asked.
I haven't considered time to implement as a metric for evaluating design, but it makes a lot of sense.
I definitely learn so much from my team's codebase. Most of what I learn is either from the good designs I see in there or from my googling trying to fix the not so good parts.
These are several years old at this point, but many open source project leaders contributed to the series "The Architecture of Open Source Applications", which is free to read online: https://aosabook.org/en/index.html
Thank you!
It's been asked a few times, here are some links to get you started:
* https://news.ycombinator.com/item?id=36370684
* https://news.ycombinator.com/item?id=30752540
* https://news.ycombinator.com/item?id=9896369 (Python specific)
Thank you!
I am not qualified enough to answer, but some ~15 years ago I enjoyed going through the book Code Reading[0], which is about this exact topic.
I think there was another one with a similar name but I can't think of it's name.
0: https://www.spinellis.gr/codereading/, check the TOC https://www.spinellis.gr/codereading/toc.html
My experience (30 years in software, 25 in practicing architecture, MIT system architecture masters) tells me there is no such thing as abstractly “good” design. There are designs with negative consequences for sure, but “good” depends on the context: what are you building, safety/security requirements etc. Probably most importantly on the implementation team and it’s structure. A team of juniors will butcher your intricate design and Conway’s Law makes your software reflect the team.
Yeah, as I've been learning more about software design, it's become pretty clear there is no silver bullet. It would be nice though. I find it a bit overwhelming to try to find the best solution for a problem when there are so many different architectures and programming paradigms.
That being said, taking into account the requirements does eliminate quite a few of the options. Right now, I work on safety-critical embedded systems which requires us to make some decisions that would most likely be way different in other environments.
It's about having most compromises be Not Totally Stupid and have the worst ones only be locally stupid. There is a big universe out there of compromises that fulfill those criteria. As you imply, a lot of the time it's more about constantly fighting for Less-Bad-and-avoiding-Dumb than about architecture as such.
The codebase is not where the design usually lives. It's where the implementation lives. You could imagine a rewrite into another programming language which would preserve the design but completely replace the implementation.
You should practice writing design docs. Don't worry about what the doc is supposed to look like, and definitely don't work off of a template. The most important thing about the doc is that another human could do the implementation if you gave it to them.
The doc can also function as a "proof of consideration". If you choose to do something one way, but there are other possible ways to do it, you can acknowledge the other possible ways, and say why they are worse. By preemptively acknowledging an alternative, you have proved to readers that you considered it.
All a "good" system designer is doing is considering a larger design space than most, and consistently finding good points in the space. Pick a problem, sample points in the design space, tell me why some points are better than others, and write it all down.
I made a small list long while ago, but it still holds today
https://medium.com/@012parth/what-source-code-is-worth-study...
Thanks! I appreciate it!
There's a free book on this topic: The Architecture of Open Source Applications
https://aosabook.org/en/index.html
Maybe that would be a good start. You can then pick a project to dive in.
As a more specific tip, I've done some hacks in Nginx long time ago and found it quite nice.
Trying to learn software design by looking at code is kind of like trying to learn about architecture looking at the bricks that make up a building.
To learn about design you need a wider perspective. You can theoretically learn it from code but it won’t be most effective. Look at great documentation and literature about design instead.
Yeah, I guess without the context I wouldn't really understand why it was designed that way. Or know what situations a design like that would be good for.
Is there any documentation or literature that has helped you?
Once upon a time there was an email MTA (email server) called qmail. It had an architecture that was surprising and Daniel Bernstein documented his philosophy. It left an impact on me by showing me how we can rely on the OS-supplied facilities to accomplish a lot. I don't think qmail is some shining star that will teach you everything you need to know about software design. It is an interesting case study to add to your repertoire.
See https://en.wikipedia.org/wiki/Qmail
I think there is value in groking entire code bases. It's not just about whether or not they are well designed though. It is an important skill to be able to analyze and see the big picture of how things work together in large systems. For me it often involves drawing diagrams (sometimes UML) to map things out. Being able to view systems in this way is a pre-requisite to intentionally designing your own systems at this level. And yes, once you can work at this level you can learn from good designs, but also see problems with bad ones.
EDIT: and to answer your question, if you're working on something that is "like X but different" then read the source code for X. You could also look at source code for software that you use from day to day: software where you already know what it does. For example, if you're in web, maybe the web framework, or web server, if you write python, maybe a core library that you use, or maybe the python interpreter, or if you use vscode ..., if you use android ..., you get the idea. At the start I would suggest smaller programs, and programs where you already know the domain (e.g. cpython might not be the best place to start if you never implemented an interpreter before, you may spend more time learning about interpreters than the design of this one, still a good thing to learn of course.)
I can recommend reading about postfix architecture if you want to learn a bit about what would nowadays be called a microservice architecture:
https://www.postfix.org/OVERVIEW.html
You might need to know a bit about how email servers work to appreciate it though.
Well maybe I'll take some time to learn about email servers then. I use email everyday so I suppose it'll be good to know a little more about how it works.
This has 2 key things:
* "well designed": What was the objectives and ideas...
* "codebases": How well that was implemented
They are a lot of lofty claims saying how this or that is "fast, secure, etc" but don't end like that in the actual implementation.
But most of the time, that could be seen in the "design claims" already! Good design is not just full of adjectives and nice sounding goals, but the concrete considerations, what was the trade-offs, false-starts, and reasons behind the decisions.
You can see some examples reading about the design of Erlang, early pascal, most RDBMS, etc.
So, you first mid/long term goal is to learn to distinguish what good design actual look like.
then, in relation with codebases then to be kinda easier: It actually follow the design?
A good example is the 'std' library of Rust. It has a lot of lofty claims about security and such things that could sound alarms, but then you dive in the code of it and see is there A LOT of care about it, and a lot of docs comments discussing this stuff and then the code match.
P.D: The "std" or equivalent of the lang is one of the most important codebases you need to learn and study, and the MAJOR way to judge how truly good is it.
Start with the question: what is the problem that you want to solve? Next, find codebases that solve that problem and study how they do it.
Good design is so deeply tied to the domain details. Wonham's Internal Model Principle applies to code.
Example: I wanted to solve the problem of unit testing for embedded targets. I found open source projects that do this and read the code critically to see how and why it is written. As I build my own approach, I revisit theirs to learn more as my understanding of the domain deepens.
While studying well-designed codebases is incredibly valuable, there's an important "tip of the iceberg" effect to consider: much of good software design lives in the "negative space" - what's deliberately not there.
The decisions to exclude complexity, avoid premature abstractions, or reject certain patterns are often just as valuable as the code you can see. But when you're studying a codebase, you're essentially seeing the final edit without the editor's notes - all the architectural reasoning that shaped those choices is invisible.
This is why I've started maintaining Architectural Decision Records (ADRs) in my projects. These document the "why" behind significant technical choices, including the alternatives we considered and rejected. They're like technical blog posts explaining the complex decisions that led to the clean, simple code you see.
ADRs serve as pointers not just for future human maintainers, but also for AI tools when you're using them to help with coding. They provide readable context about architectural constraints and compromises - "we've agreed not to do X because of Y, so please adhere to Z instead." This makes AI assistance much more effective at respecting your design decisions rather than suggesting patterns you've deliberately avoided.
When studying codebases for design patterns, I'd recommend looking for projects that also maintain ADRs, design docs, or similar decision artifacts. The combination of clean code plus the architectural reasoning behind it - especially the restraint decisions - provides a much richer learning experience.
Some projects with good documentation of their design decisions include Rust's RFCs, Python's PEPs, or any project following the ADR pattern. Often the reasoning about what not to build is more instructive than the implementation itself.
Oooh I like that idea. I may steal it. I'll be on the lookout for documents like that. It'll be interesting to see what patterns/designs were avoided. There are so many ways to accomplish the same thing it might be nice to set some limits.
I'd recommend the book "Beautiful Code: Leading Programmers Explain How They Think". Published by O'Reilly, ISBN-10 : 0596510047
I'll check it out, thanks!
>well designed codebases
One thing to keep in mind is that what was well-designed 30, 20 or 10 years ago may not be considered such now. Hardware changes and so do the design decisions involving performance.
For example, if you are looking at C++ networking libraries, learning from ACE or even Asio may not be the best idea - better look at "thread per core, share nothing" Seastar[0].
Another thing is that it may be better to read design docs, not the code. For example, the rationale for the mold linker design[1].
[0] https://docs.seastar.io/master/tutorial.html#asynchronous-pr...
[1] https://github.com/rui314/mold/blob/main/docs/design.md
The MIT course for Operating Systems has a teaching OS called xv6. There's some nifty implementations in there. I'm not sure I'd comment on the codebase itself, but I think more importantly it teaches you good abstractions which Unix is all about: https://pdos.csail.mit.edu/6.1810/2024/xv6.html
> that I study existing well designed codebases.
Proceed with extreme caution. This is highly subjective and often results in a blind leading the blind scenario. For example how do you determine which code bases are well designed if you yourself are not comfortable determining such?
Instead, heed the guidance of some of the other comments and just start building things. Plan on failing and solving for the failure. That’s called practice. With enough practice you will know what works well for you. If you are measuring things along the way you will also know what’s slow.
To answer your actual question:
The codebases I learned from the most are Git, Postgres, CPython. Not saying they are perfect designs, but they are well maintained, solve hard problems, have seen many years of evolution, and are very easy to get your hands on.
For website/web server learning I recommend the laravel code-base. It's a beauty.
https://www.instagram.com/reel/C2x4Ge5RtNC/
I've done this for years and swear by it.
Top 5 codebases for changing my mind about things:
Wietse Venema's Postfix mail server. Taught me tons about security posture, the architecture i'd describe as microservices before microservices was a thing, but contrary to the modern take on microservices (it's mostly a tool for decomposing work across large semi-isolated groups) this was primarily about security and simplicity.
Spring framework - this opened my eyes to ways of working that i hadn't really thought enough about before, the developers on that project have a culture of deeply considering the needs of their users (who are java developers often in an enterprise environment).
Git - the thing i like about the git code base is that once you've covered the objects database (e.g. blobs, trees and commits) and the implementation of refs, everything else just feels like additional incremental features. With those core concepts, everything else is kinda harmoniously built on top.
Varnish by Poul Henning-Kamp is another one - feels like he went to great lengths to make that code base a teaching tool despite the fact it's also a top tier reverse proxy.
Last one isn't a code base - but it will help with software design in the large; studying how the lieutenants model works in the linux kernel.
Thinking about my answers, i think i've highlighted something subtly different than "well designed codebases" it's more a list of codebases that left a notable long lasting impression on me because of design decisions they made.
Bitsquid didn't prevail but personally I thought the software design was spot on. If you're into this kind of project. (Game engine)
https://bitsquid.blogspot.com/2009/09/welcome-to-bitsquid-bl...
Highly recommend you read this book series (all free online)
The Architecture of Open Source Applications
https://aosabook.org/en/
Then afterwards diving into a repo based on the map given to you by the textbook.
https://github.com/newspeaklanguage https://newspeaklanguage.org https://blog.bracha.org
I dont think this is a good idea, actually. Good design is about making good trade offs at the right point in time.
A code base with excellent design will show you the end state but not how it got there but probably not the trade offs and decisions involved.
Practicing refactoring on subpar code bases and dealing with the consequences of your decisions is a better way to improve.
Are there any books or websites you'd recommend I read to learn more about design instead? I 100% need to learn by doing, but wouldn't it also be good to learn what has already been discovered so I don't have to reinvent the wheel? Or is it best to discover good design on your own?
I learned on my own. If I were to recommend anything to read it would be to find a good book on refactoring or dealing with legacy code.
I read design patterns books when i was younger but in retrospect that was a hindrance more than a help.
https://aosabook.org/en/
Go the other way.
Study and fix shitty code.
Find college students who wrote a web server and put it on GitHub for the world to see.
Find all the bugs and then see how many integer overflow vulnerabilities can be exploited to crash it … or worse.
Your own. Everyone else's is trash. Never understand someone else's code-- the whole point is that there should be just a handful of functions of yours I can call, and shouldn't ever care to see how the sausage is made. Otherwise, your abstraction sucks
I agree that the internals of my code should be abstracted away from my users. But don't I still want the internals to be well designed so they are maintainable and extendable?
https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
Whoa, I didn't know something like this existed. Thank you so much! This will make it way easier to check if something has already been asked.
Doom 3: https://github.com/id-Software/DOOM-3
I should do this more too, specially the complete auth/multi tenant solutions.
Go code is easiest to read.
One way is to become an apprentice with an experienced plumber - ideally skilled in hydronic heating.
study modern code bases of language you want to learn, and look first at their history to see if they are mature, came from a company, academics etc.
you should not only study 'good' code... how will you know what is bad code?
study code that does similar things to what u want (client/server/game/ai/datacrunching etc.) and study lot of it..different qualities, ages, and sources
My suggestion is to search for open-source codebases in your favorite languages, study them, and start practicing them.
I'm definitely going to do that. I asked because I wanted to start off with some that I'd knew would be well designed. However, it would probably be more interesting to see a wide variety of quality than just the good ones.
Or maybe open source projects are all usually well designed. I haven't looked at many in depth. The only one I've really looked into was Clang to try to figure out why clang format ignored my style rules when styling files of certain names. (Turns out there is a list of file names that automatically are considered Objective-C rather than C++)
One of the best is actually the implementation of the Tcl language. It's highly readable C code, very well structured (it was initially by Ousterhout, after all).
I also found the Tcl implementation to be easy to read and understand.
You understood what I was trying to convey.
Yes, all open-source projects are well designed with documentation and to-dos, otherwise open-source contribution becomes difficult.
As far as cryptography libraries are concerned, Monocypher.
pytudes: https://github.com/norvig/pytudes
Yanderedev source code
codemirror 6 is the product of a bunch of redesigns, and this most recent version seems to have had a lot of thought put into its structure.
UNIXv6 and the BSDs.
First do, then look.
Ones I like:
Spark, Cassandra, Beam.
There's no. Most of production codebase outthere is result of hacks.