From Paul Lutus, Confessions of a Long-Distance Sailor
“After I was overruled, I wrote a letter of resignation. In my letter I explained the risks and said I would rather resign than allow dangerous hardware onto the Shuttle. And I pressed my letter into more hands than absolutely necessary. The embarrassment level got pretty high and the managers backed down — I was allowed to make the necessary changes.
Does this make me a hotshot, a moral person? Not really. I wasn’t married, no children to support, I could afford to lose my job for a principle. Besides, I was able to imagine what would happen to me if I caused a Shuttle failure — kids on the street saying ‘There’s the guy who killed all those astronauts and the schoolteacher.’
I didn’t become an engineer just to design things. I wanted to design them right. I was a bit too idealistic for engineering profession, who’s motto is “Ship it.” So I changed careers — I got into computer science.”
This was written more than twenty years ago, and it’s funny that the term “ship it” was as connotative and full of meaning as it is today in the midst of the lean startup movement. When I read this in Paul Lutus’ very engaging sailing travelogue, it made me think about the responsibilities of engineers and programmers to the people who depend on their software. I think this is on the mind of a lot of software developers.
It doesn’t feel like a week goes by without millions of emails and accounts being hacked or a software bug causing massive downtime in a stock exchange or airline. Software’s increasing potential for good is only matched by its equal potential for harm. Testing methodologies and its culture have advanced a great deal in the last decade, but most would agree that its no panacea. Instead some argue the deeper solution is in the languages and idioms we use that make writing code safer (my favorite being Rich Hickey’s philosophies). Yet this too can never protect us if the person who writes the code doesn’t (1) see the potential for harm in their work and (2) does not feel the moral responsibility of saying no to “ship it.” It obviously can get quite messy when livelihoods and politics are at stake, and I doubt we will not all struggle with it in our careers.
I’ll leave with an excerpt of an interview with a famed Japanese woodworker, Toshio Odate:
“Toshio added something very important: there is something else that is not always being grasped by many woodworkers in the craft: the social responsibility of the craftsperson, be they woodworkers, musicians, photographers, doctors, or writers. Each of these persons practices a craft and in that craft they are expected to produce a result that carries with it a social responsibility. And that responsibility is where the person’s skill and even artistry must be used to serve others. For example, if a joint is used to show off a person’s ability to create a showy piece, but fails when it comes to joining two pieces of wood securely and efficiently, that person has failed at their responsibility to society—even if the joint “looks beautiful.” But the craftsperson who makes a solid joint, that looks “good enough,” does its job and holds for decades or centuries to come—that person has fulfilled the responsibility society asks of them. Even if that joint is hidden, it has the spirit of being a good joint…
…once you commit to making a piece for a client, or a family member, that responsibility is there, to those people. It’s your job to make sure your design and your workmanship serve the needs and desires of your clients, and that the techniques and materials you use serve those ends. Anything else is superfluous, and runs the risk of being dangerous, or at best, ugly.”
From Fine Woodworking.
MR. FITZGERALD: What did you advise them? When you were advising the students on how to make a living at this, what would you tell them? How to sell – how would they sell their furniture?
MR. KRENOV: Well, my standard formula was the better work you do, the more chance that you’ll starve.
From Oral history interview with James Krenov, 2004 Aug. 12-13
It’s hard being a developer and not becoming defensive about what you build. Just like a writer or musician, we invest a lot of ourselves in what we create. This is natural but not always helpful, and sometimes it’s downright fatal. So we try to internalize and practice being a good listener when a customer comes knocking on our door with an “idea” or “bug” to report. I use quotes because we all know that some users are, um, not to be taken too seriously. So there’s the rub, cynicism can grow and eventually the day comes where we don’t take any of our users seriously and either they leave us or the bug they tried to tell us about finally does us in. So what’s a developer to do?
Well, it’s your lucky day! After years of battling the contradicting emotions of loving and hating my users’ feedback, I’ve derived a formula representing how much credence you should give them. Take yourself out of the equation, and use this to bring balance to how you listen to a customer’s feedback. The variables above are elaborated below:
*This being the internet, I feel obliged to mention this formula is a joke, but hopefully the motivation behind having an objective process to weigh user feedback isn’t.
“Not long ago someone wrote from far away to ask me, ‘Should I make compromises?’ I read and reread the letter, realizing there wasn’t much I could say to my friend. I ended up by simply suggesting that he try to keep this problem secondary. Let the work have primary importance, let it answer the question. Don’t allow the matter of compromise to become more important than the work. This means the way you work, the things you do and how you do them, your shop, the way you live are in some sort of accord within yourself. You develop a way of living, and if that conflicts with commercial interest, then you must sort it out. Either you adapt your work to commerce or you solve your money problems another way and keep your work what it should be for you. As I’ve said to myself many times before: Try to live the way you are, be the person in your work that you are in the rest of your life. Easy to say!”
James Krenov, The Impractical Cabinetmaker
I think some of us miss the more plausible definition of what’s commonly referred to as the “10x” or “100x” programmer. When I first heard the term, I imagined a mathematical genius speaking code, half human and half machine. There certainly seem to be those folks who are far better and faster programmers as I’ll probably ever be, but I don’t think this fully covers the 10x/100x population.
In my experience, the true astronomical gains in productivity occur not when the programmer produces more code or builds more features faster than everyone else. It’s instead when they make good early design decisions, and don’t produce latent catastrophes. Company productivity often grinds to a halt when they have to pay technical debt that was sown months or years earlier by seemingly smart and productive coders. Like most debt, it tends to compound and amplify over time and the scale of an organization.
From this perspective, you too can be a 10x or higher programmer not just by producing more widgets faster than everyone else, but by producing widgets that stand the test of time and minimize debt. The trick, I think, are the early decisions you and your team make at the beginning of each coding cycle.
Reading the recent article on Markus Persson, I easily related to his recollection of how he started programming by typing snippets of code from magazines. When I was a child, my dad did the same on the family Atari, followed by me trying to type the few english words I knew at the time into the command line to see if they’d result in more spectacle. They did not, but my curiosity had been piqued.
Remembering that isolated programming environment in my family’s living room provided a stark contrast with the present software ecosystem. Today we live in the world of web services, a proliferation of open source libraries, and rapidly evolving platforms, standards and devices. The pace of innovation in software and the sheer prolificness of the community is truly humbling. Yet this interconnectedness and leveraging of power introduces its own costs, namely fragility.
From working atop a high rise of virtual stacks we know little of, to the fact that a piece of hardware’s shelf life is measured in months, as software developers, we are at the mercy of innovation, not just empowered by it. Every programmer can remember a time where a library call was unexpectedly deprecated, a crucial service call was out of commission, or a hardware platform no longer supported by an OS. All dependencies outside your direct control.
If you run your own servers, the situation is not as dire, but mobile and browser development is now dictated by users obtaining “manufacturer” upgrades and features more or less in real time, independent of your best drawn out plans.
The positive effects are well known: faster innovation, more features, better for users, less need for backwards compatibility etc. But I don’t see too many people mentioning the flip side of innovation in the era of interconnectedness, that of disposability, risk and the increasing costs for software maintenance. In my own experience, there have been numerous times where feature development has been significantly put on hold to test new iOS versions and code around deprecated library calls. At times it looks like there is a ticking time bomb inside all our code, it’s just a matter of when it’ll explode. So we wisely channel more and more resources to keep up with our dependencies. It’s a chronic situation, punctuated by acute episodes of pain.
I guess the lesson is to never get comfortable or even entertain the notion that your code is like a bridge or building that will stand the test of time. The only way to mitigate it is to embrace best practices in design, testing, and engineering. My feeling, which is a limited one obviously, is that software maintenance costs aren’t going to go down anytime soon, if anything they feel like they’re going up.
Never saw this coming, but it all started when I got tired of buying disposable furniture and looked for a sofa that wouldn’t break my back. The more I learned about furniture construction (tip: only buy a sofa with high density foam graded at 2.2lbs or more), the more obsessed I became with the history and craft of woodworking, particularly the hand tool variety.
Fast forward a couple months, and I’ve now completed a weekend course at a private workshop and just started a 10-week evening class at a city museum on using hand tools. All this has made me wonder, what led to this weird obsession?
Last week I overheard our instructor telling a student that a lot of software engineers sign up for the courses because “it’s so different than what they normally do.” This is obviously true, in woodworking you do physical work. You get gnarly blisters and you’re mostly on your feet. Couple that with precise measurements and the engineering aspects of furniture design, and you get a nice hobby and exercise for a soft handed programmer who sits too much.
Yet that explanation, as a sort of break from a day job or physical exercise didn’t really feel like the whole story. What I really feel is happening is we crave a true craft, and no matter how much people refer to it as such, programming is not a craft (curiously, mathematicians rarely feel the need to call theirs such a thing).
Data structures and algorithms can be beautiful but we don’t use our five senses to really construct them, we use our minds and work with 99 layers of abstractions over the circuit board. Which is fine, but it’s not exactly comparable to the notion of a traditional craftsmen wandering their hands along a few beautiful cocobolo logs, smelling the sawdust, and hearing the hand plane shave off hair thin shavings. What’s the difference in this picture? This woodworker is seeing, smelling, touching, hearing, (and if they really desire it, tasting) their raw materials. That’s almost all the five senses, and the layers of abstractions/separations are no where to be found! If ours is indeed a craft, it is a deprived one.
Famous woodworkers like James Krenov and George Nakashima strived to “work closer to the wood.” There is an inherent positivity to working and living close to our senses, maybe something along the lines of truth and wisdom, but that’s just a guess.
If nothing else, our senses need to be nourished.
In no particular order:
1. When performance is an issue, if you can calculate or process it at the application layer, then take it out of the database layer. order by/group by are classic examples. It’s almost always easier to scale out your application layer than your database layer. As true for MySQL on your server as it is on the sqlite in your handheld. EDIT Some great comments on HN for this one so I felt like I better clarify: we only do this for certain queries not to improve necessarily the client response time, but to relieve load if the query is battering the DB and making it a significant bottleneck for ALL clients.
2. Concurrency, avoid it if you can. If not, then remember that with great power comes great responsibility Avoid working directly with threads if you can. Work at a higher level of abstraction if possible. In iOS, for example: GCD, dispatch and operation queues are your friends. The human mind was not designed to reason about infinite temporal state—I get nauseous thinking about how I learned all this first hand.
3. Minimize state as much as possible, and keep it as localized as possible. The functionalists were/are onto something.
4. Short composable methods are your friend.
5. Comments are dangerous since they can get out of date and mislead, but so is not having them. Don’t comment the trivial, but strategically write paragraphs if needed in specific sections. Your memory will fail you, as soon as tomorrow morning, even after coffee.
6. If you feel one use-case scenario will “probably be ok”, that’s the one that’s going to lead to catastrophic failure a month in production. Trust your paranoid gut, test and verify.
7. When in doubt, over communicate all concerns with your team.
8. Do the right thing—you usually know what that thing is.
9. Your users aren’t stupid, they just don’t have the patience for your cut corners.
10. If an engineer is not tasked with the long term maintenance of the systems they build, view them with suspicion. 80% of the blood, sweat, and tears of software occurs after its been released—that’s when you become a world weary, but wiser “professional.”
11. Checklists are your friends.
12. Take initiative to purposeful enjoy your work, sometimes this will take effort.
13. Silent failures, I still have nightmares. Monitor, log, alert. But be wary of false positives and the inevitable desensitization it leads to. Keep your system senses clear and alert.
14. At the end of the day, we’re paid to manage complexity. Work accordingly.
*Side note: talks by Rich Hickey and Clean Code by Robert Martin have been very positive recent influences on my work.
“Personal work habits should be forged and tempered like fine steel tools until they are effective, are performed with control, and are as safe as possible. Then, they can be enjoyable and result in good work.
Good work cannot be produced unless the work is enjoyable; making a thing should bring pleasure, not pain. The work must never be rushed to completion, but done with care and patience. Above all, one should truly enjoy one’s work. It is important to remember that wood is a living thing—one must understand its properties and not oppose them. Whenever work is forced, it either ruins the piece, damages the tools, or injures the worker, so work must always be done with the utmost care.”
-From the “The Complete Japanese Joinery”
The results are a bit surprising as I’d expected them to have much less variance than they do. If you haven’t already, feel free to take the survey and the results below will update accordingly. One point this reinforces is that there is no single ideal culture for technology companies to strive for (at least given the attributes below). Every company wants to attract smart people who can get things done, but there are many different cultures that can do that. Given the current entrepreneurship renaissance and intense competition for talent, startups shouldn’t all rush to get a ping pong table and sponsor nightly beer soaked happy hours. There are many different types of engineers, and companies can use that to their advantage and nurture a culture that is truly their own.