Making technology choices

12th Nov

Back in 2010, the adage “the right tool for the job” was prevalent within the developer community, but by 2015 it was seen by some as motivated reasoning and in general harmful advice. Nowadays it’s common to hear that we should “Choose Boring Technology” and "optimize our engineering decisions for iteration speed".

My view is that the adage “the right tool for the job” suffered from being transmitted by engineers in an almost folkloric manner. It’s a truism (“obviously you want to pick the right tool!”) but in practice was often wrong since shoe-horning an old bit of common sense advice into the domain of software engineering gave licence to the self-interests of software engineers who could avoid thinking about its costs (operational complexity, training/ramp-up costs of employees, etc).

On the other hand, “Choose Boring Technology” treats technology choices as if the only system they interact with is a technical one and human incentives can be disregarded — it pays no attention to why “the right tool” was so-often endorsed, and how this related to the polyglot movement (both were, in a sense, marketing that allowed startups and employees that wanted to chart uncharted waters to find each other). Boring is also a misnomer — what is meant is “well-understood” but the word choice is being used to frame the domain as one in which technologies can either be “shiny” or “boring” — which isn’t the primary frame that I'd use to evaluate technology.

While reducing operational complexity, using technologies that are flexible and avoiding unknown-unknowns are all important and suggest that “Choose Boring Technology” is a sensible approach, we should also pay careful attention to whether our choices result in happy developers. Retention is important to the long-term maintenance of knowledge within a company, and developers that don’t feel like they’re learning anything new or that feel stifled by centralised decision-making over the tools they’re allowed to use can end up leaving due to feeling disrespected and disenchanted. On the other hand, there’s also “retention risk” in attracting developers that seek shiny technologies as once these technologies have lost their shine they can leave just as eagerly as they joined. Additionally, in many cases there are technologies that offer an order of magnitude faster/better ways of doing things, and so the best teams will make judgments over when to let their engineers invest their time seeking improvements.

Unchecked restraint in technology choices can eventually lead towards obsolescence and brain drain. Following “The Golden Path” and creating a team to centralise technical decisions is a good idea when a company reaches a particular scale, but you must also work out how you’ll create affordances for developers outside of that team to go above the predefined baseline of quality (for a similar idea read Lethain's "Providing pierceable abstractions").

There are also differences between individuals and differences due to scale (personal project, team, company-wide, etc):

At the individual level, while I might reach into my toolbox and find that my most well-worn tool is JavaScript and use that, another might reach into their toolbox and pick Ruby. We should use what we know best. But what if you’re just starting out? If I was a beginner I would pick up-and-coming general-purpose technologies that are growing in usage. While I might benefit from prior knowledge in an older ecosystem, I will learn more by being part of a highly-engaged community of people trying to achieve state-of-the-art approaches.

But at the scale of a company, there are further problems to be aware of due to the difficulty of migrating software stacks and a tendency for policies and processes that don't work well to be left unfixed.

What should we recommend to a company that has always written both their application and database logic using SQL Server Stored Procedures? The boring choice might be for them to continue extending and creating new software in this way, but since there is very little beginner interest in learning that stack eventually they will run into hiring problems and brain drain caused by an ageing workforce (institutional knowledge loss). To survive, they will need to rework their system in a way that gives them access to a more fungible workforce, and they will need to do this without interrupting the business.

Another common problem at large companies is for there to be policies and processes around evaluating and recording technical decisions but for these to not achieve anything other than getting developers to copy-and-paste the marketing claims of a new technology into a document without consideration of its context/impact within the organisation. These policies can often be empty ceremonies with no impact on the actual decision-making process that allow developers to rubber-stamp their desire to learn new technologies.

So how would I make technology choices?

  • Use well-understood tools. Either those you personally know well or those that are known well within the broader technology community.
  • Use the smallest set of tools you can to solve your problem(s).
  • Pay close attention to the happiness of your developers. I'm not suggesting that you should always let the whims of developers dictate a company's overall technology choices, but you should be aware of the market incentives of your employees. In any case, an important job of engineering leaders is to find ways to carve out interesting work from within boring/legacy systems.
  • Benchmark yourself against you competitors but if a problem is core or complementary to your business look for ways to use technology to gain advantage.
  • Try to find the line between policies that are too onerous for developers and those that are not rigorous enough to provide any benefit to the organisation. Non-rigorous decision making processes are really common and generally when a policy isn't providing its stated aim it should be gotten rid of. However, if we want to improve decision-making within a large company this can probably be done with structure, asynchronous collaboration and good guiding questions (e.g. Can this be achieved with technologies we already use? What if true could invalidate the benefit given by this technology? Will this require a migration? Who will operate this? Etc).