Ask the Tech Lead: How should I approach simplifying a complex system?

I’m writing a series of these posts, discussing the unwritten advice for excelling in highly technical leadership and tackling some of the hardest questions I’ve faced in my time as a lead engineer for teams and groups at Yelp.

The question: Why is my code so complex? How can I fix it?

My team owns a pile of code that has a bit of a reputation for being extra confusing and hard to work with. We often experience our Product Managers rattling off “simple” feature changes that should be easy but we know would take months of effort to execute on. I want to simplify the system but it’s a mess and I’m not sure where to realistically start.

How do I identify what parts of my system are essential complexity vs accidental complexity? And once identified, how do I remove accidental complexity?

The solution: Learn more, gain context, find opportunity, then execute

If there’s one lesson I’ve learned over and over again, it’s that complexity is the death of a software system.

Sometimes the complexity is hard to avoid (IAM permissions are horribly painful, but also extremely expressive and powerful!) but often the complexity is accidental, coming from the normal wear and tear of time, the organic evolution of requirements, and (most frequently) a lot of people making good local choices in one part of the system while failing to contain the whole of the complexity.

A large part of my job as a technical lead is applying wise, targeted back-pressure to contain the increasing complexity of the software my group produces. By default, all systems tend toward a tangled mess of code that nobody understands.

I get excited when I see a project proposal where there’s a huge gap between the complexity to fully explain the feature (it might take an expert ~10 minutes to give you all the details) and the complexity of the underlying systems of code (I’ve seen cases where all the experts in a single room couldn’t collectively give a coherent explanation of the current implementation no matter how much time they had).

An engineering lead’s role is to ask (pointedly and repeatedly): “why can’t this software be as simple to own and operate as it is to explain?”.

Learn the “why” behind your system’s complexity

The answer is nearly always extremely interesting and informative — get curious! A few reasons for complexity I’ve heard before:

• It integrates with $ANCIENT_SYSTEM and that system is very confusing and gnarled. (How can we either not rely on that system, or simplify the interaction model between them? Lots of complexity comes from the naive or outdated integration seam between two systems. Does an interface need to be rewritten to hide its implementation details?)
• I don’t know why it’s so complex. (Ah! Probably our complete lack of understanding is hiding some opportunities for simplifications. Dig deeper and ask “why” recursively until you get some hard lessons. Low hanging fruit often lives in the shadow of lack of knowledge)
• It is complex because this monolithic codebase is big and terrible. (Now we’re getting down to brass tacks. What is keeping this feature in the monolith? Is there a game plan to move it out we can help execute on with this project? Is there a way we can partially migrate in the right direction and actually speed up the delivery of the product win we’re building toward? Get curious!)
• My team’s part of it is simple, but $OTHER_TEAM’s half is a mess. (Be very cautious here. Sometimes your team’s side is simple because you abandoned complexity to the other team’s half. Sometimes the interface between teams can be easily reworked to reduce complexity for both sides, if only you discussed it! Dig deep, learn more, stay curious, and fight cynicism.)

Why curiosity works

You might be noting that “curiosity” is a key part of how I discover and dispel unnecessary complexity.

One non-technical reason I like is staying curious is it helps me avoid the danger of the Fundamental Attribution Error: if I purposefully react with curiosity when I don’t understand, I can counteract the normal human temptation to be “sure” they’re wrong (or worse, stupid!).

Without some sort of active strategy to prevent it, it’s disturbingly easy for us all to fall prey to the idea that we’re the only ones who have thought through a problem “properly”. Reminding myself to stay curious forces me to actively engage with people who disagree with me, dig into why they feel that way, and very likely learn something important along the way.

It’s not just about what I’m learning though. Being authentically interested and open to someone else’s opinions makes the projects you’re a part of more welcoming and inclusive for everyone involved. In the most extreme cases, being closed off or hostile to dissenting views can completely warp the way people interact with you. Actively being curious instead of dismissive of other views does wonders for encouraging contribution from everyone involved in a project. Encourage participation!

Curiosity also helps me build a mental model of the “why” behind the systems we have. Telling the difference between “historical quirks” and “crucial design or product decisions” is incredibly important for navigating the complexity of large production systems and the only way to know which is which is by digging deep and asking questions when you start projects.

No silver bullet

There’s no single bit of advice for diagnosing and fixing unnecessary complexity in systems. But I believe very strongly that in nearly all cases, the best thing you can do to improve the state of your code is to stay curious and learn more.

I’ve lost count of the number of times a deep dive into a system (and the people who use it!) has revealed opportunity for improvement. Applying this approach for the last couple years has produced multiple projects where we took years of accumulated complexity and reduced it to something much closer to what the feature actually required.

Engineering thrives when we can produce systems that are as close to their essential complexity as possible. Shine a light in the dark corners of your codebase and you’ll be amazed at the volume of easy-to-fix technical debt you find.

If you found this interesting, follow Scott Triglia here or on Twitter (https://twitter.com/scott_triglia).