Ask-a-Tech-Lead on Locally Optimal

Ask the Tech Lead: How should I approach simplifying a complex system?

Fri, 14 Dec 2018 01:08:01 +0000

I’m writing a series of these posts, discussing the unwritten advice for excelling in highly technical leadership and tackling some of the hardest questions I’ve faced in my time as a lead engineer for teams and groups at Yelp.

The question: Why is my code so complex? How can I fix it?

My team owns a pile of code that has a bit of a reputation for being extra confusing and hard to work with. We often experience our Product Managers rattling off “simple” feature changes that should be easy but we know would take months of effort to execute on. I want to simplify the system but it’s a mess and I’m not sure where to realistically start.

How do I identify what parts of my system are essential complexity vs accidental complexity? And once identified, how do I remove accidental complexity?

The solution: Learn more, gain context, find opportunity, then execute

If there’s one lesson I’ve learned over and over again, it’s that complexity is the death of a software system.

Sometimes the complexity is hard to avoid (IAM permissions are horribly painful, but also extremely expressive and powerful!) but often the complexity is accidental, coming from the normal wear and tear of time, the organic evolution of requirements, and (most frequently) a lot of people making good local choices in one part of the system while failing to contain the whole of the complexity.

A large part of my job as a technical lead is applying wise, targeted back-pressure to contain the increasing complexity of the software my group produces. By default, all systems tend toward a tangled mess of code that nobody understands.

I get excited when I see a project proposal where there’s a huge gap between the complexity to fully explain the feature (it might take an expert ~10 minutes to give you all the details) and the complexity of the underlying systems of code (I’ve seen cases where all the experts in a single room couldn’t collectively give a coherent explanation of the current implementation no matter how much time they had).

An engineering lead’s role is to ask (pointedly and repeatedly): “why can’t this software be as simple to own and operate as it is to explain?”.

Learn the “why” behind your system’s complexity

The answer is nearly always extremely interesting and informative — get curious! A few reasons for complexity I’ve heard before:

It integrates with $ANCIENT_SYSTEM and that system is very confusing and gnarled. (How can we either not rely on that system, or simplify the interaction model between them? Lots of complexity comes from the naive or outdated integration seam between two systems. Does an interface need to be rewritten to hide its implementation details?)
I don’t know why it’s so complex. (Ah! Probably our complete lack of understanding is hiding some opportunities for simplifications. Dig deeper and ask “why” recursively until you get some hard lessons. Low hanging fruit often lives in the shadow of lack of knowledge)
It is complex because this monolithic codebase is big and terrible. (Now we’re getting down to brass tacks. What is keeping this feature in the monolith? Is there a game plan to move it out we can help execute on with this project? Is there a way we can partially migrate in the right direction and actually speed up the delivery of the product win we’re building toward? Get curious!)
My team’s part of it is simple, but $OTHER_TEAM’s half is a mess. (Be very cautious here. Sometimes your team’s side is simple because you abandoned complexity to the other team’s half. Sometimes the interface between teams can be easily reworked to reduce complexity for both sides, if only you discussed it! Dig deep, learn more, stay curious, and fight cynicism.)

Why curiosity works

You might be noting that “curiosity” is a key part of how I discover and dispel unnecessary complexity.

One non-technical reason I like is staying curious is it helps me avoid the danger of the Fundamental Attribution Error: if I purposefully react with curiosity when I don’t understand, I can counteract the normal human temptation to be “sure” they’re wrong (or worse, stupid!).

I try to avoid imitating Homer in my professional life

Without some sort of active strategy to prevent it, it’s disturbingly easy for us all to fall prey to the idea that we’re the only ones who have thought through a problem “properly”. Reminding myself to stay curious forces me to actively engage with people who disagree with me, dig into why they feel that way, and very likely learn something important along the way.

It’s not just about what I’m learning though. Being authentically interested and open to someone else’s opinions makes the projects you’re a part of more welcoming and inclusive for everyone involved. In the most extreme cases, being closed off or hostile to dissenting views can completely warp the way people interact with you. Actively being curious instead of dismissive of other views does wonders for encouraging contribution from everyone involved in a project. Encourage participation!

Curiosity also helps me build a mental model of the “why” behind the systems we have. Telling the difference between “historical quirks” and “crucial design or product decisions” is incredibly important for navigating the complexity of large production systems and the only way to know which is which is by digging deep and asking questions when you start projects.

No silver bullet

There’s no single bit of advice for diagnosing and fixing unnecessary complexity in systems. But I believe very strongly that in nearly all cases, the best thing you can do to improve the state of your code is to stay curious and learn more.

I’ve lost count of the number of times a deep dive into a system (and the people who use it!) has revealed opportunity for improvement. Applying this approach for the last couple years has produced multiple projects where we took years of accumulated complexity and reduced it to something much closer to what the feature actually required.

Engineering thrives when we can produce systems that are as close to their essential complexity as possible. Shine a light in the dark corners of your codebase and you’ll be amazed at the volume of easy-to-fix technical debt you find.

If you found this interesting, follow Scott Triglia here or on Twitter (https://twitter.com/scott_triglia).

Ask a Tech Lead: I have to make a technical decision but I can’t know the right answer

Mon, 26 Nov 2018 22:31:48 +0000

Ask the Tech Lead: I have to make a technical decision but I can’t know the right answer

I’m hoping to make this a series of posts, discussing the unwritten advice for excelling in highly technical leadership. In the spirit of Camille Fournier’s excellent series, I’ll tackle some of the hardest questions I’ve heard from coworkers and mentees in my time as a lead engineer for teams and groups at Yelp. Huge credit to Jonathan Maltz for help refining this particular post and the overall format.

The question: How do I make an impossible technical decision wisely?

I’m struggling with making a choice about the technical direction of an upcoming project. This choice will involve significant expense: multiple engineer-months worth of work either way. I need to make a call, but I don’t feel like I have enough information to guarantee what the right direction is.

What’s the right approach when I need to make a big bet but have little guarantee of the right call in the end?

The Solution: Think like a scientist

Major decisions aren’t easy. There are many reasons a technical choice might feel impossible:

The decision is often between the status quo and some option we have no or little experience in.
Sometimes the best and worst case outcomes are either unknown or have a scary amount of variance.
Sufficiently new or different ideas often imply unknown challenges lurking. What if those unknown challenges are really bad and make it unappealing in hindsight?
New ideas might imply major architecture changes that are outright dangerous. If we fail to control risk, it may not matter how good a choice we make.

The reality is you still have to make impossible decisions, even if the right choice can’t be known until later.

My advice is to work your decision like you would a scientific experiment: deeply invest in learning about the problem, build a hypothesis about the right call, and then, most importantly, propose a series of small, incremental experiments to build confidence that your choice is correct.

As a whole, my approach to these big questions looks like:

Frame the big question and take an opinionated stance on the answer based on whatever data is currently available.
Come up with an initial experiment to partially vet that stance. It should be quick (~1 quarter) to accomplish, give meaningful directional feedback on whether the opinionated stance is still correct, and hopefully provide engineering leverage to test the next experiment more easily/quickly/safely.
Evaluate the experiment’s result. Does it suggest our opinionated answer to the original big question was right? Wrong? Have you learned something that needs to alter your answer to the big question?
If needed, update our best current proposal from this feedback.
Rinse and repeat steps 2–4 until you’ve answered the big question empirically.

Applying this to a concrete problem

In my own work, I’ve been digging into whether we should invest in using AWS Lambda more within Step Functions. Unfortunately we don’t yet have much experience using Lambdas and this would imply a pretty big technical effort to make the switch. Is it worth it? This is a big, uncertain question, so let’s see the framework in action.

Frame the big question, take a stance

We’ve been using an API-drive pull architecture up until now, but nearly all companies in the industry use Lambdas. Let’s pick the largest change for our framed question: “should we use Lambdas for all of our Step Functions tasks”?

After a little research and my anecdotal survey of industry, I’m sufficiently curious about the alleged development velocity and ease-of-use of Lambdas to take an opinionated stance: “we should use Lambdas by default with Step Functions, with the pull-based architecture only used as a fallback in rare cases”.

Come up with an initial experiment

Our initial experiment should have a few traits: it should be quite small, give us meaningful feedback, and be technically feasible. In particular, the size and scope of our experiments should start very small and grow larger as we gain confidence in our overall hypothesis.

So for our first experiment, I aimed small and replaced a trivial task with a Lambda: all it did was reformat some JSON and log the result. This is incredibly “boring” from a technical perspective, but still required making some important directional choices. Namely:

How do we deploy and monitor lambdas?
Should we use any frameworks?

This line of thought led us to discover a few important hypotheses I wanted to prove/disprove with our first experiment:

Is the serverless framework a good tool to leverage to build Lambdas?
Can we build a CI/CD pipeline that feels like best-in-class service tooling that actually deploys Lambdas instead of our normal SOA setup?
Are Lambdas performant in production?
In the end, will this change let us get a functional change through the coding lifecycle faster, without sacrificing safety or architectural sanity?

Evaluate the experiment’s result

For my particular experiment, production experimentation suggests the answer to all of these questions is “yes, Lambdas seem to do as well or better than the status quo”.

Our hypotheses happened to mostly be answering yes/no questions, but this isn’t the only way to measure success. Depending on your particular experiment you may want to check business metrics or even squishier human ideas like “oncall happiness”. The important thing is that you have a clear idea of what you want to learn from your experiment and an clear sense of how you plan to measure that learning once the experiment is live.

Update our overall proposal and iterate

After our first Lambda experiment, the result reinforced the direction of the overall plan, and no major updates were required. Excitingly, we did learn enough to reprioritize the next steps: several problems we thought would require entire future experiments to vet were actually completely solved by Serverless framework plugins. This feedback made us more confident betting on the Serverless Framework as a technology and let us tweak our roadmap to actually be more aggressive on what we tried next.

This feedback loop (experiment, see results, modify hypothesis, experiment again) is crucial to noticing problems and course correcting. Make sure that even your big nebulous bets have a clear way to learn + iterate and you’re comfortable with the cost of the worst-case outcome. This helps protect your project from losing momentum halfway due to an unsuccessful experiment.

Final thoughts

As the problems get messier and more complex, it can help to centralize this process in a single long-running document. It’s a great scratch space for thinking through future experiments and making sure others can follow along with your thought process while you’re at it. My rough format:

What’s the big idea? (add historical context, frame the opportunity)
Who are the owners this project(s)? (clear ownership helps avoid death by analysis and offers clear points of contact)
What’s my best guess at the right answer? (doesn’t have to be correct in hindsight, just has to be a useful, opinionated stance that implies action)
What experiments have we already done in this effort? What did we learn from them?
What is the next, most valuable experiment we should try?

The ability to take on a vague project of potentially massive scope without succumbing to a blind guesswork is quite challenging. But on the upside, it’s also a rare skill that you’ll use more and more as you become a more senior technical leader.

Luckily for all of us, the key isn’t to magically know the right answer up front. Instead you need to build deep context in the problem space — really live and breathe it — until an initial educated guess forms itself. Your first experiment should provide directional feedback and force you to take theoretical ideas and build them for real. Then each ensuing bet shapes your hypotheses about the overall question, and guides you down the rest of the decision tree for your project.