Kaizen Master - Solution for Technical Debt (and Legacy Code)

(a short cookbook is at the bottom of the page)

~70% of developers call technical debt the main obstacle in their job. The problem scale estimates ~$2.4 trillion/year loss only in the US (e.g. ). Yeah, that much! Programmers and managers notice: the first 1-2 years a product is fine, but then you see 30-60% time being spent on fixing. The company becomes nearly paralyzed.

As an engineering manager, I thought about it and during the last 1.5 years tried an approach based on Kaizen. Its goal is to gradually reduce technical debt without rewiring much resources. And, chief, it gave results!

Speed-Quality Dilemma

A step away from Kaizen is that we have only one person for Kaizen blitz. Why? Imagine you put an entire team (~5 devs) on it. Will the department stop for a week? Isn’t it a big resource rewire in terms of business? We probably can use a shorter blitz, say 1 day, but then we need to choose a day and sync devs, or make them choose a day each for themselves… And how to prevent them from grabbing the same TODOs - we don’t use trackers, only code comments. A lot of “if”.

So, in each team we install a Kaizen Master (KM), who after every transaction (project/epic) does blitz. As you might have guessed, it’s the technical team lead. Here is why:

• Lead reviews all pull requests and projects, thus, aware of what’s going on in code. Others will spend more time studying what they didn’t see yet.


• Lead tracks all code, thus, has an eye on global inconveniences and patterns, like lack of testing utilities and boilerplates.


• Lead is usually the top experienced employee with architectural skills and strategic vision. These items and Kaizen complement each other.


• After a transaction is finished, our leads usually spend time with stakeholders and managers, deciding what’s next, who to delegate, etc. A blitz with a comparatively low load and free roaming time (there is no plan for blitz) seems a good time for meetings, brainstorming, chaotic calls and approvals.

With this in mind, tech lead seems the most effective person for blitz. They’ll improve fast, accurately and considering long-term consequences.

When to Apply

Kaizen Master makes sense in fast growing SaaS/PaaS projects, such as fin tech, web services, cloud products, etc. They are long-term, have large multi-tenant codebases, require to often work on already implemented functionality (e.g. add new features to it). And project resources are limited. Huge amounts of work on tight deadlines happen in rich corporations all the time, so don’t take “limited resources” too straight.

KM fits well with teams that work in Shape-Up, Scrum, Rapid Development and Extremal Programming methodologies. Basically, any transactional/iterative approach fits.

When NOT to Apply

If a team has time to run improvement sessions or deadlines are absent (you heard that right, I worked like this once), Kaizen doesn’t make much sense. Short-term (<1 year), contract development and projects with generic logic (e.g. REST API limited to CRUD and filtering) will not benefit.

Profits

1. It causes systematic code and product quality to rise, though it may take ~8-10 months before impact becomes significant (which isn’t much in terms of business).


2. Resources are not drastically rewired - the team keeps working, while KM is on blitz alone. Depending on transaction size, KM will spend 10-15% of their time for improvements, which isn’t a lot.


3. No planning, tools or bureaucracy needed - improvements are small, spontaneous and informal.


4. Gives the KM time to review the codebase (not just out-of-context pull requests), which improves their knowledge of project and team, which helps in architecture and strategy.


5. KM helps to avoid scope creep, as improvements are organized - separated into blitz.

In Anuran (trip planning webservice, where I work), we have a ~4 years old codebase. 1.5 years ago, our technical debt was ~40% of work time. It’s because we started the project with prototyping - rapid development of pieces, dropping them, keeping successful ones and then merging them into a single system. But today, we spend only ~6% of time on technical debt. Sounds like success, ha?!

Cookbook

So, the finalized recipe of KM:

Transactions:

Split the work into large periods - transactions. In most teams, 6-8 weeks is enough for code to grow significantly and obtain problems.

Kaizen blitz:

1 week is enough for KM to review a lot of code and improve it. There is no work planned for the KM during blitz, and it must not be considered as spare time. It’s reserved for quality control, only urgent problems should be attended at it by KM.

(It’s why we do blitz periodically instead of event-driven - there is always something else to do urgently, so improvement is delayed indefinitely. When time is reserved, even though it’s not much, it’s used efficiently.)

Simplify:

Only simplifications are done on blitz. Bug fixing and features (even tiny) are not. Most activity should go around refactoring, erasing unneeded code, reducing dependencies, writing reusable utilities, merging boilerplates, lowering cyclomatic and time complexity.

Point improvements:

If an improvement takes more than a day, split it into smaller pieces. It’s fine to do a part and continue on the next blitz.

Choose KM:

It’s recommended to choose tech lead as a KM due to their architectural, strategical skills, best practices and actual codebase knowledge.

Write TODOs:

To make KM work, devs should write TODOs when they find a potential improvement/inconvenience.

Do not:

• Use complex tools for TODO tracking. KM is a lightweight methodology, not a new form of bureaucracy.


• Consider KM as an excuse to write badly, because there is time to fix it later. Technical debt piles up exactly because of this logic.


• Make it complicated. 3 days instead of a week, transactions of different sizes - it causes frustration. Keep blitz 1 week long and choose transaction length according to the speed of codebase growth.

Conclusion

That's it, that's a methodology we designed in Anuran to deal with technical debt. I think you'll agree it's very lightweight and informal (which is the goal) but may still bring results. If you're going to try it for your team, it'd be nice if you'd provide some feedback and observations. It may help to sharpen KM even more.