Strategies for dealing with low Code Coverage in Legacy Code
Making Changes with more Confidence in Legacy Code
Hey, I am Klaus Haeuptle! Welcome to this edition of the Engineering Ecosystem newsletter in which I write about a variety of software engineering and architecture topics like clean code, test automation, decision-making, technical debt, large scale refactoring, culture, sustainability, cost and performance, generative AI and more. In this edition of the newsletter I am touching the complex topic of code coverage in legacy cody environments.
There are many definitions of legacy code, but most resonating definition to me is that it is code without tests. The code lacks a good test suite allowing the developers to make changes to the code with the confidence of not introducing bugs. This is a very broad definition, but it highlights the importance of testing in legacy systems. Without tests, it's difficult to understand the functionality of the code, and it's even more challenging to make changes without introducing errors. This can lead to a vicious cycle of poor code quality, poor testability, which can be difficult to break out of. It is important to consider building a good test suite and testability from the beginning, since it gets much harder and costly to add tests and the testability to the system later. This blog post will focus on the importance of code coverage and describe some strategies on how to improve it in legacy systems.
Benefits and Limits of Code Coverage
Code coverage is not a perfect measure of test quality, however, it does offer a reasonable, objective industry standard metric with actionable data. It does not require significant human interaction, it applies universally to all products, and there are many tools available in the industry for most languages. You must treat it with the understanding that it’s an insufficient and indirect metric that projects a lot of information into a single number so it should not be your only source of truth to get and understanding of the quality of your tests. Instead, you must use it in conjunction with other techniques to create a more holistic assessment of your testing efforts.
High code coverage doesn't necessarily mean your software is bug-free or that your tests are effective. It's possible to have high code coverage with poor test cases that don't accurately simulate real-world usage of the application. Therefore, code coverage should be used as one of many tools in your testing strategy, not as the sole indicator of software quality. Low code coverage should receive our attention, but a low code coverage number does guarantee that large areas of the product are going completely untested by automation. This increases our risk of pushing bad code to production, so it should receive attention. In fact a lot of the value of code coverage data is to highlight not what’s covered, but what’s not covered.
Types of Code Coverage
There are several types of code coverage, each providing a different perspective on the codebase:
Statement Coverage: This is the most basic form of code coverage, measuring the percentage of statements that have been executed in the codebase.
Branch Coverage: This measures the percentage of decision points (like 'if' statements) that have been executed, ensuring that both the true and false branches have been tested.
Function Coverage: This measures the percentage of functions that have been called in the codebase.
Condition Coverage: This measures the boolean sub-expressions in decision statements, ensuring that each condition can evaluate both to true and false.
Each type of coverage provides a different level of insight and thoroughness, and the appropriate type to use depends on the specific needs and context of the project. When it comes to legacy systems, it's often useful to start with statement coverage and branch coverage, as these provide a good overview of the codebase and can help to identify areas that need further testing.
The Challenge of Legacy Systems
Legacy systems pose a set of challenges when it comes to improving code coverage. These systems have often been in use for many years, and over time, they've been patched, updated, and modified to meet changing business needs. This often results in a complex, intertwined codebase that is difficult to understand and even more challenging to test.
Lack of Adequate Testing
One of the most significant challenges with legacy systems is the lack of adequate testing. When these systems were initially developed, the importance of comprehensive testing and code coverage might not have been fully recognized or prioritized. As a result, these systems often have large sections of code that have never been tested, making it difficult to assess their functionality or predict how changes might impact them. And hence make it difficult to improve the testability of the system due to decreasing the confidence in making changes.
Poor Testability
Legacy systems are often characterized by poor testability. This is due to factors such as tight coupling between components, lack of modular design, bad interfaces and absence of automated testing capabilities. These factors make it difficult to isolate individual components for testing, leading to a lower level of code coverage.
Fear of Breaking Functionality
There's often a fear associated with making changes to legacy systems. Since these systems are typically critical to business operations, there's a risk that changes could break functionality and disrupt business processes. This fear can discourage teams from attempting to improve code coverage, as they may prefer to maintain the status quo rather than risk introducing errors.
Limited Knowledge and Expertise
Over time, the original developers of the system may have moved on, taking their knowledge of the system with them. This can leave current teams with limited understanding of the system's intricacies, making it difficult to implement effective testing strategies and improve code coverage.
Despite these challenges, improving code coverage in legacy systems is not an insurmountable task. With a strategic approach, a commitment to incremental improvement, and the right expertise, it's possible to enhance the quality and reliability of these systems. The key is to understand the unique challenges these systems present and develop strategies tailored to address them.
Strategies to improve code coverage in Legacy Systems
Just because your product has low code coverage doesn’t mean you can’t take concrete, incremental steps to improve it over time. Inheriting a legacy system with poor testing and poor testability can be daunting, and you may not feel empowered to turn it around, or even know where to start. But at the very least, you can adopt the ‘boy-scout rule’ (leave the campground cleaner than you found it) or strategically improve testability. Over time, and incrementally, you will get to a more healthy state.
Start with new code and apply the boy-scout rule to Legacy Code
In this situation, depending on the level of coverage, to get high coverage and meaningful test quality, could stop development for a long time. Since it is hardly possible to get the time and very likely also not meaningful, it is advised to implement two part strategy – first, ensure all new code is adequately covered. As new development often requires refactoring some old code, coverage will increase on both new code as well as existing code. For legacy code you should check that the trend is positive and adjust the absolute threshold to increased levels of coverage from time to time. The absolute numbers will only move slowly.
Start with the most important Legacy parts
The second part of the strategy is to pay down some of the technical debt based on a risk assessment. Introduce a risk-based priority for implementing automated tests over existing code that lacks coverage. The risks should be based on priority business risks or critical core services that are hot spots for bugs, changing requirements or other reasons for code churn. Usually there is a small part of the overall code base contributing most to customer and quality problems, therefore it makes sense to focus the investment there, starting with creating a first automated safety net and improving testability of the design and architecture. It is recommended to have colleagues highly skilled in engineering practices and test automation for driving those improvement activities.
Another approach is to define the strategy for legacy depending on the change rate of the code:
Existing code, which is not changed and there is no plan to change it: If it is not touched, it is currently working without problems. Changing to make it testable to get coverage does not justify the risk and the effort which comes with it.
Existing code changed rarely: Same as above, no absolute coverage goal for the existing code, but the addition / change to the existing code should be decoupled and tested by automated tests. So the code coverage trend should be positive.
Existing code hotspot of changes: Here similar goals as for new code should be targeted for. Either the code contains many bugs or needs to be adjusted to changing functional or non-functional requirements. Implementing this strategy in legacy code requires much more effort than in new code. It needs to be evaluated if a bigger investment would be valuable to improve the testability.
Limitations of the Boy-Scout Rule in Legacy Systems
The Boy-Scout Rule, a principle in software development, encourages developers to "always leave the campground cleaner than you found it." In essence, it means improving the code each time it's touched, leaving it in a better state than it was before. However, when applied to legacy systems, this rule encounters significant limitations, particularly when it comes to improving code coverage incrementally. Legacy System lack testability and code coverage, which can make it difficult or very expensive to add tests incrementally.
Lack of Isolation: Legacy systems often have components that are tightly coupled, making it difficult to isolate changes. This lack of isolation makes it challenging to apply the Boy-Scout Rule without risking system-wide effects.
Inadequate Test Coverage: Legacy systems often lack comprehensive test coverage, making it risky to refactor code or add new features. Without a robust suite of tests, it's difficult to ensure that changes made to improve the code won't introduce new bugs or regressions.
Resistance to Change: Legacy systems are often critical to business operations, and there can be significant resistance to making changes that could potentially disrupt these operations. This resistance can limit the ability to make incremental improvements to the codebase.
Technical Debt: Legacy systems often carry significant technical debt, which can make it difficult to improve the code incrementally. The Boy-Scout Rule assumes that developers have the ability to clean up code as they go, but in a legacy system, the level of technical debt can be so high that this isn't feasible.
Lack of Documentation: Legacy systems often lack up-to-date documentation, making it difficult to understand the system's intricacies and dependencies. This lack of understanding can limit the ability to improve the code incrementally.
In conclusion, while the Boy-Scout Rule is a valuable principle in software development, its application in legacy systems comes with challenges. Overcoming these limitations can require a strategic approach that includes investing in test coverage and improving testability. When the testability of the system is in a better shape, it will be much easier to apply the boy-scout rule again. Besides, it will simply the testing of new code and hence increase the confidence in making changes.
Conclusion
Code coverage is an important metric for assessing the quality of your tests and identifying areas that need improvement. However, it's important to understand the limitations of code coverage and use it in conjunction with other techniques to create a more holistic assessment of your testing efforts. In legacy systems, it's often useful to start with statement coverage and branch coverage, as these provide a good overview of the codebase and can help to identify areas that need further testing. It is important to consider building a good test suite and testability from the beginning, since it gets much harder and costly to add tests and the testability to the system later. If you have the need to increase confidency in making changes and hence need to improve code coverage in legacy systems, I can recommend the following resources:
Clean Code: Writing maintainable, readable and testable code
Shared Language for talking about Test Strategy with a focus on the Test Pyramid
The Hidden Costs of Flaky Tests: Why You Need to Fix Them Now,
Get Higher Quality and Productivity by tackling the Broken Window Effect
Thanks so much for a great year and being part of the journey. I hope the content has been helpful for you. To help me pick future topics , please add a comment.
If you’re finding this newsletter valuable, share it with a friend and co-workers.
Also, if you have feedback about how I can make the newsletter better, let me know via your preferred channel on LinkedIn, Mastodon or by leaving a comment in the newsletter edition🙏
Thanks for reading Software Engineering Ecosystem! Subscribe for free to receive new posts and support my work.