The Value and Limitations of AI for Large Scale Refactoring
Automated Upgrades, Migrations and Recurring Changes
Hey, I am Klaus Haeuptle! Welcome to this edition of the Engineering Ecosystem newsletter in which I write about a variety of software engineering and architecture topics like clean code, test automation, decision-making, technical debt, large scale refactoring, culture, sustainability, cost and performance, generative AI and more.
Large Scale Refactoring tooling allows making changes across a large code base with many different Multi-repo or MonoRepo projects. E.g. in order to automatically upgrade Java or Spring Boot, migrate from technology to another or fix security issues across a large amount of repositories in a consistent and repeatable manner and hence reduce the manual effort, the risk of human error and enable innovation due to faster technology modernization. In Automated Upgrades and Migrations: Balancing AI and Deterministic Recipes in Code Migrations I wrote about the benefits of deterministic rule-based refactoring versus AI based refactoring. Rule-Based refactoring recipes can be used for basic refactorings like replacing method calls, renaming variables, adding or removing annotations and lot more, in a consistent and repeatable manner. In this article, I will focus on the additional value of AI for large scale refactoring and tech stack modernization initiatives. The disadvantage of AI-assisted refactoring is that it is not as deterministic as rule-based refactoring and there can be unforeseen surprises and bugs requiring intensive human review. Since human code review is becoming a bottleneck and at the same time human code review is probably also not sufficient for the unpredictable glitches produced by AI, it is important to leverage rule-based recipes as much as meaningful and focus the use of AI on things, which were not possible before.
Deterministic Rule-Based Refactoring Recipes
Deterministic rule-based refactoring refers to the process of making systematic and predictable changes to a codebase using predefined rules or recipes. These rules are designed to perform specific refactoring tasks, such as renaming variables, replacing method calls, or adding annotations, in a consistent and repeatable manner. The key characteristics of deterministic rule-based refactoring include:
Predictability: The changes made by rule-based refactoring are predictable and consistent, as they follow a set of predefined rules.
Repeatability: The same set of rules can be applied multiple times to achieve the same results, ensuring uniformity across the codebase.
Automation: Rule-based refactoring can be automated using tools and frameworks, reducing the need for manual intervention and minimizing human error.
Reliability: Since the rules are deterministic, the refactoring process is reliable and less prone to introducing bugs or unforeseen issues. Overall, deterministic rule-based refactoring is a valuable approach for performing large-scale code changes in a controlled and efficient manner.
At the same time rule-based refactoring has limitations for certain use cases, which may be overcome by leveraging AI.
Use Cases for AI
In the context of Large Scale Refactoring, there are several major use cases for AI:
AI can be used to better understand existing code bases and make better decisions on how to evolve the system.
AI can be used to for semantic code search to be able to find places in the code, which need to be changed.
AI can be used to support with the creation of deterministic recipes.
AI can be used to find applicable recipes
AI can be used for use cases for which it is difficult or impossible to write deterministic recipes. While there are some special use cases, which worked, it is still a research area.
Need for Reliable Refactoring
In the context of Large Scale Refactoring we need a very reliable way of performing refactoring across a large amount of repositories. Therefore, it is preferred to leverage rule-based refactoring for large scale changes. Still there are some early promising experiments in leveraging AI in combination with rule-based refactoring while minimizing the potential negative impact of hallucinations. At the same time success stories about leveraging AI for large scale migrations are promoted, while the biggest part of the actual work was done with rule-based refactoring (e.g. the story automating Java migrations with Amazon Q Transformation). You can find more details in the blog Automated Upgrades and Migrations: Balancing AI and Deterministic Recipes in Code Migrations
Refactoring Recipes combining rule-based with AI capabilities
OpenRewrite is a framework for refactoring recipes, which can be used for automated code refactoring and analysis. OpenRewrite supports also Hybrid Recipes, which combine rule-based recipes with AI capabilities for specific tasks. This can reduce the negative impact of Generative AI, but leverage it for making refactoring possible, which are not feasible with pure rule-based refactoring. More details on leveraging Moderne and OpenRewrite can be found in the ebook AI for Mass Scale Code Refactoring and Analysis.
Google also wrote some interesting papers on leveraging AI for different kinds of refactoring (most of those not in the context of large scale changes). Still it shows in which direction the industry could move in the next years. Making more refactoring possible by leveraging AI. A key example from Google helped automate the process of updating ID types in their system from 32-bit integers to 64-bit integers. The challenges included:
There were tens of thousands of locations in the codebase where these IDs were used.
The IDs were defined as generic numbers like int32_t in C++ or Integer in Java, making them hard to identify and where AI has helped to identify them.
Changes had to be made across many different files, and the interfaces using the IDs also needed to be updated.
Tests related to these IDs had to be adjusted to verify the new implementation.
Despite these challenges, the AI generated and validated most of the code changes, reducing migration time by 50%, while extensive test automation and human engineers ensured accuracy by doing code reviews. In fact, the model predicted the need to edit Java files with 91% accuracy, further enhancing the efficiency of the migration process. In cases were pure rule-based refactoring is not sufficient combining AI with rule-based refactoring holds promise for more powerful large-scale refactoring in the future. More details about the described case and further insights from Google can be found in the following papers:
Conclusion
Leveraging AI for refactoring often requires intensive human code reviews, which quickly can become a bottleneck and is therefore not a good fit for large scale refactoring. Therefore, rule-based recipes are preferred, but AI can help to overcome certain limitations. And in the next years AI will probably help to overcome more limitations and make more large scale refactoring possible.
Mark as not spam: : When you subscribe to the newsletter please do not forget to check your spam / junk folder. Make sure to "mark as not spam" in your email client and move it to your Inbox. Add the publication's Substack email address to your contact list. All posts will be sent from this address: ecosystem4engineering@substack.com.
❤️ Share it — The engineering ecosystem newsletter lives thanks to word of mouth. Share the article with someone to whom it might be useful! By forwarding the email or sharing it on social media.