Generative AI and Test Automation

Code Coverage is only an indicator

Hey, I am Klaus Haeuptle! Welcome to this edition of the Engineering Ecosystem newsletter in which I write about a variety of software engineering and architecture topics like clean code, test automation, decision-making, technical debt, large scale refactoring, culture, sustainability, cost and performance, generative AI and more.

Generative AI can be a great assistance in writing automated tests. The problem comes when there is too much focus on fulfilling code coverage goals and not so much on test code maintainability, test effectiveness, flakyness, testability and how TDD can be a driver for better design. And this problems get worse when more test are generated with the intention to fulfill code coverage goals. Currently, I have the impression that many colleagues focus on fulfilling code coverage goals and develop tailored tools leveraging Generative AI to achieve that goal. I would strongly recommend to prioritize the quality of the test suite over code coverage. Having a test suite, which increases code coverage, but does not increase your confidence in making changes efficiently is a form of technical debt and a burden.

Tools like GitHub CoPilot can be quite helpful when writing automated tests, so would I would recommend to explore them, but to always keep the quality of the test suite in mind.

Interesting research questions can be derived from how LLM can be used to generate test cases, which are not only increasing code coverage, but also consider all qualities and increase the confidence in making changes. The most advanced approach I have seen so far is from Meta, which generates additional test cases in the style of an existing suite.

Please share if there are other interesting approaches, which also consider the quality of the test suite.

Mark as not spam: : When you subscribe to the newsletter please do not forget to check your spam / junk folder. Make sure to "mark as not spam" in your email client and move it to your Inbox. Add the publication's Substack email address to your contact list. All posts will be sent from this address: ecosystem4engineering@substack.com.

❤️ Share it — The engineering ecosystem newsletter lives thanks to word of mouth. Share the article with someone to whom it might be useful! By forwarding the email or sharing it on social media.