How to Conduct a Team Retrospective to move towards an Effective, Maintainable and Robust Test Suite
A test suite is a collection of automated tests that verify the functionality and quality of a software system. A test suite can be composed of unit tests, integration tests, system / scenario tests, performance tests, and other types of tests. A test suite is an essential tool for ensuring that the software meets the requirements and expectations of the stakeholders and users.
One way to assess and improve the quality of a test suite is to conduct a team retrospective. A team retrospective is a meeting where the team members reflect on their work, identify what went well and what went wrong, and come up with action items to improve their processes and practices. A team retrospective can be done at regular intervals, such as after each sprint or release, or whenever there is a significant event or issue that affects the team.
A team retrospective on a test suite can focus on four main aspects: confidence, flakiness, maintainability, and speed. These aspects can be measured by asking the following questions:
Confidence in Tests (Effectiveness of Tests):
How confident are you, that the tests ensure, that the code works in production (1-10)? A low confidence in the tests needs to be addressed and the root causes identified. E.g. sometimes developers think that something is not testable, that is quite often not the case or is an indicator that there is an issue in the architecture or used technology. It is worth to ask this question from time to time to see if the trend is improving.
Which severe bugs did occur in the past and how can we avoid them in the future?
Flaky Tests
How often is the pipeline red?
How often is the red pipeline caused by a real bug?
How long does it take until a red pipeline is green again?
Maintainability
How well can I read tests, understand tests quickly and adjust them quickly (1-10)?
Fast
How long does it take to execute the tests?
Are the tests fast enough (1-10)?
How often do you run the tests locally?
Confidence in Tests (Effectiveness of Tests)
The confidence in tests reflects how well the tests cover the functionality and quality attributes of the software. A high confidence in tests means that the team trusts that the tests can detect any defects or regressions in the code before they reach production. A low confidence in tests means that the team doubts that the tests are sufficient or reliable enough to ensure the quality of the software.
Flaky Tests
The robustness of a test suite reflects how stable and consistent it is. A robust test suite produces reliable and reproducible results regardless of changes in the code or environment. A robust test suite also minimizes false positives (tests that fail when they should pass) .
Fast
The speed of a test suite reflects how quickly it can provide feedback to the team. A fast test suite can run frequently and efficiently, saving time and resources. A fast test suite can also enable faster development cycles and is a major driver of developer productivity.
Maintainability
The maintainability of a test suite reflects how easy it is to modify and update it. A maintainable test suite can adapt to changes in the software and environment without requiring too much effort or causing too much risk. A maintainable test suite can also facilitate collaboration and knowledge sharing among team members.
Resources
Why a maintainable and readable test suite is important to stay agile
Shared Language for talking about Test Strategy with a focus on the Test Pyramid
Subscription: If you want to get updates, you can subscribe to the free newsletter:
Mark as not spam: : When you subscribe to the newsletter please do not forget to check your spam / junk folder. Make sure to "mark as not spam" in your email client and move it to your Inbox. Add the publication's Substack email address to your contact list. All posts will be sent from this address: ecosystem4engineering@substack.com.
❤️ Share it — The engineering ecosystem newsletter lives thanks to word of mouth. Share the article with someone to whom it might be useful! By forwarding the email or sharing it on social media.