Assessing automatically-generated tests code quality: beyond traditional test smells

Abstract

Various studies have examined the quality of automatically generated test cases, focusing particularly on the presence of test smells. However, recent research revealed that not all test smells have been identified, suggesting that there are still unexplored test smells within generated tests. This led us to uncover and categorize these new test smells. This paper presents a taxonomy of 13 new test smells grouped into four categories through a manual analysis of 2,340 automatically generated tests from an external dataset. Based on the proposed taxonomy, we introduce the tool Smelly, a static analysis tool that automatically reports warnings for code patterns consistent with the defined test smells. We use Smelly to examine the presence of these smells in tests generated by EvoSuite and JTExpert. Our results reveal that all 13 test smells were identified in the tests generated by EvoSuite, with eight of them also present in those generated by JTExpert. Moreover, Smelly’s warnings appeared across a considerable number of classes and projects in both tools, providing insight into the prevalence and distribution of these smells. Our results suggest two key points: (i) test quality evaluation should extend beyond the tests to consider the tested code, and (ii) automatically generated tests exhibit flaws uncommon in manually created tests, necessitating specific quality-checking tools. We share five lessons learned for test generators to enhance the quality and utility of automatically generated tests.