Test flakiness’ causes, detection, impact and responses: A multivocal review

Simple item page
dc.citation.volume206
dc.contributor.authorTahir A
dc.contributor.authorRasheed S
dc.contributor.authorDietrich J
dc.contributor.authorHashemi N
dc.contributor.authorZhang L
dc.date.accessioned2024-10-02T02:05:42Z
dc.date.available2024-10-02T02:05:42Z
dc.date.issued2023-12
dc.description.abstractFlaky tests (tests with non-deterministic outcomes) pose a major challenge for software testing. They are known to cause significant issues, such as reducing the effectiveness and efficiency of testing and delaying software releases. In recent years, there has been an increased interest in flaky tests, with research focusing on different aspects of flakiness, such as identifying causes, detection methods and mitigation strategies. Test flakiness has also become a key discussion point for practitioners (in blog posts, technical magazines, etc.) as the impact of flaky tests is felt across the industry. This paper presents a multivocal review that investigates how flaky tests, as a topic, have been addressed in both research and practice. Out of 560 articles we reviewed, we identified and analysed a total of 200 articles that are focused on flaky tests (composed of 109 academic and 91 grey literature articles/posts) and structured the body of relevant research and knowledge using four different dimensions: causes, detection, impact and responses. For each of those dimensions, we provide categorization and classify existing research, discussions, methods and tools With this, we provide a comprehensive and current snapshot of existing thinking on test flakiness, covering both academic views and industrial practices, and identify limitations and opportunities for future research.
dc.description.confidentialfalse
dc.edition.editionDecember 2023
dc.identifier.citationTahir A, Rasheed S, Dietrich J, Hashemi N, Zhang L. (2023). Test flakiness’ causes, detection, impact and responses: A multivocal review. Journal of Systems and Software. 206.
dc.identifier.doi10.1016/j.jss.2023.111837
dc.identifier.eissn1873-1228
dc.identifier.elements-typejournal-article
dc.identifier.issn0164-1212
dc.identifier.number111837
dc.identifier.piiS0164121223002327
dc.identifier.urihttps://mro.massey.ac.nz/handle/10179/71569
dc.languageEnglish
dc.publisherElsevier Inc
dc.publisher.urihttps://www.sciencedirect.com/science/article/pii/S0164121223002327
dc.relation.isPartOfJournal of Systems and Software
dc.rights(c) 2023 The Author/s
dc.rightsCC BY 4.0
dc.rights.urihttps://creativecommons.org/licenses/by/4.0/
dc.subjectFlaky tests
dc.subjectNon-deterministic tests
dc.subjectTest bugs
dc.subjectSoftware testing
dc.subjectMultivocal review
dc.titleTest flakiness’ causes, detection, impact and responses: A multivocal review
dc.typeJournal article
pubs.elements-id480673
pubs.organisational-groupOther
Files
Original bundle
Now showing 1 - 1 of 1
Thumbnail Image
Name:
Published version.pdf
Size:
1.69 MB
Format:
Adobe Portable Document Format
Description:
480673 PDF.pdf
Download
License bundle
Now showing 1 - 1 of 1
Thumbnail Image
Name:
license.txt
Size:
9.22 KB
Format:
Plain Text
Description:
Download
Collections
Journal Articles