3rd Workshop on
Validation, Analysis and Evolution of Software Tests

Abstract - With the rise of agile development, software teams all over the world embrace faster release cycles as *the* way to incorporate customer feedback into product development processes. Yet, faster release cycles imply rethinking the traditional notion of software quality: agile teams must balance reliability (minimize known defects) against agility (maximize ease of change). This talk will explore the state-of-the-art in software test automation and the opportunities this may present for maintaining this balance. We will address questions like: Will our test suite detect critical defects early? If not, how can we improve our test suite? Where should we fix a defect? The research underpinning all of this has been validated under "in vivo" circumstances through the TESTOMAT project, a European project with 34 partners coming from 6 different countries.

Short Bio - Serge Demeyer is a professor at the University of Antwerp and the spokesperson for the ANSYMO (Antwerp System Modelling) research group. He directs a research lab investigating the theme of "Software Reengineering" (LORE - Lab On REengineering). Serge Demeyer is a spokesperson for the NEXOR interdisciplinary research consortium and an affiliated member of the Flanders Make Research Centre. In 2007 he received a "Best Teachers Award" from the Faculty of Sciences at the University of Antwerp and as a consequence remains very active in all matters related to teaching quality. His main research interest concerns software evolution, more specifically how to strike the right balance between reliability (striving for perfection) and agility (optimising for improvements). He is an active member of the corresponding international research communities, serving in various conference organization and program committees. He has written a book entitled "Object-Oriented Reengineering" and edited a book on "Software Evolution". He also authored numerous peer reviewed articles, many of them in top conferences and journals.

Slides - Keynote slides available at Slideshare

Abstract - Finding a bug in the software is an expensive task, however, debugging is a crucial part of the software development life cycle. Spectrum-Based Fault Localization (SBFL) algorithms can reduce the time spent with debugging. Despite the fact that SBFL is a very well researched topic, there are not many tools that implement it. Many studies have dealt with the effectiveness of SBFL algorithms, although these have been evaluated on Java and C++ programming languages. We performed an empirical study on JavaScript programs (using BugsJS benchmark) to evaluate the relationship between algorithms efficiency and the bug-fix types. First we implemented three popular SBFL approaches, i.e. Tarantula, Ochiai and DStar, then examined whether there was a correlation/connection between the positions of the faulty methods in the suspiciousness ranks and bug-fix types. Results show that certain bug-fix types can be significantly differentiated from the others (in both positive and negative direction) based on the fault localization effectiveness of the investigated algorithms.

Abstract - Automated test case generation techniques usually aim to maximize some coverage criteria. For object oriented languages, like Java, the branches that can be reached in source code, frequently depend on the internal object state. Meaning certain branches will only be taken, if fields inside the tested class are set to specific values. It is however not obvious how much of the internal object state can be controlled. In this paper, we analyzed a corpus of 110 open source systems to evaluate how settable their classes are. Meaning we looked for ways that fields inside classes can be written. For instance, we analyzed the source code to identify setter methods that can be used to set the values of a field. Our results show that 66.5% of fields can be set to a desired value, while 31.5% of fields may be only settable to particular values or require a more in depth analysis. Only for 2% of fields, we did not find any way to set their values.

Abstract - Mutation testing is the state-of-the-art technique for detecting weaknesses in a test suite. Unfortunately, alleviating these weakness (i.e. "killing the surviving mutants") is quite labour-intensive. In this paper we investigate a recommender system which expands test cases with extra asserts for the easy-to-fix mutants. We evaluated a proof-of-concept tool on ten open-source projects, and killed up to 6% of the surviving mutants. This illustrates that such a test expansion system would free up valuable time to address the harder-to-fix mutants.

Abstract - Component-based software development (CBSD) is one of the main programming paradigms of the last decades. The main idea of CBSD is to modularize a system as a configuration of multiple interacting components. Components interact with each other via dedicated component interfaces hiding a component's implementation and making components interchangeable. In this paper, we present an early investigation of unit testing practices of open source component-based software systems with the goal to find out how component-based software systems are actually tested and how to improve unit testing practices as part of future research. Our preliminary results show that unit tests typically directly test the component implementation and not dedicated component APIs. The method coverage of component APIs varied between 17% and 34% in the analyzed projects.

Abstract - Software testing helps developers to identify bugs. However, awareness of bugs is only the first step. Finding and correcting the faulty program components is equally hard and essential for high-quality software. Fault localization automatically pinpoints the location of an existing bug in a program. It is a hard problem, and existing methods are not yet precise enough for widespread industrial adoption. We propose fault localization via Probabilistic Software Modeling (PSM). PSM analyzes the structure and behavior of a program and synthesizes a network of Probabilistic Models (PMs). Each PM models a method with its inputs and outputs and is capable of evaluating the likelihood of runtime data. We use this likelihood evaluation to find fault locations and their impact on dependent code elements. Results indicate that PSM is a robust framework for accurate fault localization.

Abstract - Test flakiness (non-deterministic behavior of test cases) is an increasingly serious concern in industrial practice. However, there are relatively little research results available that systematically address the analysis and mitigation of this phenomena. The dominant approach to handle flaky tests is still detecting and removing them from automated test executions. However, some reports showed that the amount of flaky test is in many cases so high that we should rather start working on approaches that operate in the presence of flaky tests. In this work, we investigate how test flakiness affects the effectiveness of Spectrum Based Fault Localization (SBFL), a popular class of software Fault Localization (FL), which heavily relies on test case execution outcomes. We performed a simulation based experiment to find out what is the relationship between the level of test flakiness and fault localization effectiveness. Our results could help the users of automated FL methods to understand the implications of flaky tests in this area and to design novel FL algorithms that take into account test flakiness.

The event is open to all workshop and conference attendees
and it will start at 18:30.

Details see https://saner2020.csd.uwo.ca/socialEvents