SEAT: A Taxonomy to Characterize Automation in Software Engineering

by   Shipra Sharma, et al.

Reducing cost and time required to build high quality software is a major goal for software developers. Building tools and techniques that can help achieve such a goal is the chief aim for Automated Software Engineering (ASE) researchers. However, in order to be effective an ASE researcher or professional must understand the characteristics of both successful and not-so-successful ASE tools, and the constituent techniques employed by such ASE tools. In this paper we present such a characterization of ASE tools and major constituent techniques from different areas of computer science and engineering that have been employed by such ASE tools. To develop the characterization we carried out an extensive systematic literature review over about 1175 ASE research articles. One of our key goal was to identify useful relationships among ASE tools, their constituent techniques and the software development life cycle activities that these tools targeted. In terms of changes in popularity of constituent techniques with time we did not observe any clear trend. We organized the results of our characterization as a taxonomy called SEAT (Software Engineering Automation Taxonomy). A salient feature of SEAT is that it focuses on automation of activities from all phases of SDLC. Such a taxonomy, among other applications, shall enable synthesizing new automation tools for different SDLC activities. Recomposing existing systems to achieve better features will also be possible. Further, the taxonomy has been realized as a graph database using neo4j(an open source graph database), which can be queried using an SQL like language. The graph database allowed us to uncover hidden relationships by way of exhaustive search for connections and paths between different nodes (i.e. concepts). We demonstrate the efficacy of SEAT by discussing few practical use cases.



There are no comments yet.


page 1

page 2

page 3

page 4


Building a Sustainable Structure for Research Software Engineering Activities

The profile of research software engineering has been greatly enhanced b...

What helped, and what did not? An Evaluation of the Strategies to Improve Continuous Integration

Continuous integration (CI) is a widely used practice in modern software...

Game Development Software Engineering Process Life Cycle: A Systematic Review

Software game is a kind of application that is used not only for enterta...

Software engineering and the SP theory of intelligence

This paper describes a novel approach to software engineering derived fr...

Cognition in Software Engineering: A Taxonomy and Survey of a Half-Century of Research

Cognition plays a fundamental role in most software engineering activiti...

Ways of Applying Artificial Intelligence in Software Engineering

As Artificial Intelligence (AI) techniques have become more powerful and...

A Taxonomy of Data Quality Challenges in Empirical Software Engineering

Reliable empirical models such as those used in software effort estimati...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

With ever growing adoption of IT and software systems, building quality software in shorter time and at lower cost has been a major goal for software engineering professionals. Automated Software Engineering (ASE) community over the years has made significant contribution towards achieving this goal by developing tools and techniques to automate various activities of different phases of Software Development Life Cycle (SDLC). Such automation has helped in reducing the cost and time required to build good quality software. Almost every fundamental activity in SDLC has seen some type of automation. These activities typically are requirement engineering, design, implementation, testing and product’s maintenance [Sommerville (2010)].

Though, life cycle of software development can follow varied methodologies, the basic activities in each methodology remain more or less the same. For instance, whether one employs the Waterfall, V or Spiral model, the phases such as requirements analysis, architecture, design, testing and so on are always involved at some level in software development. Each phase has its own set of automation tools and techniques (henceforth called ASE tools) which are intended for reducing the manual work in respective phases. In order to automate tasks in the targeted SDLC activities such ASE tools leverage and depend on techniques (henceforth called constituent techniques

) from several other areas of computing such as formal methods, semantic computing, natural language processing, information retrieval, knowledge representation and so on.

For example, many earlier ASE tools employed formal methods while several of the recent ones rely on semantic computing techniques. Diversity in constituent techniques on which such ASE tools are built leads to varying levels of quality, adoption and success for those ASE tools.

From the perspective of end users, researchers and academicians who are engaged in using, developing and teaching about ASE tools it is important to understand the whole landscape of ASE tools and techniques, underlying design approach of each tool and many other related factors.

However, even after many years and a great deal of advancements in automation of tasks in SDLC phases, it is difficult to find in literature a methodical and SDLC-centric characterization of these automation tools and techniques. Though there are few studies, such as [Rafi et al. (2012)], [Anand et al. (2013)] that examin testing phase, [Gulwani (2010)] that examins works in program synthesis etc., they all address only a subset of SDLC activities and none of them considers the entire SDLC.

We carried out a systematic state-of-the-art literature review of the automation tools and techniques developed by ASE researchers for various phases of SDLC. A major aim of our study was to identify useful relationships and patterns if any existed among ASE tools, their constituent techniques and the SDLC activities that these tools targeted.

Based on our findings from this study, we synthesized a non-orthogonal taxonomy – that we call Software Engineering Automation Taxonomy (SEAT) – that characterizes ASE tools, their constituents techniques and the SDLC activities that they target. We implemented this taxonomy in a graph database for easy querying along different dimensions. A graph database allowed us to uncover hidden relationships by way of exhaustive search for connections and paths between different nodes (i.e. concepts).

Rest of this paper is structured as follows. In Section-2 we describe our overall research methodology; we outline our key research questions in this section. Section-3 presents our approach for eliciting relevant information from research literature. A detailed discussion, along our research questions, of the patterns we observed in ASE research literature is presented in Section-4. We then leverage these observations to develop a taxonomy for automation in software engineering in Section-5. Finally we conclude the paper in Section-6.

2 Research Method

In order to synthesize our taxonomy we first carried out a Systematic Literature Review (SLR) of the ASE domain. We used a tool called StArt ([Fabbri et al. (2012)], [Fabbri et al. (2016)]) to study and analyze the ASE research articles for various phases of SDLC. This review and analysis of data was done as per the guidelines outlined in [Keele (2007)]. We selected StArt after examining various SLR tools [Al-Zubidy and Carver (Al-Zubidy and Carver)]. After trying few of these tools we found StArt to be the most user-friendly and following all the steps prescribed for conducting an SLR. StArt assists in a step wise paper selection and extraction process as mentioned in [Keele (2007)].

Based on certain pre-defined keywords and parameters (discussed shortly), the StArt tool allowed us to catalogue all the ASE contributions (i.e. ASE tools and techniques) at one place, and also helped in analysis and visualization of the information along different dimensions.

2.1 Research Questions

A contextual understanding of the relationship of various aspects of ASE tools with the constituent techniques is essential for identifying and understanding the limitations and opportunities in the domain of ASE. To address the existing gaps in such an understanding is one of the motivation factors for us to review state of the art for ASE tools meant for all SDLC phases. Our main objective was to determine whether there exists any interesting patterns in respect of:

  1. relationships between the ASE tools and the issues/activities that these tools addressed.

  2. relationships between the constituent techniques and the ASE tools that leveraged them.

Another key issue that we intend to address is related to methodical selection of:

  1. an ASE tool by a practitioner who is looking to find a best fit for his/her needs, and

  2. constituent techniques by ASE researcher who is looking to develop the next ASE tool.

The above motivating factors led us to more descriptive research questions which we describe shortly. These research questions have been framed according to the five criteria recommended by Kitchenham et al. [Keele (2007)] i.e., Population, Intervention, Comparison, Outcome and Context (PICOC). The PICOC criteria for our SLR is depicted in Table-2.

The key research questions that we explored are as follows:

  1. [label=RQ.0: ]

  2. Are there any patterns of relationships between SDLC activities, the ASE tools and the constituent techniques.

    1. [label=0. ]

    2. What are the various constituent techniques that are used to automate different phases of SDLC? In other words, what methods and techniques from different areas of computing are leveraged when constructing automation tools of SDLC?

    3. Are these constituent techniques “SDLC activity specific”? In other words, does a particular constituent techniques find more acceptance in a particular activity of SDLC?

    4. Does a particular constituent technique span across more than one activity of SDLC?

    5. Have these constituent techniques gained/lost acceptance with time?

  3. To what extent does the automation of a particular phase takes place? Does this extent depend on that phase?

    1. [label=0. ]

    2. Is there a particular SDLC activity which is more difficult/easy to automate?

    3. Is level of automation constrained by availability of constituent techniques?

Fig. 2 depicts our SLR model and its expected outputs.

Figure 1: Research Model
Population Software Engineer (may be in one or more role of architect, designer, developer, tester)
Intervention Tools and Technologies to perform (semi) automation in fundamental software process development activity.
Comparison None
Outcome SEAT (Taxonomy)
Context Software Engineering and Automation
Figure 2: PICOC Criteria

2.2 Search Strategy

In order to find answers for our research questions we fetched relevant research literature. For this we first identified suitable sources/repositories of such research literature. Next, we formulated search strings in order to filter the most relevant articles from those sources. Details of these steps are as follows.

2.2.1 Repositories Searched

Due the vast amount of literature that exists in the ASE domain we restricted ourselves to ACM portal. This choice was due to following reasons: All major conferences in automation or software (e.g. Automated Software Engineering, International Conference on Software Engineering) are indexed by ACM. All journals of ACM related to software are highly ranked.

Also, if an ASE tool got improved in another paper (not indexed by ACM), we added the paper which discussed the improved version of that ASE tool. Finally, so as not to miss any important contribution we also added all those relevant paper which were cited in the selected papers (known as reference chaining [Achimugu et al. (2014)]). While searching for literature on specific repositories we used suitable search options which allowed multiple keyword searches. No restriction on the date of publishing or number of citations or any other constraint was put. Everything that was found via such search queries was considered for examining and analysis.

2.2.2 Composition of search queries

We formulated a template search query in which keyword parameters were dynamically set based on the SDLC phase for which we wanted to retrieve the ASE tools literature. Search queries were also executed for: a) synonyms of the given keyword/concept, and b) multiple keywords connected by booleans operators such as AND, OR.

2.3 Paper Selection

Our primary search elicited papers. These papers were then filtered based on title and abstract analysis. For each of these papers we went through their abstract and title and removed those which did not match the inclusion criteria (Table-2.3). The inclusion criteria is depicted in Table-2.3. As a result we were left with relevant papers. The number of papers was greatly reduced due to (among other reasons) duplicated and irrelevant matches that search queries returned. For example, keyword ‘automation’ had also matched papers related to ‘automation software industry’.

Paper Identification And Filtering Process Process description # of papers Papers obtained in primary search 1175 After Filtering Based on Title and Abstract Analysis 326 Papers obtained from Secondary Methods 19 Papers Rejected After Complete Reading 179 Papers Accepted 164 Quality Criteria Parameter Applicability area Is the aim of the research/article clearly articulated Generic Are the results mentioned in paper credible Generic Is the software development phase clearly mentioned SDLC Does the tooltechnique in practice decrease/replace any manual effort Automation

Inclusion and Exclusion Criteria Include a paper if: Exclude a paper if: It described automation of any SDLC activity. It presents merely an empirical study about an ASE problem, or It is not related to automation of an SDLC activity, or It describes automation in hardware platforms, even if via software.

We then added papers from secondary referencing. That is, relevant literature references that were present in the set of papers resulting from narrowed search. Also, any tool/technique which was proposed in the selected papers but had an improved version elsewhere was added. This lead to addition of more papers to the overall set selected. These papers were analyzed by complete reading. It was checked whether they really satisfied the inclusion criteria and related to our research questions. We chose a relatively narrow inclusion criteria because we wanted to select only those papers which fully/semi-automated one or more of the SDLC phases. We rejected papers which: (i) Did not propose a tool/technique/approach for automation, or (ii) If a paper provided an automation technique which did not explicitly assist in any SDLC phase (for example “automation in structured data management”), or (iii) If the automation is specific to some particular computing platform requirements like “automation for parallel distributed system”. These checks further led to removal of papers resulting in only articles. We assigned a unique ID (P1 through P164) to each of these papers for easy referencing. These IDs and other meta-data of all these papers can be accessed at: In subsequent sections we refer to a paper by its unique ID, e.g. as [P12].

2.4 Assessment of Relevance of Selected Papers

In this section we discuss how papers were assessed for their relevance for our study. This quality assessment was inspired by quality check idea provided in [Keele (2007)]. We assessed each paper in terms of whether it had adequate information to answer our research questions. The quality criteria is depicted in Table-2.3. For each parameter we marked the paper with “Yes” or “No”. A Paper having more than “No” was discarded. As all the selected papers had less than “No”, none was rejected in this step.

3 Information Extraction from Research Literature

To address our research questions, we created a data extraction form in StArt tool. Our main objective here was to scrutinize each of the papers individually and gather information to have an overall analysis of automation in SDLC. To this end we extracted two types of information from all these papers:

  1. Meta-data: Bibliographic information (such as title, author), a unique id, journal/conference details. Table-3 depicts the complete list.

  2. Qualitative Information: Information specific to our research questions. This was initially done on the basis of what the articles reported. Table-3 depicts entire list of qualitative features on which this data is based.

The extraction was conducted by the first author. This was entered in the StArt tool and shared with the second author. For validation, the second author checked few randomly selected papers on two parameters:

  1. Application of inclusion/exclusion criteria to a paper.

  2. The information that was extracted from a paper.

Any anomalies or differences in opinion were discussed. There were no differences in the second parameter (due to the very specific nature of questions in the extraction form). Although, in case of first parameter there were few differences due to difference in opinion of whether the work can be considered automation of task/s in SDLC phase. A consensus about such papers was reached by dividing each SDLC phase into a sub-phases (discussed in detail in Section-5 while discussing the proposed taxonomy, SEAT). If any sub-phases were being automated then the paper was included, otherwise it was excluded. All those papers that finally met the selection criteria were then included because they were found to be automating one or the other sub-phase of a phase of SDLC.

Extracted Meta-data Data item Description PID Each Paper has unique identifier generated automatically by StArt Title, Name, Author Details, Year Extracted from individual paper Article Type Whether a conference, journal paper or a dissertation Keywords In addition to those present in paper, also considered the ones added by us to help in further classification Type of work Whether a working tool, a theoretical approach or extension of existing tool

Extracted Qualitative Information Data item Description Why extracted SDLC phases automated The number and name of SDLC phases that were semi-/automated by the paper Its impact on our synthesized Taxonomy Degree of Automation Does the paper provide complete automation or semi-automation This is will decide the extent to which manual work has decreased. How was automation done? Which technique/s were used to achieve this automation Identify if there were any patterns in automating a phase with specific methodologies? Degree of user-effort Does it require the user to learn some new tool /technology Is automation really decreasing manual effort? Usage Does any software (open-source or otherwise) use the proposed tool or methodology? Has the tool found a practical or inspirational use in other such tools etc.

4 Discussion of Results in Context of Research Questions

In order to infer useful patterns and insights from these papers we extracted from these papers all such information which is related to research questions of our study. Further, to develop a systematic understanding of automation in different phases of SDLC, and to synthesize SEAT taxonomy we determined the pair-wise correlation among different types of information which we extracted from the selected papers. For example, the correlation between SDLC phases and the constituent techniques used by the ASE tools targeted for respective SDLC phase. Our findings from such analysis are discussed next. We first present the interesting patterns and related information which was observed. Then we discuss these observations in context of our research questions.

(For a paper specific detailed discussion around individual findings you may refer to the “extramaterial.pdf” that was uploaded with the submission)

4.1 Key Observations

Salient findings from our analysis are as follows:

Description of Constituent Techniques Constituent Techniques Specific Variant Mathematical Modelling (a) Logic: Inductive/Deductive Inferences, Logic Frameworks like Fluent Linear Temporal Logic, Boolean satisfiability problem (SAT) solvers, Satisfiability Modulo Theories (SMT) etc. ; (b) Automaton: Context Free Grammar (CFG), Context Sensitive Grammar (CSG), Finite State Machine (FSM); (c) Model Based Design and Verification. Artificial Intelligence

(a) Machine Learning: Clustering Hypothesis, Learning Algorithms; (b) Search-Based Techniques like hill-climbing, simulated annealing (c) Genetic Programming; (d) Expert System; (e) Programming By Example; (f) Evolutionary Algorithms.

Human-Computer Interactions (a) Natural Language Processing tools and techniques. Probabilistic Modelling (a) Probabilistic Model; (b) Language Model Statistical Inferencing (a) Statistical Model; (b) Statistical Analysis tools Programming Paradigms (a) Symbolic Execution; (b) Object Constrained Language; (c) Other Modeling Languages. Applied Mathematics

(a) Optimization techniques: Forward and Backward Slicing; (b) Numerical Analysis: Interpolation; (c) Graph Theory: Graph Analysis, Graph Based Modeling, Influence Graph, Decision Tree etc.

Pure Mathematics (a) Linear Algebra; (b) Refinement Calculus. Correlation-Based Inferencing (a) Knowledge Base: Ontology; (b) Repositories. Information Retrieval (a) Contextual Search; (b) Clustering Hypothesis Domain Engineering (a) Model Driven Architecture.

Distribution of articles Constituent techniques SDLC activity Requirement Architecture Design Implementation TestingVerification Maintenance Mathematical Modelling 11 3 6 16 47 7 Artificial Intelligence 1 6 1 10 19 8 Human-Computer Interaction 2 1 0 3 1 2 Probabilistic Modelling 0 0 0 3 2 0 Statistical Inferencing 1 0 0 0 2 1 Programming Paradigms 0 1 0 1 15 3 Applied Mathematics 5 0 3 2 9 3 Pure Mathematics 0 0 0 4 0 0 Correlation-Based Inferencing 5 2 0 7 4 4 Information Retrieval 0 0 0 1 4 1 Domain Engineering 1 0 0 0 6 5

  1. Our first major observation is the set of constituent techniques that have been leveraged by the researchers for developing ASE tools targeted for different SDLC activities. We classified the constituent techniques into categories. These categories of techniques and the specific technique under a category that we observed being used in each of papers are depicted in Table-4.1.

  2. Most of the constituent techniques span across all the SDLC activities and have changed over time due to advent of new tools and frameworks and advances in computing technologies. For example, basic logic constructs have been used to hand craft more sophisticated ones for solving ASE issues since early eighties, however, the current trend seem to be to use a pre-existing logic synthesis tools for the purpose. So instead of writing codes for deductive inferences many automation tools use available SAT solvers directly. In addition, newer techniques like evolutionary and genetic algorithms have been used with good results for automating activities in some SDLC phases such as Testing and Maintenance. By success of an automation tool or technique we mean: (i) The concerned ASE tool is able to reduce the manual effort of the corresponding SDLC activity, and (ii) The ASE tool has been further used in academia or industry directly or as an inspiration or basis for better tools.

  3. There are some constituent techniques which are suitable/preferred for automating a specific set of SDLC activities. This is due to the type of Input and/or Output expected by different activities. For example, Object Constrained Languages are used to automate architecture and design phase as it is used to define rules for UML. It is not seen to be used in automation of any other activity in SDLC.

  4. A significant number () of ASE tools have only a single constituent technique working as a core approach in achieving the tool’s goal. This seems to be in conformity with a common best practise to limit the technology diversity/sprawl in a software solution.

  5. Constituent techniques from Mathematical Modelling category are almost universally used by ASE tools across SDLC phases. Artificial Intelligence (AI) is another very popular category. Human-Computer Interaction (HCI), Statistical Inferencing, Probabilistic Modelling, Pure Mathematics and Information Retrieval (IR) are the categories from which fewer techniques have been used in ASE tools.

  6. The following SDLC phases, in that order, appear to be favoured the most by ASE researchers: Testing and verification, Implementation, Maintenance, Requirement Engineering. Architecture and Design phases seem to be the least favoured ones. Such popularity may imply the relative ease or difficulty in automating activities in these phases.

  7. Though almost all SDLC phases have seen increased automation over the years, implementation and testing and verification phases have received the most attention during last 10-15 years. Overall, there has been a sudden spurt in ASE contributions from the year 2000 onwards. Fig. 4 shows the trend.

  8. Over the years Mathematical Modelling remained a popular category to draw from when building ASE tools. AI is another category which has remained popular since 1990s. Techniques from IR category after having seen a dip in 1990s have again started to gain traction among ASE researchers. Other recent categories are Domain Engineering and Probabilistic Modelling. Fig. 5 shows the trend.

  9. We also determined the impact of ASE tools that we studied. Fig. 3 depicts our findings about impact. A large percentage () of the ASE tools were not used further in any manner. About of them are being used in other software systems or research projects. About of the ASE tools have been viewed as seminal works that lead to further work to develop better ASE tools. For example, [P63]111Complete list of papers that we studied is available at proposes Randoop222 which has found extensive usage in other research projects and in some OSS. It uses random testing which is feedback-directed and extended its functionality by using non-deterministic lexical analysis. Similarly, [P28] is a semi-automatic approach to assist in both architecture and design phase. It proposed SPE (Software Performance Engineering) and is considered a pioneering work. [P48] uses model checking for automation and is used in other research projects for code generation.

    The impact was determined mainly by exploring the citations as well as other types of references (such as on an OSS project which may have used the tool as a basis). We searched Github333On-line version control repository., Apache444Hosts number of open-source projects and Google® Search555On-line search engine to find their usage in Open Source Software (OSS) and any other systems.

    Figure 3: Automation Techniques in Terms of Usage
  10. If we study constituent techniques in isolation from what they are automating then the trends are mostly confusing. This is because a particular constituent technique may have found success in some ASE tool whilst not in others. Form our analysis we inferred that this success or failure is based on what is being automated. For example, evolutionary algorithms when used to automate testing and maintenance activities have shown considerable success than when used for automating requirement analysis.

Figure 4: Period vs. Automation in SDLC activities
Figure 5: Period vs. Usage of constituent approaches

4.2 Observations related to research questions

The research questions which we explored are mainly rooted around the following points:

  1. Identify if there exist any relationships between the ASE tools and the issues/activities that these tools addressed.

  2. Identify if there exist any relationships between the constituent techniques and the ASE tools that leveraged them.

  3. How these relationships evolved/changed with time (i.e. when technology landscape all around advanced).

In preceding section we have highlighted our key observations addressing the above root questions. Here we present the findings which relate to the specific questions that we enumerated in Section-2.1.

RQ.1: Our findings about patterns of relationships between SDLC activities, the ASE tools and the constituent techniques are depicted in Fig. 4, 5, and 3.

  • RQ.1.1: The constituent techniques that are used to automate different phases of SDLC have been categorised and listed in Table-4.1.

  • RQ.1.2: There is no clear pattern about a constituent technique being “SDLC activity specific”. In other words, it is difficult to say from our findings that a particular constituent technique finds more acceptance in a particular activity of SDLC.

  • RQ.1.3: There are constituent techniques which span across more than one activity of SDLC. Table-4.1 shows the distribution. For example, techniques from Mathematical Modelling category have been used in ASE tools catering to almost all SDLC phases.

  • RQ.1.4: As we have outlined in preceding section, constituent techniques have undergone changes in their reception. Fig. 5 depicts the observed trend of constituent techniques usage with time.

Figure 6: Activity Wise Automation

RQ. 2: To what extent does the automation of a particular SDLC activity take place? Does this extent depend on that activity? RQ. 2.1: Is there a particular SDLC activity which is more difficult/easy to automate?

The difficulty (or ease) of automation of activities in a particular SDLC phase can be deduced from the extent of automation available via ASE tools targeted for such activities. Only about papers in our SLR, proposed an ASE tool which completely automates the target activities. Other provide semi-automation. Our observations about extents of automation in different SDLC activities are depicted in Fig. 6. The actual number of papers in each activity were converted to percentage of papers for that activity, so as to provide a normalized value. As can be observed, requirement activity has the least number of completely-automated tools and testing the most.

An ASE tool is said to be offering complete automation if it accepts input in the form as is delivered from an earlier SDLC phase and gives output in the form which can be directly fed in to the next phase with minimum intervention from the user. The ASE tools which do not fall in the complete automation category are said to provide semi-automation.

Whether an ASE tool can be categorized as providing complete or semi automation can be inferred by considering the degree of effort required from a user when using the tool. Such evaluation usually involves examining the format of input and output of the tool. Table-4.2 and-4.2 describe the evaluation points used for such categorization.

Classification in Semi-automation Based on Input (In order of increasing user effort) S.No. Description of semi-automation 1. User interacts with tool in natural language 2. Code or Design in some preprocessed form 3. User has to learn a query language 4. User has to learn some high-level language 5. User has to learn some mathematical formalism like First-Order logic Classification in Semi-automation Based on Output (In order of increasing user effort) S.No. Description of Semi-automation 1. Fix the output at right place in that particular phase 2. User fixes the output at the right place 3. User has to choose one output (from a list of Outputs) which needs further modification 4. User intervention (during the running of algorithm) can fine-grain the output/s
Is there a particular SDLC activity which is more difficult/easy to automate?

If we distinguish in terms of extent of automation we find that most of the early activities of SDLC such as requirement engineering, architecture and design are mostly semi-automated and later activities like implementation, debugging and testing have completely-automated tools666With the exception of maintenance. This may be because maintenance requires more manual interventions than all other later activities. To recall, a completely-automated tool will require minimum possible human intervention/effort in addition to Input/Output constraints.. This makes us aware of a deeper relation between extent of automation and ease of automation of a particular activity. Each automation tool of SDLC activity would have aimed to completely automate it. The lesser it was able to do it suggests the difficulty of automation in that software activity. Thus, as shown in Fig. 6, requirements phase activities seem difficult to automate and those of testing phase appear to be the easiest to automate. We believe this is attributed to the input and output artefacts of these activities. The artefacts related to SDLC phases coming in the beginning of a software development project are found to be more abstract than the artefacts related to later SDLC phases.

4.3 Threats to Validity

The domain in which we conducted SLR is huge. The number of proposed ASE tools and techniques in few SDLC phases is enormous777Surveys exist for SDLC phase specific automation tools.. To conduct a reliable study in a reasonable time frame, we decided to limit the scope of our search to an authentic and widely accepted on-line repository – ACM portal – of research literature that provides latest research in the domain of our interest. This narrowing down may have led to missing of few studies. However, we have tried to make up for it by using reference chaining.

Another limitation could be the exclusion of those ASE tools which were targeted for specific platforms. For example, an ASE tool which assisted in automatic code generation for embedded systems was excluded from our study. This is because we desired to study those ASE tools which were agnostic to specific target platforms.

5 SEAT – A Taxonomy for Software Engineering Automation

Appropriate level of abstraction is desirable when describing and organizing scientific knowledge so as to effectively apply such knowledge for addressing practical issues [Sjoberg et al. (2007)]. The concepts in a domain and the inter-relationships among them can be structured at several relevant levels of abstraction by arranging such knowledge as a taxonomy. The observations about ASE domain as presented in preceding sections provide a basis for us to develop a taxonomy for automation in software engineering; we call this taxonomy SEAT – Software Engineering Automation Taxonomy.

5.1 Motivation and development process

A taxonomy of automation in software engineering is desirable because, among other uses, it will enable synthesizing new automation tools for different SDLC activities. Re-composing existing systems to achieve better features will also be possible. Such a taxonomy can serve as a semantic tool which will allow for methodical reasoning about various aspects of automation in software engineering tasks. For instance, an ASE researcher can infer the alternatives when identifying constituent techniques to use in an ASE tool that he/she is building. The taxonomy, in addition to being useful for research community, will also serve as an effective aid when teaching automation in software engineering.

A taxonomy can be developed either in a top-down approach or in a bottom-up. In a top-down approach one requires an existing schema of concepts classification in the concerned domain. The bottom-up approach starts by first capturing the knowledge/identification of existing concepts in the domain for which taxonomy is to be developed [Unterkalmsteiner et al. (2014)].

We adopted a bottom-up approach when building SEAT. We chose this approach mainly because to the best of our knowledge there wasn’t any classification schema which exists for the ASE domain covering all SDLC activities (though few literature surveys in specific SDLC phases exist). As such, a top-down approach would not be suited. The evolution process of SEAT is depicted in Fig. 7.

Figure 7: SEAT - The Evolution

We developed an initial hypothesis considering two fundamental aspects of an ASE tool: a) what is it that the tool is automating, and b) how is such an automation achieved.

The aim was to capture inter-relationships among constituents of ASE approaches, tools and techniques by the how dimension. Similarly the what dimension was chosen to enable our taxonomy to capture relationships with respect to intended goals of various ASE approaches, tools and techniques. As a next step we performed a systematic literature review (SLR) in the ASE domain. The categories generated by SLR were then used to validate888Validation was performed on the basis of first research question discussion in Section-4. our initial hypothesis. The classification with which we began in our hypothesis and the one which we inferred from SLR were found to be largely in sync. As such we used the classification schema which we inferred from SLR to build our taxonomy.

5.2 Discussion

SEAT is a two-dimensional, non-orthogonal taxonomy. The overall structure of SEAT is depicted in Fig. 8.

Figure 8: Taxonomy of Automation tools for SDLC activities

We observed from our SLR that automation in software engineering has largely been aligned with the fundamental activities of software engineering processes. These activities are: 1) Requirement Engineering 2) Design 3) Implementation 4) Validation (testing/verification) and 5) Evolution of software. Objectives, inputs and outputs of each of these activities are well understood and defined by the software engineering community [Bourque et al. (1999)][Sommerville (2010)]. Therefore it seems fit to classify various ASE tools along these these activities (i.e. the “what” dimension).

In Section-4 we identified several constituent techniques, from different areas of computing, that have formed the basis of major work in ASE domain. Such constituent techniques have been used to automate different software engineering activities. The identified categories of the constituent techniques are shown in Fig 8.

The abstractions/concepts that we found in both dimensions (i.e. what and how) are depicted as the second level branches in Fig.-8. We further refine them by identifying the specific SDLC activity which the ASE tool automates (what dimension), and also by identifying the specific constituent technique (how dimension) that the ASE tool used. Further, the third level of branching in SEAT depicts the set of concept instances that were obtained from our systematic literature review.

An ASE tool may automate more than one activity of SDLC. Also, the ASE tool may make use of more than one constituent technique to achieve automation. For instance, [P162] automates both implementation and testing activity of SDLC using machine learning and a pre-defined language model. In essence, the tool spans two activities of “what” dimension - namely Software Implementation/Construction and Testing (Level 2, left branch in Fig. 8). In addition, two approaches of “how” dimensions are used - Artificial Intelligence and Natural Language Processing (Level 2, right branch in Fig. 8). This makes SEAT a non-orthogonal taxonomy since a particular ASE tool may span across different sub-categories of “how” and “why” dimension.

5.3 SEAT In Action

We observe that SEAT is a directed graph which follows a hierarchy. Therefore we stored it in a graph database for extracting inferences and further synthesis of additional knowledge. We created a graph using neo4j® to depict all nodes and their relationships in SEAT. neo4j is an open-source graph database. It provides all the characteristics of a traditional database, in addition to implementing property-graph model. Each node corresponds to a category of SEAT. The level of node in SEAT is depicted as its property in neo4j to maintain hierarchy. We further populated the graph by creating relationships between leaves that are shown in Fig. 8. These inter-leave relationships are based on the actual data obtained from our systematic literature review. The schema of the graph database storing SEAT is depicted in Fig. 9. It is basic structure of SEAT and shows various types of nodes in our graph database.

An advantage of storing the taxonomy in a graph database is that it allows attaching additional data to nodes as well as relationships. For example, in Fig. 9 we have added two additional properties in the graph: a) time_factor: A numeric measure that indicates the relative usage of a ‘constituent technique’ in the last years. b) usage_type: It is a property that indicates the impact of a constituent technique in terms of influencing/spurring more ASE research.

Figure 9: Schema (basic structure) of SEAT stored as graph database

The idea behind storing SEAT in graph is two-folds: a) The graph can further be extended and scaled as more data in this domain is gathered. b) A user can query the graph directly to find the relations between various concepts present in SEAT, instead of manually going through the sizeable information that the taxonomy represents. Also, additional information in the graph helps a user in making useful deductions via graph queries999neo4j provides a user friendly, SQL like, query language called Cypher (

In order to test the practicality and utility of SEAT-graph we depict its functioning with three unique and distinct use cases.

  1. USE CASE-1: A basic scenario involves applying SEAT to an ASE tool101010These tools are not part of the SLR we conducted. for classifying the tool. For instance, a software engineer studies [Goldstein and Segall (2015)]-paper and wants to see its placement w.r.t to SEAT. She can run Query-1 (written in Cypher) by providing relevant information from the paper in question. Line and in Query-1 correspond to the process of matching ‘What’ and ‘How’ dimensions with ‘Architecture Validation’ and ‘Graph’ respectively. The in these statement means that the relationship may traverse more than path in the graph.

    1 MATCH (a)-[*1]->(b) WHERE a.Dimension contains What
    2 AND contains ‘Architecture Validation’
    3 MATCH (c)-[*1]->(d) WHERE c.Dimension contains How
    4 AND contains ‘Graph’
    5 RETURN (a)-[]->(b), (c)-[]->(d)
    Query 1: Applying SEAT to existing ASE tool

    row in Table-1 depicts the results of Query-1 when executed against our database. “Software Architecture/High-level Design” is related to ‘Architecture Validation’ in What dimension and “Applied Mathematics” is related to ‘Graph’ in How dimension.

    Results of queries against SEAT-graph Query-No. Results Query1 ( Software ArchitectureHigh-level Design, node.Dimension: What), (node.Belongsto: SDLC, Architecture Evaluation or Architecture Validation) ( Applied Mathematics, node.dimension: How, node.timefactor: 3), (node.Belongsto: CT, Graph Theory) Query2 ( Search-Based Techniques) ( automates, rel.usage:researchproject) ( Design Analysis) ( Graph Theory) ( automates, rel.usage:researchproject) ( Requirement Analysis) Query3 ( Mathematical Model, cnt:11), ( Artificial Intelligence, cnt:10), ( Correlation-based Inferencing, cnt:9), ( Domain Engineering, cnt:8), ( Programming Paradigms, cnt:7), ( Probabilistic Modelling, cnt:6), ( Human-Computer Interaction, cnt:5), ( Pure Mathematics, cnt:4), ( Information Retrieval, cnt:3), ( Applied Mathematics, cnt:3), ( Statistical Inferencing, cnt:1)

    row of Table -1 depicts the application of SEAT on [Goldstein and Segall (2015)]-paper. Similarly, the graph of SEAT can be used to find application of SEAT to any other existing ASE tool (Table-1 is populated with two other similar examples).

    ASE approaches w.r.t SEAT ASE tool Dimension-What Dimension-How Automatic and Continuous Software Architecture Validation [Goldstein and Segall (2015)] Architecture EvaluationArchitecture Validation Applied Mathematics Speculative Requirements: Automatic Detection of Uncertainty in Natural Language Requirements [Yang et al. (2012)] Requirement Analysis Human-Computer Interaction and Mathematical Modelling AutoComment: Mining Question and Answer Sites for Automatic Comment Generation [Wong et al. (2013)] Maintenance Analysis Mathematical Modelling and Information Retrieval

  2. USE CASE-2: Suppose a software engineering researcher desires to find all constituent techniques that can be used to automate ‘analysis activity’ of an SDLC phase. Further, she is interested in identifying only those constituent techniques which have influenced the development of further ASE tools. Accordingly, Query-2 is executed. Line in Query-2 filters the results based on usage property of the relationship.

    1 MATCH (a)-[r]->(b) WHERE CONTAINS Analysis
    2 AND r.usage CONTAINS ‘research’
    3 RETURN (,[type(r)],[r.usage],(
    Query 2: Using SEAT to search for most effective constituent approach

    The results, depicted in row of Table-1, show that “Graph Theory” when used in automation of requirement analysis and “Search-Based Techniques” in analysis of software design has led to ASE tools which have further found usage in other research tools.

  3. USE CASE-3: Suppose a researcher is exploring the possibility of developing an ASE tool for assisting in an SDLC activity. She would like to identify the most recent constituent techniques which she may consider. Without the SEAT graph, identifying such information would require sifting through a significant volume of literature. However, this task can be quickly done by running Query-3 which identifies such a ranked list of constituent techniques. Result of the query is depicted in row of Table-1. The results show that “Mathematical Modelling” has been the most extensively used techniques in the last six years and “Statistical Inferencing” the least.

    1 MATCH (n:Constituent_Technique)
    2 RETURN, (n.time_factor) AS cnt
    3 ORDER BY cnt DESC
    4 LIMIT 15
    Query 3: Using SEAT acquire ranked list of constituent techniques

In addition to specific use cases discussed above, we provide examples of few other direct inferences that can be extracted from SEAT-graph. These answers give an idea about the scope of research or tool development that can be done in the domain of ASE.

  • Q. Is there any SDLC activity whose ASE tools although being used in research projects/inspiring further research but never used in any software system (open or otherwise)?

    Ans. Implementation/Code generation

  • Q. What are the constituent techniques which have formed the basis of impactful ASE tools (the tool was further used) in different SDLC phases?

    Ans. Requirement Engineering- Graph Theory
    High-level Design- Graph Theory, Logic
    Low-level Design- Search-based techniques, Logic
    Implementation- Logic, knowledge-base
    Testing/verfification- Model Based Design and Verification
    Maintenance- Logic and Contextual Search

  • Q. In which phase of SDLC an ASE tool is likely to be most impactful?

    Ans. Testing/Verification

  • Q. Suppose a software enginner desires to find all those constituent techniques which whenever used resulted in an impactful ASE tool.

    Ans. Programming Paradigms ( rank in terms of recent usage), Information Retrieval ( rank in terms of recent usage).

6 Conclusions

Enhancing the automation of software development tasks has significant far-reaching implications. Automated software engineering (ASE) is an active area of research that aims to replace much of human programmer efforts by offering automation of such tasks. The work in ASE domain relies on tools and techniques from other diverse areas (e.g. information modelling, semantic computing, NLP etc.) of computing and related disciplines. In order to be effective in ASE domain not only is one required to have a systematic understanding of the ASE tools, but also the constituent techniques that such tools leverage. As such a comprehensive characterisation of ASE tools, their constituent techniques as well as their relationships with the SDLC activities that they automate is highly desirable. One of the major contribution of this paper is to provide such a characterisation.

We have developed a taxonomy called SEAT through an extensive systematic literature review of the ASE domain research articles. About 1175 research articles were collected and after applying methodical filtering and elimination we studied about 167 articles in depth. We found that the predominant constituent techniques which are used in various ASE tools can be grouped into 11 categories (shown in Table-4.1). From our comprehensive study we have been able to identify important relationships among ASE tools and their constituent techniques. We have identified interesting trends and patterns in the way various ASE tools use different constituent techniques.

For example we observe that in case of about 74% ASE tools there is a single constituent technique which serves as the core idea behind the tool. Similarly, we observed that more than 50% ASE tools that we sampled targeted activities in the Testing and Verification phase of SDLC. One implication of this observation may be that the SDLC activities where population of ASE tools is relatively low are difficult to automate. We also observed that certain constituent techniques have been consistently relied upon for automating activities across almost all SDLC phases. Mathematical Modelling and Artificial Intelligence are two such example categories. In our study we also examined how the use of various constituent techniques varied over the years, and for what types of SDLC activities. Key observations here are that: a) The automation in SDLC phases has in general seen upward trend with Testing and Verification having more than double the activity than any other phase, and b) Techniques from Information Retrieval category after having seen a dip in 1990s have again started to gain traction among ASE researchers. We also observed that a significant number () of ASE tools from our sample were not used further in any manner.

The relationships among ASE tools, their constituent techniques and the SDLC activities, and also the useful properties that we observed formed the basis for the taxonomy (SEAT) that we proposed. SEAT has been realized as a graph database. The graph database allows uncovering hidden relationships via exhaustive search for connections and paths between different nodes (i.e. concepts). We have demonstrated the efficacy of SEAT by discussing few practical use cases. We believe that SEAT will enable better comprehension of the ASE domain and assist in identification of new important research opportunities in the domain of automated software engineering.


  • [1]
  • Achimugu et al. (2014) Philip Achimugu, Ali Selamat, Roliana Ibrahim, and Mohd Naz’ri Mahrin. 2014. A systematic literature review of software requirements prioritization research. Information and Software Technology 56, 6 (2014), 568–585.
  • Al-Zubidy and Carver (Al-Zubidy and Carver) Ahmed Al-Zubidy and Jeffrey C Carver. Review of Systematic Literature Review Tools. (????).
  • Anand et al. (2013) Saswat Anand, Edmund K Burke, Tsong Yueh Chen, John Clark, Myra B Cohen, Wolfgang Grieskamp, Mark Harman, Mary Jean Harrold, Phil McMinn, and others. 2013. An orchestrated survey of methodologies for automated software test case generation. Journal of Systems and Software 86, 8 (2013), 1978–2001.
  • Bourque et al. (1999) Pierre Bourque, Robert Dupuis, Alain Abran, James W Moore, and Leonard Tripp. 1999. The guide to the software engineering body of knowledge. IEEE software 16, 6 (1999), 35.
  • Fabbri et al. (2012) Sandra Fabbri, Elis Montoro Hernandes, André Di Thommazo, Anderson Belgamo, Augusto Zamboni, and Cleiton Silva. 2012. Managing Literature Reviews Information through Visualization.. In ICEIS (2). 36–45.
  • Fabbri et al. (2016) Sandra Fabbri, Cleiton Silva, Elis Hernandes, Fábio Octaviano, André Di Thommazo, and Anderson Belgamo. 2016. Improvements in the StArt tool to better support the systematic review process. In Proceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering. ACM, 21.
  • Goldstein and Segall (2015) Maayan Goldstein and Itai Segall. 2015. Automatic and continuous software architecture validation. In Software Engineering (ICSE), 2015 IEEE/ACM 37th IEEE International Conference on, Vol. 2. IEEE, 59–68.
  • Gulwani (2010) Sumit Gulwani. 2010. Dimensions in program synthesis. In Proceedings of the 12th international ACM SIGPLAN symposium on Principles and practice of declarative programming. ACM, 13–24.
  • Keele (2007) Staffs Keele. 2007. Guidelines for performing systematic literature reviews in software engineering. In Technical report, Ver. 2.3 EBSE Technical Report. EBSE.
  • Rafi et al. (2012) Dudekula Mohammad Rafi, Katam Reddy Kiran Moses, Kai Petersen, and Mika V Mäntylä. 2012. Benefits and limitations of automated software testing: Systematic literature review and practitioner survey. In Proceedings of the 7th International Workshop on Automation of Software Test. IEEE Press, 36–42.
  • Sjoberg et al. (2007) Dag IK Sjoberg, Tore Dyba, and Magne Jorgensen. 2007. The future of empirical methods in software engineering research. In 2007 Future of Software Engineering. IEEE Computer Society, 358–378.
  • Sommerville (2010) Ian Sommerville. 2010. SOFTWARE ENGINEERING. Pearson.
  • Unterkalmsteiner et al. (2014) Michael Unterkalmsteiner, Robert Feldt, and Tony Gorschek. 2014. A taxonomy for requirements engineering and software test alignment. ACM Transactions on Software Engineering and Methodology (TOSEM) 23, 2 (2014), 16.
  • Wong et al. (2013) Edmund Wong, Jinqiu Yang, and Lin Tan. 2013. Autocomment: Mining question and answer sites for automatic comment generation. In Automated Software Engineering (ASE), 2013 IEEE/ACM 28th International Conference on. IEEE, 562–567.
  • Yang et al. (2012) Hui Yang, Anne De Roeck, Vincenzo Gervasi, Alistair Willis, and Bashar Nuseibeh. 2012. Speculative requirements: Automatic detection of uncertainty in natural language requirements. In Requirements Engineering Conference (RE), 2012 20th IEEE International. IEEE, 11–20.