Establishing a Search String to Detect Secondary Studies in Software Engineering

12/18/2019
by   Bianca Minetto Napoleao, et al.
0

Search for secondary studies is essential to establish whether the review on the intended topic has already been done, avoiding waste time. In addition, secondary studies are the inputs of a tertiary study. However, one critical step in searching for secondary studies is to elaborate a search string. The main goal of this work is to analyze search strings to establish directions to better detect secondary studies in Software Engineering (SE). We analyzed seven tertiary studies under two perspectives: (1) structure - strings' terms to detect secondary studies; and (2) field: where searching - titles alone or abstracts alone or titles and abstracts together, among others. We also performed a validation of the results found. The suitable search string for finding secondary studies in SE contain the terms "systematic review", "literature review", "systematic mapping", "mapping study", "systematic map", "meta-analysis", "survey" and "literature analysis". Furthermore, we recommend (1) researchers use the title, abstract and keywords search fields in their searches to increase studies recall; (2) researchers choose carefully their paper title, abstract and keyword terms to increase the chance of having such studies found on digital libraries.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

04/27/2021

Grey Literature in Software Engineering: A Critical Review

Context: Grey Literature (GL) recently has grown in Software Engineering...
07/10/2020

Secondary Studies in the Academic Context: A Systematic Mapping and Survey

Context: Several researchers have reported their experiences in applying...
04/21/2020

On the Performance of Hybrid Search Strategies for Systematic Literature Reviews in Software Engineering

Context: When conducting a Systematic Literature Review (SLR), researche...
09/16/2021

Inclusion and Exclusion Criteria in Software Engineering Tertiary Studies: A Systematic Mapping and Emerging Framework

Context: Tertiary studies in software engineering (TS@SE) are widely use...
04/03/2020

A Tertiary and Secondary Study Canvas

Over the past years, more secondary (Systematic Literature Reviews and S...
10/07/2021

How Tertiary Studies perform Quality Assessment of Secondary Studies in Software Engineering

Context: Tertiary studies are becoming increasingly popular in software ...
02/20/2020

How to Evaluate Solutions in Pareto-based Search-Based Software Engineering? A Critical Review and Methodological Guidance

With modern requirements, there is an increasing tendancy of considering...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Systematic Literature Reviews (SLR) also known as Systematic Reviews (SR) and Systematic Mappings (SM) are known as secondary studies. Their goal is to identify and summarize research evidence on several research topics in Software Engineering (SE) [32]. This synthesis provides a complete and fair evaluation of the state-of-the-art of all relevant research available for a specific topic of interest. Tertiary studies are considered as a review that focuses only on secondary studies [32].

One challenge in conducting secondary and tertiary studies is related to searches for relevant studies [4, 5, 16, 51, 59]. Dieste et al. [15] argue that one critical step in conducting secondary studies is to design and execute appropriate and effective search strategy, which needs to be carefully planned and implemented. Definition of a search strategy is a time-consuming and error-prone step.

Overall, Digital Libraries (DLs) searches using search strings is the main strategy used to carrying out searches to conduct secondary and tertiary studies. This search approach is known as automatic search. However, DLs are not well-suited to support automated searches in secondary studies, i.e, a search string in a given format does not work in all DLs [30]. In this scenario, a key issue is to elaborate search strings adapted to each DL [5]. The use of a broader search string could help reviewers during the search for studies [54]. For instance, a broader search string with few terms is easier to adapt, however, choosing few terms can result in loss of evidence due to some relevant studies can not be returned.

Another challenge is the definition of keywords and their appropriate combinations for search purposes [7]. One reason for this difficulty is the lack of formalization of the terminologies in most of the research topics/domain in which SLRs have been conducted [56]

. In general, the probability that two researchers use the same term to refer to the same concept is often lower than 20%

[53]. Sjøberg et al. [50] agree that there is no common terminology and appropriate descriptors and keywords in the SE area.

For building a search string, researchers need to be familiar with the specific search terms with respect to the research topic. The search string definition is a fundamental step for secondary studies success. However, the terms combination capable of finding the largest possible number of relevant studies accurately requires experience and knowledge about the research area [30]. Moreover, a search string can be formed by different combinations of terms. These terms have synonyms, spelling variations, acronyms, and correlates. The identification of the most suitable search string for each review is, in fact, an inherently iterative activity [17]. Therefore, it is essential to verify which terms have real impact in detecting relevant studies and also eliminate terms that reduce search accuracy and do not aggregate relevant studies to the study.

SE researchers look for secondary studies for two main reasons: (1) To establish whether a secondary study on a topic has already been done, avoiding time wasting and energy in its conduction. If the review already exists however it needs to be updated then it is recommended to update it; (2) To perform a tertiary study. A tertiary study is an SLR of SLRs and it follows exactly the same method used to conduct SLR [32]. According to Kitchenham [32], tertiary studies are performed when a considerable number of secondary studies on a particular topic of interest exist. A tertiary study is a kind of SLR, however the inputs are secondary studies: SLRs and SMs. There are many challenges associated with how it is conducted, including the search for the inputs.

In this context, which term should be used to find a secondary study? Many are the possibilities: “review of studies” OR “structured review” OR “systematic review” OR “literature review” OR “literature analysis” OR “in-depth survey” OR “literature survey” OR “meta-analysis” OR “past studies” OR “subject matter expert” OR “analysis of research” OR “empirical body of knowledge” OR “evidence-based software engineering” OR “overview of existing research” OR “body of published research”. Which of these options have been adopted by SE researchers conducting secondary studies?

Some authors have suggested methodologies to select databases for conducting SLRs [8], approaches/guidelines and tools [57, 20, 38, 48] for assisting the review process. However, there is no consensus on what terms should compose search strings and the preferable field(s) to detect secondary studies. We argue that it is unclear how to decide which terms to include in the search string to find secondary studies. Furthermore, it is difficult to determine when the search string is complete and which DLs select to perform the search.

In this paper, we propose a suitable search string to detect secondary studies on SE, addressing issues as quantity of applied terms, relevance and effectiveness. We evaluated search strings used by SE researchers to detect secondary studies in SE. Furthermore, we provide recommendations for researchers conducting their SLR work regarding to the choose of terms for title, abstract and keywords.

The remainder of this paper is organized as follows: Section 2 details the study design applied to evaluate search strings for detecting secondary studies, followed by addressing research questions (Sections 3 and 4), search string validation (Section 5), discussions of the results (Section 6), threats to the validity (Section 7) and related works (Section 8). Finally, Section 9 concludes our work.

2 Study Design

Garousi and Mäntylä [19] performed a tertiary study where they also have identified ten tertiary studies in SE. The list of these studies are presented in Table I sorted according to the years they were published. It is possible to observe that tertiary studies in SE have started to appear after 2009 and that most of the studies focus on the general SE [9, 10, 11, 27, 29, 33, 59]. Only three studies are related to two specific SE sub-areas: (1) distributed software development [23, 36]; and (2) global software development [55]. Table II partially describes search strings of Garousi and Mäntylä’ list, which are used to detect relevant secondary studies in SE. We decided to use this list as initial basis for conducting our research.

ID Title Year Reference
S1 Systematic literature reviews in software engineering – A systematic literature review 2009 [29]
S2 Systematic literature reviews in software engineering – A tertiary study 2010 [33]
S3 Six years of systematic literature reviews in software engineering: an extended tertiary study 2010 [10]
S4 A critical appraisal of SRs in SE from the perspective of the research questions asked in the reviews 2010 [11]
S5 Identifying relevant studies in software engineering 2011 [59]
S6 Research synthesis in software engineering: a tertiary study 2011 [9]
S7 Signs of agile trends in global software engineering research: a tertiary study 2011 [23]
S8 Systematic literature reviews in distributed software development: a tertiary study 2012 [36]
S9 A tertiary study: experiences of conducting systematic literature reviews in software engineering 2013 [27]
S10 Risks and risk mitigation in global software development: a tertiary study 2014 [55]
TABLE I: Tertiary studies in SE [19]
Ref. Search string
[29] No search string
[33] “review of studies” OR “structured review” OR “systematic review” OR “literature review” OR “literature analysis” OR “in-depth survey” OR “literature survey” OR “meta-analysis” OR “past studies” OR “subject matter expert” OR “analysis of research” OR “empirical body of knowledge” OR “Evidence-based software-engineering” OR “evidence-based software engineering” OR “overview of existing research” OR “body of published research”
[10] “review of studies” OR “structured review” OR “systematic review” OR “literature review” OR “literature analysis” OR “in-depth survey” OR “literature survey” OR “meta analysis” OR “past studies” OR “subject matter expert” OR “analysis of research” OR “overview of existing research” OR “body of published research” OR “evidence-based” OR “evidence based” OR “study synthesis” OR “study aggregation”
[11] No search string
[59] (systematic OR controlled OR structured OR exhaustive OR comparative) AND (review OR survey OR “literature search”)
[9] Title=(systematic review)
[23] systematic review OR systematic literature review OR systematic map OR systematic mapping OR mapping study
[36] (“systematic review” OR “systematic literature review” OR “systematic map” OR “systematic mapping” OR “mapping study”)
[27] (“review of studies” OR “structured review” OR “systematic review” OR “literature review” OR “literature analysis” OR “in-depth survey” OR “literature survey” OR “meta analysis” OR “past studies” OR “subject matter expert” OR “analysis of research” OR “overview of existing research” OR “body of published research” OR “Evidence based” OR “evidence based” OR “study synthesis” OR “study aggregation” OR “systematic literature review” OR SLR)
[55] (“review of studies” OR “structured review” OR “systematic review” OR “literature review” OR “systematic literature review” OR “literature analysis” OR “in-depth survey” OR “literature survey” OR “meta-analysis” OR “analysis of research” OR “empirical body of knowledge” OR “overview of existing research” OR “body of published knowledge”)
TABLE II: Search strings raised in the SE tertiary studies

Out of the 10 tertiary studies identified by Garousi and Mäntylä [19], we considered seven studies (ID S2, S3, S6, S7, S8, S9, S10). These studies were selected for two main reasons: (i) they were conducted and double-checked by reviewers with experience in conducting secondary studies; and (ii) they contained all necessary data (e.g., list of included studies and search string) to be used in our analyses. Consequently, we excluded study S1 because it just used manual search. Study S4 used in its research data collected in other two tertiary studies [29, 33], both already included in our analysis. Finally, to perform a deeper analysis, as mentioned above, we considered only tertiary studies that have the list of included secondary studies available, therefore S5 was excluded.

We analyzed the set of included secondary studies in the selected tertiary studies to answer two Research Questions (RQs) under two perspectives, detailed as follows:

– Perspective 1 (Structured analysis): RQ1: Which terms should be used to detect secondary studies for tertiary studies? The focus was to identify terms and synonyms that compose search strings created to detect secondary studies. As shown in Figure 1, the structured analysis was performed in two steps. During step 1.1, we considered string terms related to secondary studies, i.e., terms related to research domain were not considered. During step 1.2, these terms were organized in a list to reveal the terms and their synonyms most used by researchers to detect secondary studies. The results are presented in Section 3 – (RQ1).

– Perspective 2 (Search field): RQ2: What is the preferable field(s) to search secondary studies: (e.g. titles alone or titles and abstracts together, among others)? The focus was to identify preferable field(s) (search in title alone, or abstract alone, or title and abstract together, etc.) to detect secondary studies for tertiary studies. The analysis was executed in two steps. Initially, during step 2.1, we listed the secondary studies included in each tertiary study and downloaded all of them. A total of 337 were detected, however one study was not available for download, totalizing 336 studies. In the sequence, step 2.2, we checked the appearance of string terms in title, abstract and keywords of each secondary study. The results are showed in Section 4 – (RQ2).

Fig. 1: Study design

We used a data extraction form to extract information and enable to answer our RQs. The extracted data is available online on http://goo.gl/k5bNp6.

3 RQ1: Which terms should be used to detect secondary studies for tertiary studies?

Considering the seven tertiary studies selected by us, we extracted and grouped in a single list all search string terms used to find secondary studies. These terms were counted and the numbers of appearance are listed in Table III.

The terms “meta analysis” and “meta-analysis” were considered as a unique term (T12) in Table III, since according to Singh and Singh [48] the character “-” is interpreted as a blank space by several DLs, such as IEEEXplore, ACM DL, SpringerLink, Science Direct and Wiley.

ID Search String Term Occurrence
T1 systematic review 7
T2 literature review 4
T3 systematic literature review 4
T4 review of studies 4
T5 structured review 4
T6 literature analysis 4
T7 in-depth survey 4
T8 literature survey 4
T9 analysis of research 4
T10 empirical body of knowledge 4
T11 overview of existing research 4
T12 meta analysis or meta-analysis 4
T13 past studies 3
T14 subject matter expert 3
T15 body of published research 3
T16 evidence based 2
T17 study synthesis 2
T18 study aggregation 2
T19 systematic mapping 2
T20 mapping study 2
T21 body of published knowledge 1
T22 evidence-based software-engineering 1
T23 SLR 1
T24 systematic map 1
TABLE III: Terms from tertiary studies string sorted by appearance number

The most used term in tertiary studies was “systematic review” (T1). This term appeared in all seven tertiary strings (100%), see Table III – line 2. The terms “literature review” (T2) and “systematic literature review” (T3) were used in four search strings (57.14%) individually. However, the term “literature review” is part of the term “systematic literature review” and in six (85.71%) of the seven tertiary studies at least one of these terms composed the string. Moreover, in the search strings of studies S9 and S10 both terms (T2 and T3) appeared together. Terms related to systematic mapping (T19, T20, T24) had a lower appearance than terms related to SLR (T1, T2, T3). Similarly, there were less synonyms to SM than to SLR.

Tertiary Study Terms combination
S2 T1, T2, T6, T8, T9, T12
S3 T1, T2, T4, T5, T8, T9, T12, T16
S6 T1
S7 T2, T3
S8 T1, T3, T24
S9 T1, T2, T3, t4, T8, T16, T23
S10 T1, T2, T3, T6, T9
TABLE IV: String terms combination that returned secondary studies from tertiary studies
ID S2 S3 S6 S7 S8 S9 S10 Totals
Total of Included
Secondary Studies
33 67 49 12 23 116 37 337
Terms T1 9(27.27%) 41(61.19%) 37(75.51%) 0(0.00%) 7(30.43%) 67(57.76%) 13(35.14%) 174
T2 8(24.24%) 26(38.81%) 0(0.00%) 4(33.33%) 0(0.00%) 50(43.10%) 15(40.54%) 103
T3 0(0.00%) 0(0.00%) 0(0.00%) 4(33.33%) 10(43.48%) 46(39.66%) 13(35.14%) 73
T4 0(0.00%) 2(2.99%) 0(0.00%) 0(0.00%) 0(0.00%) 3(2.58%) 0(0.00%) 5
T5 0(0.00%) 1(1.49%) 0(0.00%) 0(0.00%) 0(0.00%) 0(0.00%) 0(0.00%) 1
T6 1(3.03%) 0(0.00%) 0(0.00%) 0(0.00%) 0(0.00%) 0(0.00%) 1(2.70%) 2
T8 1(3.03%) 2(2.99%) 0(0.00%) 0(0.00%) 0(0.00%) 2(1.72%) 0(0.00%) 5
T9 1(3.03%) 1(1.49%) 0(0.00%) 0(0.00%) 0(0.00%) 0(0.00%) 1(2.70%) 3
T12 2(6.06%) 1(1.49%) 0(0.00%) 0(0.00%) 0(0.00%) 3(2.58%) 0(0.00%) 6
T16 0(0.00%) 4(5.97%) 0(0.00%) 0(0.00%) 0(0.00%) 24(20.69%) 0(0.00%) 28
T23 0(0.00%) 0(0.00%) 0(0.00%) 0(0.00%) 0(0.00%) 19(16.38%) 0(0.00%) 19
T24 0(0.00%) 0(0.00%) 0(0.00%) 0(0.00%) 1(4.35%) 0(0.00%) 0(0.00%) 1
T1: systematic review; T2: literature review; T3: systematic literature review; T4: review of studies; T5: structured review;
T6: Literature analysis; T8: literature survey; T9: analysis of research; T12: meta analysis or meta-analysis; T16: evidence basead;
T23: SLR; T24: systematic map
TABLE V: String terms that returned secondary studies

Analyzing Tables IV and V it is possible to identify string terms that returned secondary studies. Terms T7, T10, T11, T13, T14, T15, T17, T18, T19, T20, T21 and T22 are synonymous to SLR, however, in SE context, they are not being used for investigations to describe their reviews. For this reason, they are not described in Tables IV and V. For example, the tertiary study S2 analyzed 33 secondary studies. The string adopted by S2 was composed by terms T1, T2, T6, T8, T9, T12 (see Table IV – line 2). Terms T7, T10, T11 and T15, which also composed the study search string, did not return studies. Term T1 (see Table V – line 3) returned nine of the 33 studies (27.27%). On one hand, T3 did not find studies because it was not part of the string (see Table V – line 3). On the other hand, although T4 composed the string, it did not return studies. Due to these two reasons, both terms (T3 and T4) did not appear in Table IV – line 2.

T1 returned 174 (51.78%) studies of the 336 secondary studies detected by the seven tertiary studies altogether. A total of 103 (30.65%) studies are returned by T2 and 73 (21.73%) was achieved using T3.

One point to be discussed is that one study can be returned by more than one term. Therefore, sum of the studies may exceed 100%. For example, the total number of included studies in S3 is 67, however the sum returned by all terms is 77 (T1 = 41 + T2 = 26 + T4 = 2 + T5 = 1 + T8 = 2 + T9 = 1 + T6 = 4). Similarly, studies returned by combining T1, T2 and T3 added 350 studies (174 + 103 + 73).

Fig. 2: Term x Year of publication

Figure 2 illustrates the arrangement of the terms used to search for secondary studies in relation to the year of publication of these studies. Secondary studies in SE started to appear after the publication of the first procedures to perform SLR [34] published in 2004. This fact justify the year of the included studies start in 2005. The terms “systematic review” and “ literature review” most appeared in 2009 while the term “systematic literature review” in 2011. In addition, the biggest concentration of studies are found by these three terms distributed between 2008 and 2011. The abbreviation “SLR” started be used in the selected studies after 2008. In 2008 Petersen et al. published the first study addressing specifically systematic mapping in SE [41], as a consequence we believe that this is the reason for the appearance of the term in our set of included studies only in 2009.

Based on our results until now, a partial search string to find secondary studies should be formed by the terms: “systematic review” OR “literature review” OR “systematic mapping” OR “mapping study” OR “systematic map”.

4 RQ2: What is the preferable field(s) to search secondary studies?

Several DLs allow the execution of search strings in titles, abstract and keywords, however, actually there is no consensus in what would be the preferable field(s) for searching relevant secondary studies. Table VI describes where (title, abstract and keywords) each string term was found in the list of included secondary studies. In order to obtain a global overview from all secondary studies included in the list of all tertiary studies considered, we searched for duplicated studies among the lists of secondary studies as well as we analyzed if one respective term was already analyzed before in duplicated papers. For example, studies S2 and S4 had, in their included secondary studies list, a same secondary study and both had the term “systematic review” in their search string. Therefore, we analyzed and counted the presence of the term just once. This approach was adopted to avoid bias in our overall analysis.

String Terms Title Abstract Keywords
literature review 37 67 44
systematic review 82 95 57
systematic literature review 31 42 43
evidence based 4 16 15
SLR 17 1 2
review of studies 1 3 0
literature survey 1 4 1
meta analysis or meta-analysis 1 4 2
literature analysis 0 1 1
analysis of research 2 1 0
structured review 1 0 1
systematic map 1 0 1
TOTAL 178 234 167
TABLE VI: Terms appearance on title, abstract and keywords

With respect to the 12 terms that returned secondary studies (Table VI), abstracts have a higher appearance of terms, totalizing 234 appearances following by titles with 178 and keywords with 167. One possible reason is that abstracts present a summary of the study containing more words than title and keywords, consequently the probability of terms appearance is bigger than in titles and abstracts. Nevertheless, the presence of some terms in title were interesting. The term “SLR” appeared more times in title and the term “systematic review” in abstracts (see Table VI – line 6 and 3). The term “systematic literature review” appeared more times in keywords (see Table VI – line 4).

We recommend that abstract must be considered since it returned the higher number of relevant included studies. However, our research showed that title and keywords also are capable to find a considerable number of studies. Therefore, the preferable fields are title, abstract and keywords altogether. Although it seems obvious that the search should be done in all three fields, it was important to check whether searching limited to the title or abstract field could reduce the number of results retrieved. An essential task in a secondary study is to retrieve all relevant studies. Not finding all relevant studies from a review may result in bias and, consequently, false or imprecise conclusions.

5 Search String Validation

The search string suggested by us (RQ1) was tested for retrieving secondary studies. The definition of a control group was essential for its calibration. When relevant publications of the control group were not found, new terms were added to the search string during the calibration activity. In order to define the control group, we performed a search on the DL Scopus to find more tertiary studies in SE. We used the search string: ((“tertiary study” OR “tertiary review” OR “tertiary systematic review” OR “systematic review of systematic review”) AND “software engineering”) applied on title, abstract and keywords. The search string was based on the reading of tertiary studies listed in Table I. We included a tertiary study only if the study be within the SE context and if the list of secondary studies included in the tertiary study is available. We excluded a study if it is just published as an abstract, not written in English, an older version of other study already considered, not in the scope of SE and the list of secondary studies included is unavailable.

We then constructed a control group of 986 secondary studies based on the list of included secondary studies of the 14 tertiary studies selected after the application of the inclusion and exclusion criteria mentioned above (see Table VII). In order to determine if the proposed search string was able to detect these 986 secondary studies, we further tested the presence of the terms defined in our string in the title of the studies. Only if the term was not found in the title, we checked its presence in the abstract and then in the keywords. Figure 3 illustrates the string validation process. The results from the validation process is presented in Tables VIII and IX.

ID Tertiary study – Title Year Number of secondary studies
Verner et al. Systematic literature reviews in global software development: A tertiary study 2012 24
Kitchenham and B. A systematic literature review of systematic literature review process research in SE 2013 68
Bano et al. Systematic literature reviews in requirements engineering: A tertiary study 2014 53
Zhou et al. Quality assessment of systematic reviews in software engineering: A tertiary study 2015 110
Goulão et al. Quality in model-driven engineering: A tertiary study 2016 22
Nurdiani et al. The impacts of agile and lean practices on project constraints: A tertiary study 2016 41
Napoleão et al. Practical similarities and differences between SLRs and SMs: A tertiary study 2017 170
Hoda et al. Systematic literature reviews in agile software development: A tertiary study 2017 28
Marimuthu and C. Systematic Studies in Software Product Lines: A Tertiary Study 2017 60
Khan et al. Systematic Literature Reviews of Software Process Improvement: A tertiary Study 2017 24
Budgen et al. Reporting systematic reviews: Some lessons from a tertiary 2017 37
Singh et al. How do Secondary Studies in Software Engineering report Automated Searches? A Preliminary Analysis 2018 171
Rios et al. A tertiary study on technical debt 2018 13
Ampatzoglou et al. Identifying, categorizing and mitigating threats to validity in SE secondary studies 2018 165
TABLE VII: Included tertiary studies for the validation process
Fig. 3: Validation process
ID “Systematic review” “Literature review” “Systematic mapping” “Mapping study” “Systematic map” Total
Verner et al. 7 (29.17%) 7 (29.17%) 1 (4.17%) 0 (0%) 1 (4.17%) 16/24 (66.67%)
Kitcheham and B. 23 (33.82%) 9 (13.24%) 2 (2.94%) 3 (2.94%) 0 (0%) 36/68 (52.94%)
Bano et al. 20 (37.74%) 16 (30.19%) 5 (9.43%) 0 (0%) 0 (0%) 41/53 (77.36%)
Zhou et al. 44 (40%) 37 (33.64%) 11 (10.0%) 1 (0.91%) 0 (0%) 93/110 (84.55%)
Goulão et al. 8 (36.36%) 5 (22.73%) 1 (4.55%) 0 (0%) 0 (0%) 14/22 (63.64%)
Nurdiani et al. 12 (29.27%) 11 (26.83%) 2 (4.88%) 1 (2.44%) 0 (0%) 26/41 (63.41%)
Napoleão et al. 43 (25.29%) 45 (26.47%) 36 (21.18%) 9 (5.29%) 3 (1.76%) 136/170 (78.82%)
Hoda et al. 8 (28.57%) 11 (39.28%) 3 (10.71%) 0 (0%) 0 (0%) 22/28 (80.0%)
Marimuthu and C. 14 (23.33%) 17 (28.33%) 19 (31.67%) 5 (8.33%) 0 (0%) 55/60 (91.67%)
Khan et al. 8 (33.33%) 8 (33.33%) 1 (4.17%) 1 (4.17%) 0 (0%) 18/24 (75.0%)
Budgen et al. 12 (32.43%) 17 (45.95%) 4 (10.81%) 0 (0%) 1 (2.70%) 34/37 (91.89%)
Singh et al. 19 (11.11%) 61 (35.67%) 43 (25.15%) 2 (1.17%) 0 (0%) 125/171 (73.10%)
Rios et al. 0 (0%) 3 (23.08%) 4 (30.77%) 0 (0%) 0 (0%) 7/13 (53.85%)
Ampatzoglou et al. 35 (21.34%) 60 (36.59%) 44 (26.83%) 11 (6.71%) 1 (0.61%) 150/165 (92.07%)
TABLE VIII: Presence of string’ terms in title

Considering the presence of string’ terms studies title (see Table VIII), of the 986 secondary studies of the control group, our string was able to detect 774 studies (78.50%).

ID Studies retrieved by title Studies retrieved by title + abstract Studies retrieved by title + abstract + keywords Not retrieved
Verner et al. 16/24 (66.67%) 19/24 (79.17%) 19/24 (79.17%) 5 (20.83%)
Kitcheham and B. 36.68 (52.94%) 57/68 (83.82%) 61/68 (89.71%) 7 (10.29%)
Bano et al. 41/53 (77.36%) 52/53 (98.11%) 52/53 (98.11%) 1 (1.89%)
Zhou et al. 93/110 (84.55%) 106/110 (96.36%) 107/110 (97.27%) 3 (2.73%)
Goulão et al. 14/22 (63.64%) 17/22 (77.27%) 19/22 (86.36%) 3 (13.64%)
Nurdiani et al. 26/41 (63.41%) 37/41 (90.24%) 38/41 (92.68%) 3 (7.32%)
Napoleão et al. 136/170 (80.0%) 166/170 (97.64%) 170/170 (100%) 0 (0%)
Hoda et al. 22/28 (78.57%) 27/28 (96.42%) 27/28 (96.43%) 1 (3.57%)
Marimuthu and C. 55/60 (91.67%) 59/60 (98.33%) 59/60 (98.33%) 1 (1.67%)
Khan et al. 18/24 (75.0%) 22/24 (91.66%) 23/24 (95.84%) 1 (4.17%)
Budgen et al. 34/37 (91.89%) 36/37 (97.30%) 36/37 (97.30%) 1 (2.7%)
Singh et al. 125/171 (73.10%) 166/171 (97.08%) 167/171 (97.66%) 4 (2.34%)
Rios et al. 7/13 (53.85%) 11/13 (84.62%) 13/13 (100%) 0 (0%)
Ampatzoglou et al. 151/165 (92.07%) 157/165 (95.73%) 157/165 (95.73%) 8 (4.27%)
TOTAL 774/986 (78.50%) 932 (774+158)/986 (94.52%) 948 (932+16)/986 (96.15%) 38/986 (3.85%)
TABLE IX: Presence of string’ terms in title, abstract and keywords

From the remaining 212 studies, we verified the presence of the string terms in the abstract. More 158 (16.02%) studies were detected. 14 studies were retrieved by keywords, totalizing 948 of 986 studies (774+158+16 – 96.15%). 38 (3.85%) studies were not retrieved by our string.

The string was not able to detect studies categorized as meta-analysis (6 studies), survey (17 studies) and literature analysis (2 studies).

Pickard et al. [44] affirm that meta-analysis is appropriate for homogeneous studies when raw or quantitative data are available. Until a few years ago, SE simply lacked a sufficient body of replicated empirical studies to support meta-analysis application. In 2007, Dybå et al. affirmed: “In SE, primary studies are often too heterogeneous to permit a statistical summary.” In 2011, Cruzes et al. [9]

conducted a tertiary study to assess the types and methods of research synthesis in secondary studies in SE. They analyzed 49 reviews and concluded that: (i) half of the studies did not contain any synthesis; (ii) two thirds performed a narrative or a thematic synthesis; and (iii) only a few studies adequately adopted a research synthesis. Only two reviews, from the same research group, were classified as meta-analysis. We believe that several initiatives have changed this scenario, for example: (i) premier journals, such as, the journal Empirical Software Engineering strives to provide its audience with detailed data from the studies it publishes; (ii) there has been a growing awareness in the SE community of the importance of replicating studies

[28, 21]. Some authors [21] have suggested a classification which is intended to provide experimenters with guidance on what types of replication they can perform.

Literature analysis is “the practice of looking closely at small parts to see how they affect the whole” [12]. Survey is an empirical method that allows researchers to collect data from a large population aiming to generalize the findings [43]. Fowler [18] states that “statistical evidences can be obtained in a survey”. Dawson [12] adds that “surveys draw either qualitative or quantitative data from population”. One reason for using the term survey in reviews is the comprehensive definition of the term. Among other several definitions, Cambridge dictionary defines survey as: to look at or examine all of something, especially carefully. Therefore, it is possible to adopt survey and literature analysis as synonymous to SLR if: (i) population/small parts = available evidence (primary studies) systematically searched; (ii) findings/whole = summary of quantitative/qualitative data from different studies (statistical techniques can be used).

Search string Studies retrieved
(“systematic review” OR “literature review” OR “systematic mapping” OR “mapping study” OR “systematic map”) 948/986 (96.15%))
(“systematic review” OR “literature review” OR “systematic mapping” OR “mapping study” OR “systematic map” OR “meta-analysis”) 954/986 (97.72%)
(“systematic review” OR “literature review” OR “systematic mapping” OR “mapping study” OR “systematic map” OR “meta-analysis” OR “survey”) 971/986 (98.48%)
(“systematic review” OR “literature review” OR “systematic mapping” OR “mapping study” OR “systematic map” OR “meta-analysis” OR “survey” OR “literature analysis”) 973/986 (98.68%)
TABLE X: Improvements of search string

For the reasons given above, we decided to add the terms meta-analysis, survey and literature analysis to our string. Therefore, after the calibration (see Table X), the final string suggested by us is shown as follows:

(‘‘systematic review’’ OR ‘‘literature review’’ OR ‘‘systematic mapping’’ OR ‘‘mapping study’’ OR ‘‘systematic map’’ OR ‘‘meta-analysis’’ OR ‘‘survey’’ OR ‘‘literature analysis’’)

In summary, our search string was able to locate 973 of the 986 secondary studies included in the tertiary studies (98.68%). Three new terms were added during the calibration and a new search string was defined.

Some specific results from the validation process execution can be highlighted. Before the string calibration two secondary studies set from two tertiary study [39, 46] were completely retrieved by the proposed string. However, after string calibration (final string) the number of sets of secondary studies 100% retrieved increased to seven. This behavior was observed in the studies of Bano et al. [3], Goulão et al. [22], Nurdiani et al. [40], Hoda et al. [26], Marimuthu and C. [35] and Singh et al. [49].

The strings format must be verified for each DL. Such verification is necessary because the web search interfaces provided by these libraries continually change their rules, as also stated in [16, 45, 54, 30, 2].

Table XI illustrates different adaptations required in our search string to Web of Science, IEEE Xplore and ACM DL considering search in the title, abstracts and keywords of the studies.

DL Adapted Search String
Web of Science TS=(“systematic review” OR “literature review” OR “systematic mapping” OR “mapping study” OR “systematic map” OR “meta-analysis” OR “survey” OR “literature analysis”)
IEEE Xplore ((((((((“systematic review”) OR “literature review”) OR “systematic mapping”) OR “mapping study”) OR “systematic map”) OR “meta-analysis”) OR “survey”) OR “literature analysis”)
ACM DL “query”: (“systematic review” “literature review” “systematic mapping” “mapping study” “systematic map” “meta-analysis” “survey” “literature analysis”)
TABLE XI: Adaptations of Search Strings for DLs

Even after the string improvements, 13 studies were not found. In all of them, the authors mentioned the type of study only in Introduction or Methodology section. This fact reinforces the recommendation of complementary searches, added to the automatic searches as well as a reminder to researchers to explicit the research method in the title, abstract or keywords.

6 Discussions

In this section, we discuss issues related to the results obtained in our analysis and the limitations of our findings.

Elaborating and refining search strings is an important step in increasing the quality of the review process. Kitchenham and Charters [32] and Petersen and colleagues [42] recognize that the review process is iterative and typically requires revisions. For example, the construction of the search string is not a linear process; indeed, it implies a continuous refinement process [58]. New candidate keywords typically emerge during the calibration activity, and it is essential to refine the search string considering new terms and their synonyms.

Refinements are essential to ensure good search strategies. During the calibration of our search string, three new terms were identified. In summary, our recommendation for secondary studies searching is described in Figure 4.

Fig. 4: Suitable search string and preferable fields for searching secondary studies

As illustrated in Figure 4, a search string for searching secondary studies should be formed by: (1) terms and their synonymous related to research domain connected by OR logical operator; (2) logical operator AND; (3) terms and their synonymous related to secondary studies connected by OR logical operator.

The synonyms suggested by our study to find secondary studies are: “systematic review” OR “literature review” OR “systematic mapping” OR “mapping study” OR “systematic map” OR “meta-analysis” OR “survey” OR “literature analysis”. After the search string elaboration, it must be applied in title, abstract and keywords.

It is clear that the string to search for secondary studies proposed by us contains the most obvious terms. However, this work demonstrates that this terms are enough to search for secondary studies. In Kitchenham and colleagues book [31], it is highlighted that researchers in computing, more specifically in SE are prone to introduce new terms to describe their papers. However, they recommend avoiding this approach since the terminology used in SLRs is already adequate to describe the conduction of this type of study. Moreover, they reinforce that using standard terminology makes it easier to find the paper.

It is worth mentioning that each DL has a different input format, so it is necessary to adapt the string according to the rules required by DLs. Table XI presents examples of adapted strings for some of the most known DLs: Web of Science, IEEE Xplore and ACM DL.

Generally, researchers give little thought to define title and abstract of their studies. Keywords get even lesser attention, often being typed directly in the conference/journal’s submission system. However, these three elements – title, abstract, and keywords – are essential to ensure the publication success. Without them, most papers may never be found by interested readers. A sloppy attitude towards these three elements makes the search inefficient, even with “golden” search strings. The title and abstract are often the only elements of a study freely available on-line.

Based on our results, we suggest that the preferable field for searching secondary studies is to search in title, abstract and keywords together. Writing good titles, abstracts and keywords can insure that your study will be found by readers looking for it. Some recommendations are:

– Title

Good titles generally use terms that highlight the aim of the paper [47]. In Medicine, Cochrane Handbook for SLRs of Interventions [25], recommends that the title succinctly states the methodology and the problem for which the methodology is being assessed.

Our research showed that the terms most used to describe secondary studies are “systematic review”, “literature review” and “systematic literature review”. The “literature review” term is part of “systematic literature review” term, i.e., studies returned by “systematic literature review” are returned by “literature review” term as well. Therefore, we can say that the use of the terms “systematic review” or “literature review” are capable to return majority of relevant studies.

One point to be highlighted is that most of search strings analyzed did not present terms related to SM. From seven tertiary studies analyzed, only two (S7 and S8) had SM synonyms described in their strings. In addition, three of the 24 terms listed (see Table III – lines 20, 21 and 25) are related to SM. Moreover, only the term T24 returned one included secondary studies (Study S8 – see Table V – line 14); We suggest that SE secondary studies’ titles be formed by two parts – research topic: research methodology. Some examples are: “Concurrent Software Testing: A Systematic Review” [6]; “Knowledge Management Practices in GSD: A Systematic Literature Review” [1].

– Abstract

Abstracts must present a synthesis of research carried out. It is the part of the text that the researcher must pay close attention to describing the most relevant points of his research. Abstracts when compared to titles and keywords have the highest amount of words written, however if the abstract are not properly written, the study detection by search engines becomes more difficult. Recent studies have provided evidence that conventional and poorly written abstracts can hinder the study detection [17, 24].

Our research showed that abstracts were the field that most returned secondary studies (See Table VI - line 14, column 3). Therefore, researchers must be careful explaining the research method in the abstract. In addition, it is important verify if the terms that are being used for describing the method or methodology addressed are straightforward. For example, writing ”We performed a literature search following the guidelines proposed by Kitchenham and Charters [32], rather than ”We performed a systematic literature review”.

– Keywords

Keywords should be also found in the title, and in the abstract several times [47].

SE researchers could also consult secondary studies ontologies to identify keywords, e.g., de Almeida et al. [13] developed an ontology to describe knowledge regarding experimental studies, including SLRs.

7 Threats to the validity

The main limitation of this research is that we had no access to the list of secondary studies returned by automated search performed by authors. For this reason, we made use of the lists of the included secondary studies that were available. We assumed that the other studies were excluded because they were not relevant. Thus, as a first-cut assessment, we believe our study met its goal. Other threats to validity are described below.

Internal validity. Concerning internal validity, it is important to observe the limitation existing in relation to the universe of studies analyzed. The terms of our string were initially extracted from seven tertiary studies in SE mentioned by Garousi and Mäntylä. [19]. However, we constructed a control group of 986 secondary studies based on other eight tertiary studies. Three new terms were identified during this validation activity and 98.68% of the secondary studies were retrieved. Thus, we believe that our string is suitable for retrieving secondary studies.

Construct validity. The focus of our study was to analyze terms of the investigated research method (secondary studies). However, we suggest three different set of domain terms for three different areas. Others domain specific areas are not analyzed. For example, what are the synonyms for SE? For the elaboration of a complete search string, researchers must indicate these synonyms. Different synonyms could result in different studies, even adopting the same terms indicated by us for secondary studies.

Reliability. The terms of our search string are quite broad aiming to retrieve the maximum number relevant studies. However, studies that adopted different terminology will not be found. In order to mitigate this threat, complementary search strategies should be executed.

External validity. We suggest a generic search string, which is not generalizable for all DLs and their particularities, e.g., regarding plural, specific symbols (*, etc), among other details.

8 Related works

Dieste and colleagues [14] conducted a study to identify optimal search strategies to SLR. They evaluated recall and precision of search strategies in order to find an optimum strategy presenting recommendations on search strategies approaching research terms adoption and they found that title and abstracts are better than full text to search for studies. However, they concluded that it is impossible for a search strategy to return 100% of the relevant studies, due to lack of standardization in ES terminology. Also, the work was conducted considering the term “experiment” and its synonymous unlike the approach chosen for this work that analyzes a range terms related to secondary studies.

Singh and Singh [48] pointed inconsistencies in automated search resources in some of the DLs most commonly used by researchers to conduct secondary studies. Therefore, they have developed a caution guide that should be considered by researchers during the search for studies. Among the caution points raised are: the definition and customization of search strings for each DL, the complete learning of the behavioral aspects expected and observed in the resources of an individual DL, the accomplishment of a specific validation study for the specifics search string defined for each DL and to report in the section of threats to validity the limitations faced during the search process. However, the authors does not propose a search string ready-to-use and the preferable fields to search for secondary studies in DLs.

Tools to support automatic search for secondary studies are desired by SE researchers [37]. Ghafari et al. [20] developed a unified search tool that integrates an automatic search engine with the best-known SE DLs. However, for the use of the tool it is necessary to define the search terms, logical operators and fields which the search should be applied (for example: title, abstract and keywords). Souza et al. [52]

also propose an approach to automate search strings for secondary studies called Search-based String Generation (SBSG) through the application of the algorithm Hill Climbing, an Artificial Intelligence technique. To use the SBSG, researchers still need to define a set of parameters: terms, keywords, synonyms, number of iterations (how many times SBSG will run), and a list of control studies.

Singh et al. [49] conducted a preliminary tertiary study addressing 50 recently published secondary studies (2016-2017). As a result, this study highlights that most of the secondary studies adopt automatic search using at least four different DLs and complement the search with a manual search. It is also highlighted that the authors who perform secondary studies in SE do not adequately describe the search process, especially regarding to the search string used and the limitations and implications of the search process. In general, the authors highlighted that secondary studies are difficult to reproduce and that in the future they intend to investigate search strings to analyze the possibility of constructing reusable search strings, subject presented in this paper.

9 Conclusions

This paper aimed to assist investigations in SE area to create a search string to find secondary studies. In light of our findings, the main contributions of this research are as follows:

We constructed a suitable search string that can efficiently retrieve secondary studies wherein eight search terms were able to retrieve 98.68% of the secondary studies considered in your analysis;

The proposed suitable search string may help SE researchers detect secondary studies. We believe that this suitable search string for retrieve secondary studies may provide a time-saving useful tool for SE researchers who need to analyze secondary studies.

We presented and demonstrated that the preferred fields to search for secondary studies are title, abstract and keywords. Moreover, we provided recommendations to researchers conducting their SLR work regarding to the choose of terms for title, abstract and keywords

As a future work, we intend to develop a tool to (semi) automatically adapt the search strings in Digital Libraries for SE studies. In addition, we intend to further investigate search strategies as a whole, including selection of DLs and the adoption of other search methods (such as manual and snowballing) combined with automated search.

Acknowledgment

The authors thank the financial support received from CNPq (project 401033/2016-3), Brazil.

References

  • [1] S. Arshad, M. Usman, and Ikram (2012) Knowledge management practices in gsd: a systematic literature review. In 7 Int. Conference on Software Engineering Advances (ICSEA’12), pp. 516–523. Cited by: §6.
  • [2] M. A. Babar and H. Zhang (2009) Systematic literature reviews in software engineering: preliminary results from interviews with researchers. In 3 Int. Symposium on Empirical Software Engineering and Measurement (ESEM’09), pp. 346–355. External Links: ISBN 978-1-4244-4842-5 Cited by: §5.
  • [3] M. Bano, D. Zowghi, and N. Ikram (2014) Systematic reviews in requirements engineering: a tertiary study. In 4 Int. Workshop on Empirical Requirements Engineering (EmpiRE’ 14), pp. 9–16. Cited by: §5.
  • [4] J. Biolchini, P.G. Mian, A.C. Natali, and G.H. Travassos (2005) Systematic review in software engineering: relevance and utility. Technical report PESC/COPPE, Federal University of Rio de Janeiro, Brazil. Cited by: §1.
  • [5] P. Brereton, B.A. Kitchenham, D. Budgen, M. Turner, and M. Khalil (2007) Lessons from applying the systematic literature review process within the software engineering domain. Journal of Systems and Software 80 (4), pp. 571–583. Cited by: §1, §1.
  • [6] M. Brito, K.R. Felizardo, P. Souza, and S. Souza (2010) Concurrent software testing: a systematic review. In 22 IFIP Int. Conference on Testing Software and Systems (ICTSS’10), pp. 79–84. Cited by: §6.
  • [7] J.C. Carver, E. Hassler, E. Hernandes, and N.A. Kraft (2013) Identifying barriers to the systematic literature review process. In 7 Int. Symposium on Empirical Software Engineering and Measurement (ESEM’13), pp. 203–212. Cited by: §1.
  • [8] M. Chen, A. Babar, and H. Zhang (2010) Towards an evidence-based understanding of electronic data sources. In 14 International Conference on Evaluation and Assessment in Software Engineering (EASE), pp. 135–138. Cited by: §1.
  • [9] D.S. Cruzes and T. Dybå (2011) Research synthesis in software engineering: a tertiary study. Information and Sofyware Technology 53 (5), pp. 440–455. Cited by: TABLE I, TABLE II, §2, §5.
  • [10] F.Q.B. da Silva, A.L.M. Santos, S. Soares, A.C.C. França, and C.V.F. Monteiro (2010) Six years of systematic literature reviews in software engineering: an extended tertiary study. In 32 Int. Conference on Software (ICSE’10), pp. 1–10. Cited by: TABLE I, TABLE II, §2.
  • [11] F.Q.B. da Silva, A.L.M. Santos, S.C.B. Soares, A.C.C. França, and C.V. F. Monteiro (2010) A critical appraisal of systematic reviews in software engineering from the perspective of the research questions asked in the reviews. In 3 International Symposium on Empirical Software Engineering and Measurement (ESEM), pp. 33:1–33:4. Cited by: TABLE I, TABLE II, §2.
  • [12] C.W. Dawson (2005) Projects in computing and information systems: a student’s guide. Pearson Education. Cited by: §5.
  • [13] B. de Almeida, C. Jorge, P.G. Mian, A.C.C. Natali, T.U. Conte, and G.H. Travassos (2007) Scientific research ontology to support systematic review in software engineering. Advanced Engineering Informatics 21 (2), pp. 133–151. Cited by: §6.
  • [14] O. Dieste, A. Griman, and N. Juristo (2009) Developing search strategies for detecting relevant experiments. Empirical Software Engineering 14 (5), pp. 513–539. Cited by: §8.
  • [15] O. Dieste and A.G. Padua (2007) Developing search strategies for detecting relevant experiments for systematic reviews. In 1 Int. Symposium on Empirical Software Engineering and Measurement (ESEM’07), pp. 215–224. Cited by: §1.
  • [16] T. Dybå, T. Dingsøyr, and G.K. Hanssen (2007) Applying systematic reviews to diverse study types: an experience report. In 1 Int. Symposium on Empirical Software Engineering and Measurement (ESEM’07), pp. 225–234. Cited by: §1, §5.
  • [17] S.C.P.F. Fabbri, K.R. Felizardo, F.C. Ferrari, E.C.M. Hernandes, F.R. Octaviano, E.Y. Nakagawa, and J.C. Maldonado (2013) Externalising tacit knowledge of the systematic review process. IET Software 7 (6), pp. 298–307. Cited by: §1, §6.
  • [18] F.J. Fowler Jr. (2013) Survey research methods. Vol. 4, Sage publications. Cited by: §5.
  • [19] V. Garousi and M.V. Mäntylä (2016) A systematic literature review of literature reviews in software testing. Information and Software Technology 80 (C), pp. 195–216. Cited by: TABLE I, §2, §2, §7.
  • [20] M. Ghafari (2012-03) A federated search approach to facilitate systematic literature review in software engineering. 3, pp. 13–24. Cited by: §1, §8.
  • [21] O.S. Gómez, N. Juristo, and S. Vegas (2014) Understanding replication of experiments in software engineering: a classification. Information and Software Technology 56 (8), pp. 1033–1048. Cited by: §5.
  • [22] M. Goulão, V. Amaral, and M. M. (2016-09) Quality in model-driven engineering: a tertiary study. In Software Quality Journal, Springer (Ed.), Vol. 24, pp. 601–633. Cited by: §5.
  • [23] G.K. Hanssen, D. Smite, and N.B. Moe (2011-08) Signs of agile trends in global software engineering research: a tertiary study. In 6 International Conference on Global Software Engineering Workshop (ICGSEW), pp. 17–23. External Links: Document, ISSN 2329-6305 Cited by: TABLE I, TABLE II, §2.
  • [24] E. Hassler, J.C. Carver, N.A. Kraft, and D. Hale (2014) Outcomes of a community workshop to identify and rank barriers to the systematic literature review process. In 18 Int. Conference on Evaluation and Assessment in Software Engineering (EASE’14), pp. 1–10. Cited by: §6.
  • [25] J.P.T. Higgins and S. Green (2011) Cochrane handbook for systematic reviews of interventions. Version 5.1.0 [updated March 2011] The Cochrane Collaboration. Cited by: §6.
  • [26] R. Hoda, N. Salleh, J. Grundy, and H. M. Tee (2017) Systematic literature reviews in agile software development: a tertiary study. Information and Software Technology 85, pp. 60 – 70. External Links: Document, ISSN 0950-5849, Link Cited by: §5.
  • [27] S. Imtiaz, M. Bano, N. Ikram, and M. Niazi (2013) A tertiary study: experiences of conducting systematic literature reviews in software engineering. In 17 Int. Int. Conference on Evaluation and Assessment in Software Engineering (EASE’13), pp. 177–182. Cited by: TABLE I, TABLE II, §2.
  • [28] N. Juristo and O.S. Gómez (2012) Empirical software engineering and verification. B. Meyer and M. Nordio (Eds.), pp. 60–88. Cited by: §5.
  • [29] B.A. Kitchenham, P.O. Brereton, D. Budgen, M. Turner, J. Bailey, and S. Linkman (2009) Systematic literature reviews in software engineering – a systematic literature review. Information and Software Technology 51 (1), pp. 7–15. Cited by: TABLE I, TABLE II, §2, §2.
  • [30] B.A. Kitchenham and P.O. Brereton (2013) A systematic review of systematic review process research in software engineering. Information and Software Technology 55 (12), pp. 2049–2075. Cited by: §1, §1, §5.
  • [31] B.A. Kitchenham, D. Budgen, and P. Brereton (2015) Evidence-based software engineering and systematic reviews. Chapman & Hall/CRC Innovations in Software Engineering and Software Development Series, Chapman & Hall/CRC. Cited by: §6.
  • [32] B.A. Kitchenham and S. Charters (2007) Guidelines for performing systematic literature reviews in software engineering. Technical report Technical Report EBSE 2007-001, Keele University and Durham University, UK. Cited by: §1, §1, §6, §6.
  • [33] B.A. Kitchenham, R. Pretorius, D. Budgen, P.O. Brereton, M. Turner, M. Niazi, and S. Linkman (2010) Systematic literature reviews in software engineering – a tertiary study. Information and Software Technology 52 (8), pp. 792–805. Cited by: TABLE I, TABLE II, §2, §2.
  • [34] B.A. Kitchenham (2004) Procedures for performing systematic reviews. Technical report Technical Report TR/SE-0401, Keele University. Cited by: §3.
  • [35] C. Marimuthu and K. Chandrasekaran (2017) Systematic studies in software product lines: a tertiary study. In 21 Int. Systems and Software Product Line Conference – Volume A (SPLC’ 17), pp. 143–152. Cited by: §5.
  • [36] A.B. Marques, R. Rodrigues, and T. Conte (2012-08) Systematic literature reviews in distributed software development: a tertiary study. In 17 International Conference on Global Software Engineering (ICGSE), pp. 134–143. External Links: Document, ISSN 2329-6305 Cited by: TABLE I, TABLE II, §2.
  • [37] C. Marshall, P. Brereton, and B. Kitchenham (2015) Tools to support systematic reviews in software engineering: a cross-domain survey using semi-structured interviews. In 19 International Conference on Evaluation and Assessment in Software Engineering (EASE), New York, NY, USA, pp. 26:1–26:6. Cited by: §8.
  • [38] E. Mourao, M. Kalinowski, L. Murta, E. Mendes, and C. Wohlin (2017) Investiganting the use of a hybrid search strategy for systematic review. In 11 International Symposium on Empirical Software Engineering and Measurement (ESEM), Cited by: §1.
  • [39] B.M. Napoleao, K.R. Felizardo, E.F. Souza, and N.L. Vijaykumar (2017) Practical similarities and differences between systematic literature reviews and systematic mappings: a tertiary study. In 29 International Conference on Software Engineering, pp. 6. Cited by: §5.
  • [40] I. Nurdiani, J. Börstler, and S. A. Fricker (2016) The impacts of agile and lean practices on project constraints: a tertiary study. Journal of Systems and Software 119, pp. 162 – 183. External Links: Document, ISSN 0164-1212, Link Cited by: §5.
  • [41] K. Petersen, R. Feldt, S. Mujtaba, and M. Mattsson (2008-06) Systematic mapping studies in software engineering. In 12 Int. Conference on Evaluation and Assessment in Software Engineering (EASE’12), pp. 1–10. Cited by: §3.
  • [42] K. Petersen, S. Vakkalanka, and L. Kuzniarz (2015) Guidelines for conducting systematic mapping studies in software engineering: an update. Information and Software Technology 64 (1), pp. 1–18. Cited by: §6.
  • [43] S. L. Pfleeger (1995) Experimental design and analysis in software engineering. Annals of Software Engineering 1 (1), pp. 219––253. Cited by: §5.
  • [44] L.M. Pickard, B.A. Kitchenham, and P.W. Jones (1998) Combining empirical results in software engineering. Information and Software Technology 40 (14), pp. 811–821. Cited by: §5.
  • [45] M. Riaz, M. Sulayman, N. Salleh, and E. Mendes (2010) Experiences conducting systematic reviews from novices’ perspective. In 14 Int. Conference on Evaluation and Assessment in Software Engineering (EASE’10), pp. 44–53. Cited by: §5.
  • [46] N. Rios, M. G. M. Neto, and R. O. Spínola (2018) A tertiary study on technical debt: types, management strategies, research trends, and base information for practitioners. Information and Software Technology 102 (October 2018), pp. 117–145. Cited by: §5.
  • [47] M. Shaw (2003) Writing good software engineering research papers: minitutorial. In International Conference on Software Engineering (ICSE), pp. 726–736. Cited by: §6, §6.
  • [48] P. Singh and K. Singh (2017) Exploring automatic search in digital libraries: a caution guide for systematic reviewers. In 21 International Conference on Evaluation and Assessment in Software Engineering (EASE), pp. 236–241. Cited by: §1, §3, §8.
  • [49] P. Singh, M. Galster, and K. Singh (2018-05) How do secondary studies in software engineering report automated searches? a preliminary analysis. In 22 International Conference on Evaluation and Assessment in Software Engineering (EASE), Cited by: §5, §8.
  • [50] D. I. K. Sjøberg, T. Dybå, and M. Jørgensen (2007) The future of empirical methods in software engineering research. In Future of Software Engineering (FOSE’ 07), Washington, DC, USA, pp. 358–378. External Links: ISBN 0-7695-2829-5 Cited by: §1.
  • [51] M. Skoglund and P. Runeson (2009) Reference-based search strategies in systematic reviews. In 9 Int. Conference on Evaluation and Assessment in Software Engineering (EASE’09), pp. 1–10. Cited by: §1.
  • [52] Cited by: §8.
  • [53] I. Spasic, S. Ananiadou, U. Mcnaught, and A. Kumar (2005) Text mining and ontologies in biomedicine: making sense of raw text. Brief Bioinform 6 (3), pp. 239–251. Cited by: §1.
  • [54] M. Staples and M. Niazi (2007) Experiences using systematic review guidelines. Journal of Systems and Software 80 (9), pp. 1425–1437. Cited by: §1, §5.
  • [55] J.M. Verner, O.P. Brereton, B.A. Kitchenham, M. Turner, and M. Niazi (2014) Risks and risk mitigation in global software development: a tertiary study. Information and Software Technology 56 (1), pp. 54–78. External Links: ISSN 0950-5849 Cited by: TABLE I, TABLE II, §2.
  • [56] C. Wohlin (2014) Writing for synthesis of evidence in empirical software engineering. In 8 International Symposium on Empirical Software Engineering and Measurement (ESEM), pp. 1–10. Cited by: §1.
  • [57] H. Zhang and M. Ali Babar (2010) On searching relevant studies in software engineering. In 14 International Conference on Evaluation and Assessment in Software Engineering (EASE), pp. 111–120. Cited by: §1.
  • [58] H. Zhang and M.A. Babar (2011) An empirical investigation of systematic reviews in software engineering. In 5 Int. Symposium on Empirical Software Engineering and Measurement (ESEM’11), pp. 1–10. Cited by: §6.
  • [59] H. Zhang, B.A. Muhammad, and T. Paolo (2011) Identifying relevant studies in software engineering. Information and Software Technology 53 (6), pp. 625–637. Cited by: §1, TABLE I, TABLE II, §2.