Assessing Agile Transformation Success Factors

by   Amadeu Silveira Campanelli, et al.
Universidade FUMEC

Research on success factors involved in the agile transformation process is not conclusive and there is still need for guidelines to help in the transformation process considering the organizational context (culture, values, needs, reality and goals). The usage of success factors as a tool to help agile adoption raises the following research question: What are the success factors for an organization and their teams in preparation for the agile transformation process? This research presents an assessment of the organizational environment including the company's goals and the perception of the team members to provide awareness of how the organization should prepare for the next steps in the agile transformation. The findings show that a company based in Chicago, USA, succeeded implementing customer involvement and self-organized teams but faces challenges with measurement models and training. The main contribution of the research is understand which success factors exist in their environment and how they can be used during agile adoption.



There are no comments yet.


page 1

page 2

page 3

page 4


Agile Islands in a Waterfall Environment: Requirements Engineering Challenges and Strategies in Automotive

[Context motivation] Driven by the need for faster time-to-market an...

How (Un)Happiness Impacts on Software Engineers in Agile Teams?

Information technology (IT) organizations are increasing the use of agil...

Using the agile adoption framework to assess agility and guide improvements

Agility implies a set of principles that need to be followed in order to...

Measuring and improving information systems agility through the balanced scorecard approach

Facing an environment increasingly complex, uncertain and changing, even...

A Mapping Study on Software Process Self-Assessment Methods

Assessing processes is one of the best ways for an organization to start...

A Study of an Agile Methodology with Scrum Approach to the Filipino Company-Sponsored I.T. Capstone Program

Purpose - The research aims to show the relevance of company client spon...

DevOps Adoption: Eight Emergent Perspectives

DevOps is an approach based on lean and agile principles in which busine...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Organizations are searching for ways to overcome software development problems and achieve its business goals. Agile methods are an option many organizations have chosen to try to reach success [14, 9, 1, 13, 18]. As the number of organizations adopting agile methods continue to grow over the years [18], the need to guide organizations through the adoption process is also growing [1, 16, 8]. Researchers noticed there is no unique agile adoption model and it is hard to adopt out of the box agile methods [1, 16, 8, 2, 6]. The absence of systematic guidelines and frameworks for the agile adoption process considering the organization’s context (culture, values, needs, reality and goals) directed organizations to unsuccessful adoption paths and to failure [1, 16, 4].

During agile transition initiatives, organizations go through important transformations that impact its culture, hierarchy, management, environment and people [6, 12]. Understanding the challenges or success factors for an agile transformation helps to prepare the people involved and increase chances of success.

Agile transformation projects that do not consider the challenges to be faced along the way do not bring positive results to the organization and it can also create rejection from the professionals involved in it. Understanding the impacts of the changes in the organizational environment is an important step in the transformation process [12].

Multiple articles in the literature list the success factors involved in an agile transformation process (ATP) [6, 3, 17, 7, 5] but there is still no direct guidelines of how to use these success factors in specific organizational contexts. Most of the articles found on success factor describe the success factors but do not specify ways to use those in the ATP. The usage of success factors as a tool to help in the agile transition initiatives raises the following research question: How to assess the current state of an organization and their teams in preparation for the agile transformation process?

We propose an assessment to provide awareness of the status of agile transformation success factors in the organization. Our approach starts with a consolidation of agile transformation success factors found in the literature and the mapping of success factors into phrases to be used in the assessment. The assessment was planned to evaluate both the organization leadership and team members points of view. Using these views of the organizational context, we applied the Rasch algorithm [10] to create a rank of difficulty of implementation of the success factors in the organization. We validated the assessment with a single case study in a software development company based in Chicago, USA.

The remainder of the paper is organized as follows. Section 2 introduces the success factor groups and proposes the agile transformation success factors assessment. The results of the assessment and the findings are discussed on Section 3. Section 4 outlines the conclusions and future work opportunities.

2 Method

The agile transformation success factors assessment is a tool to provide awareness of how leadership evaluates the organization usage of the success factors and to help define goals for the ATP. It also considers the feedback from team members and the organizational context. The first step to create the assessment was to consolidate the success factors found in the literature into groups. Then, the success factors were mapped into representative phrases to be used in the assessment.

The assessment was planned to evaluate the organization goals and the view of the people involved in the ATP of the current state of the organization. The organization’s view of the success factors existence provides a reference for the target state, defining the level the company would like to achieve for each success factor. The team member’s view of the success factors existence in the organization provides the current state of the organization.

Success factor are the basis of the assessment since the evaluation occurs at their level. During the assessment definition, we found a need for a higher level of evaluation to allow a consolidated management view of the success factors.

We identified multiple success factors in the literature and aggregated them according to their concepts into terms and then into a set of groups. There were multiple references to the same concepts in different articles and we used terms we considered more representative of the concepts to aggregate all the references found. The six groups were proposed organizing the terms according to the areas of the organizations affected by them: customer, management, organization, process, team and tools. We select the Rash algorithm to be used to define how hard was to implement the sucess factors in the organization. The decision was based on the work of Lahrmann et al. [10] on maturity models using the Rasch algorithm where it was used to define which Business Intelligence capabilities are harder to be implemented and finally generate a maturity model clustering the capabilities into levels. In our case, we do not generate levels but instead we produce a ranking of the difficulty of implementing success factors in the organization.

2.0.1 Case Study

In order to validate the agile transformation success factors assessment we conducted a single case study in a software development company based in Chicago, USA called Company A in this research. The case study methodology was selected to allow an exploratory and qualitative view of the agile transformation success factors [15].

The case study was designed as a single case study to validate the assessment following guidelines by Runeson and Höst [15]. The case to be evaluated was the development process for Company A and the theory used in the case study was gathered from agile transformation success factors literature. Data collection was direct composed of interviews with the project sponsor and the major roles of the development process.

Company A has 110 employees and it has been using agile methods and practices for about 9 years. Scrum is the agile method used as the base for the process. This organization works with a mature and consistent agile process and that is the reason we selected it to validate the assessment. We met with the agile adoption sponsor and director of the company to understand their ATP, how they currently work and also his impressions on the challenges they faced during agile transformation and their current challenges. He responded the assessment to define the target state from the organization point of view.

We also interviewed 14 members of the organization with different profiles and skill sets: developers, designers, managers, testers and project managers/Scrum masters. Their responses to the assessment were considered the current state of the agile adoption in the organization. 71.4% of the interviewees have been working with agile methods and practices for at least 6 years and 57.1% of them have been working for Company A for at least 3 years. The teams represented by the interviewees were development (50.0% of the interviewees), design (21.4% of the interviewees), project management (14.3% of the interviewees) and quality assurance (14.3% of the interviewees).

Among the interviewees, 11 (78.6%) considered the assessment very useful to provide the status of the success factors usage in the organization. Based in the expertise of the interviewees and the results collected in the interviews in Company A, we were able to validate the assessment.

3 Results

Rank Group Success Factor Logit Error Infit Outfit
1 Process Measurement model 2.42 0.33 0.96 0.90
2 Organization Training 1.68 0.33 1.22 1.24
3 Organization Agile champions 1.11 0.35 0.48 0.58
4 Organization New mindset/roles 0.98 0.36 0.52 0.56
5 Management Changes in management style and decentralized decision making 0.85 0.37 0.19 0.23
6 Team Distributed teams 0.85 0.37 1.78 1.04
7 Organization Knowledge sharing 0.55 0.40 1.62 1.21
8 Team Technical activities/skills 0.55 0.40 1.71 1.20
9 Organization Business goals 0.21 0.43 0.33 0.36
10 Process Lightweight documentation 0.21 0.43 1.59 1.66
11 Process Process is compatible with the organizational context 0.21 0.43 0.48 0.54
12 Team Ability to build trustworthy relationships 0.21 0.43 0.72 0.74
13 Team Team involvement 0.02 0.45 0.72 0.66
14 Organization Incentives/motivation to adopt agile methods -0.20 0.48 0.62 0.59
15 Organization Communication flow in the organization -0.20 0.48 0.74 0.75
16 Management Management buy-in -0.44 0.50 0.38 0.32
17 Organization Coaching/mentoring -0.44 0.50 1.73 1.88
18 Team Collaboration -0.70 0.53 0.79 0.91
19 Tools Tool set -0.70 0.53 1.76 1.87
20 Organization Cultural changes -1.30 0.57 0.10 0.09
21 Management Changes in mind set of project managers -1.63 0.57 1.08 1.05
22 Team Self-organized teams -1.95 0.57 1.38 1.41
23 Customer Customer involvement -2.28 0.56 0.42 0.38
Table 1: Rasch algorithm results with success factors ordered from harder to easier to implement in Company A’s context.

We executed the Rasch algorithm using the Winsteps software [11] based on the data collected in Company A

to obtain the item calibration. The item calibration resulting of the Rasch algorithm classified the success factors in the assessment according to their difficulty to be implemented in

Company A.

Lahrmann et al. [10] proposed modifications to the data preparation for the Rasch algorithm for researches on maturity model for Information Systems (IS). We reproduced the same modifications in our research but at the end of the process we do not cluster the success factors in maturity models since that is not the goal of the research.

The first modification was to use a 5-point Likert scale instead of the dichotomous scale (yes or no). The idea behind it is that in this case there is no right or wrong but instead, opinions to be expressed in a rating scale [10]. The second modification was the usage of a representation of the desired or target state. In our research, the organization’s goals for the success factor represents the desired or target state. The delta value representing the potential improvement is calculated based on the difference between current state and target state [10]. Negative values would indicate that the current state is higher than the target state and that the success factor is already implemented in the Company A according to the defined goal.

The third modification proposed by Lahrmann et al. was the coding of the delta value to encompass all negative values into a single category to simplify the expression of over-compliance [10]. The coding used in this research is based on the work of Lahrmann et al. [10].

In order to validate the results from the Rasch algorithm we used to the recommendations provided by Winsteps documentation [11] and the guidelines used by Lahrmann et al. [10]. The fit statistics values (Infit and Outfit) are around 1.00 and satisfy the fit expectations validating the results. The results are summarized in Table 1.

Based on the results, the success factors that would be harder for Company A to implement are: measurement model, training, agile champions, new mindset/roles, and changes in management style and decentralized decision making. Furthermore, the results point out the there is already an adoption of customer involvement, self-organized teams, changes in mindset of project managers and cultural changes success factors by Company A.

4 Conclusions

In this study, we consolidated success factors for agile transformation into groups and used them to propose the agile transformation success factors assessment. We validated the assessment in a case study in a company based in Chicago, USA. The agile transformation success factors assessment is a tool to help organizations to understand the current state of agile success factors in the organization based on their team members view and to prepare for the agile transformation process.

The assessment produces a success factors difficulty ranking according to the difficulty to implement the success factors in the organization considering its context (culture, reality, goals, hierarchy). For Company A, the ranking showed that measurement model, training, agile champions, new mindset/roles, and changes in management style and decentralized decision making are the harder success factors to be implemented. Meanwhile, customer involvement, self-organized teams, changes in mindset of project managers and cultural changes would be the easier to implement.


  • [1] Ayed, H., Habra, N., Vanderose, B.: Am-quick: a measurement-based framework for agile methods customisation. In: Software Measurement and the 2013 Eighth International Conference on Software Process and Product Measurement (IWSM-MENSURA), 2013 Joint Conference of the 23rd International Workshop on. pp. 71–80. IEEE (2013)
  • [2] Campanelli, A.S., Parreiras, F.S.: Agile methods tailoring – a systematic literature review. Journal of Systems and Software 110, 85 – 100 (2015)
  • [3] Chow, T., Cao, D.: A survey study of critical success factors in agile software projects. Journal of Systems and Software 81(6), 961–971 (2008)
  • [4] Gandomani, T.J., Zulzalil, H., Ghani, A.A., Sultan, A.B.M., Sharif, K.Y.: How human aspects impress agile software development transition and adoption. International Journal of Software Engineering and its Applications 8(1), 129–148 (2014)
  • [5] Gandomani, T.J., Zulzalil, H., Ghani, A.A.A., Sultan, A.B.M., Nafchi, M.Z.: Obstacles in moving to agile software development methods; at a glance. Journal of Computer Science 9(5), 620 (2013)
  • [6] Gregory, P., Barroca, L., Sharp, H., Deshpande, A., Taylor, K.: The challenges that challenge: Engaging with agile practitioners’ concerns. Information & Software Technology 77, 92–104 (2016)
  • [7] Gregory, P., Barroca, L., Taylor, K., Salah, D., Sharp, H.: Agile challenges in practice: a thematic analysis. In: Agile Processes, in Software Engineering, and Extreme Programming, pp. 64–80. Springer (2015)
  • [8] Javdani Gandomani, T., Ziaei Nafchi, M.: An empirically-developed framework for agile transition and adoption. Journal of Systems and Software 107(C), 204–219 (2015)
  • [9] Jyothi, V.E., Rao, K.N.: Effective implementation of agile practices ingenious and organized theoretical framework. IJACSA - International Journal of Advanced Computer Science and Applications 2(3), 41–48 (2011)
  • [10] Lahrmann, G., Marx, F., Mettler, T., Winter, R., Wortmann, F.: Inductive design of maturity models: Applying the rasch algorithm for design science research. In: Service-Oriented Perspectives in Design Science Research - 6th International Conference, DESRIST 2011, Milwaukee, WI, USA, May 5-6, 2011. Proceedings. pp. 176–191 (2011)
  • [11] Linacre, J.: Smartpls 3. (2016), accessed: 2016-07-20
  • [12] Nerur, S.P., Mahapatra, R., Mangalaraj, G.: Challenges of migrating to agile methodologies. Commun. ACM 48(5), 72–78 (2005)
  • [13] Nishijima, R.T., Dos Santos, J.G.: The challenge of implementing scrum agile methodology in a traditional development environment. International Journal of Computers & Technology 5(2), 98–108 (2013)
  • [14] Qumer, A., Henderson-Sellers, B.: Crystallization of agility back to basics. In: ICSOFT 2006, First International Conference on Software and Data Technologies, Setúbal, Portugal, September 11-14, 2006. pp. 121–126 (2006)
  • [15] Runeson, P., Höst, M.: Guidelines for conducting and reporting case study research in software engineering. Empirical software engineering 14(2), 131 (2009)
  • [16] Soundararajan, S., Balci, O., Arthur, J.D.: Assessing an organization’s capability to effectively implement its selected agile method(s): An objectives, principles, strategies approach. In: 2013 Agile Conference, AGILE 2013, Nashville, TN, USA, August 5-9, 2013. pp. 22–31 (2013)
  • [17] Taylor, P.S., Greer, D., Coleman, G., McDaid, K., Keenan, F.: Preparing small software companies for tailored agile method adoption: Minimally intrusive risk assessment. Software Process: Improvement and Practice 13(5), 421–437 (2008)
  • [18] VersionOne: 10th annual state of agile development survey. (2015), accessed: 2015-10-05