The influence of software-based systems on the in-car User Experience (UX) has changed from purely operating the car, through adding simple infotainment devices to the highly complex systems we experience today (Harvey.2016). Modern cars are equipped with various In-Vehicle Information Systems (IVISs) that offer a large variety of features, interaction possibilities, and can be controlled via multiple interfaces (Lamm.2019). The expectations towards those systems are not only driven by the customer’s experiences with other car manufacturers but also by modern smartphones, websites, and other digital products. Baker.2016 (Baker.2016) report that customers are willing to pay up to of a car’s list price, or as much as $10,000 for connected car and infotainment technology. This paradigm shift leads to a growing and diverse competition in the automotive domain and unlocks potential for advanced and intelligent in-vehicle features (Pfleging.2012; Schroeter.2012; Ebel.2020). As a consequence, the task of designing user interfaces that meet customer demands has become more challenging and represents a crucial part of automotive development (Harvey.2016).
User-centered Design (UCD) requires experienced designers and exhaustive user studies and, therefore, is a costly, yet critical, task. This applies in particular to the design and evaluation of IVISs. Compared to many digital products, the complex automotive architecture, the critical requirements regarding functional safety, and the traditional structures in the automotive industry are challenging (Broy.2006; Alvarez.2017). Additionally, the driver’s experience does not only depend on the system itself but also on the current driving and traffic situation (Schneegass.2013; Ahmad.2015). While driving a car, the interaction with the IVIS is only a secondary task, which makes the interactions highly context-sensitive (Harvey.2016). Therefore, it is necessary to create a realistic simulation of the driving situation when evaluating IVISs. Simulating the driving situation provides value to the designers but requires high investments. While more cost-effective alternatives (Gerber.2019) are developed, they are still in an early research stage. The analysis of field user interaction data can reduce the effort for extensive user studies and expensive simulations by gaining insights about user behavior through data analysis (Orlovska.2018).
Despite the generally growing awareness of the potentials of big data analysis, there is a lack of research on how data-driven approaches may support the automotive UX development process. In this paper, we present results from a qualitative study on the current role of field user interaction data in the automotive industry and highlight the differences to digital products. We conducted semi-structured interviews with 14 UX professionals, where 8 are currently employed in the automotive industry and 6 in other industries. In the interviews, we addressed the current state-of-the-art, the challenges, and the potentials of field user interaction data in the respective UX development lifecycles. Our key findings indicate that implicit feedback through field user interaction data is currently not evident in the automotive UX development process. However, practitioners confirm its potential to make the UX development lifecycle more evidence-based and user-centered. Additionally, participants are concerned about insurmountable organizational, legal, or technical restrictions that prevent data collection. Participants from digital companies share most of the needs and potentials with the automotive participants but are generally concerned with more advanced issues like data interpretability and triangulation between qualitative and quantitative feedback.
In this section, we discuss the concept of UX, its different perception in industry and academia, and its special role in the automotive industry. We present a definition of field user interaction data and give an overview of related studies.
2.1. UX and its Role in the Automotive Industry
A good UX is the main goal of most product development processes. However, UX is perceived differently in academia and industry. Academics commonly agree that UX is a holistic and subjective concept (Roto.2009), which goes beyond solving usability problems or creating a pleasant UI. Usability aspects contribute to the overall UX but do not suffice as stand-alone UX measures since they do not yield insights on how the interaction was perceived by the user (Roto.2011). The perceived UX is mainly influenced by the user’s internal state, the characteristics of the system at use, and the context in which the interaction occurs (Hassenzahl.2006).
This holistic concept of UX leads to challenges in practice (Gray.2014; Vaananen.2008) with some of them being particularly relevant in automotive UX development. On the one hand, UX is strongly affected by the context in which the interaction occurs (Harvey.2011; Orlovska.2020). The interaction with an IVIS is therefore dependent on environmental conditions, such as the driving scenario, the dual-task environment, and the frequency of use (Harvey.2010). Those aspects must be addressed by the designers, increasing the complexity of the design task (Fastrez.2008) by the need for a context-dependent presentation of information (Loecken.2017). On the other hand, practitioners and organizations are more interested in the long-term experience rather than in temporary emotions (Law.2009). Regarding the automotive industry with its long product lifecycles (Broy.2006), its different touch-points, and diverse and global user base (Heimgaertner.2017) the question of how to capture an ‘overall’ UX score (Law.2009) gets even more interesting. Additionally, Frison.2019b emphasize the importance of UX, as it influences trust in the vehicle (Frison.2019b) and gets even more important with the transition toward automated driving (Frison.2019).
2.2. Field User Interaction Data
In line with the definition of Harvey.2011 (Harvey.2011), we define user interaction data as every kind of interaction between the user and the IVIS. User interactions are neither bound to a specific type of interface nor a specific type of interaction and therefore vary in their granularity. User Interaction data can be generated both in the field and in the lab. In the following, we will mainly focus on user interaction data generated during field usage. We define this as field user interaction data, which is continuously and implicitly recorded automatically in all production line vehicles.
2.3. Related Work
The approach of using field user interaction data to evaluate and enhance a product’s UX is already well researched and widely established in digital domains like web and app development. In 2006, Atterer.2006 (Atterer.2006) presented an approach for detailed user tracking on web pages that benefits usability evaluation by incorporating implicit user interactions. While Atterer.2006 track multiple different user actions, Navalpakkam.2012 (Navalpakkam.2012) propose an approach that predicts the overall experience of web page users by mouse tracking alone. Nebeling.2013 (Nebeling.2013) developed a framework that combines automated usability testing with crowdsourcing. They argue that the benefits of large-scale online user testing outweigh the trade-offs compared to lab studies. Compared to digital domains, the research field on the usage of implicit interaction data to evaluate IVISs is not yet widely explored.
This aligns with the findings of Lamm.2019 (Lamm.2019), who found that model-based approaches and automated evaluations do not play an important role in automotive UX development. However, they do not reflect on the views of practitioners and what might prevent them from applying these approaches.
A method to measure UX in an automotive context based on the fulfillment of psychological needs is proposed by Korber.2013b (Korber.2013b; Korber.2013). They argue that the presented questionnaire is not only able to measure the UX and compare products quantitatively but also to predict possible experiences in early prototype stages.
Another approach is presented by Harvey.2011 (Harvey.2011). They propose a framework based on thirteen methods that measure the objective and subjective levels of performance and workload of users interacting with IVIS. Similarly, Alvarez.2017 (Alvarez.2017) present a set of tools and methodologies that aim to benefit the creation of a holistic automotive design space. The work of Riener.2017 (Riener.2017) gives an extensive overview on how the driver-interaction loop can be modeled. Although some of the presented methods are based on user interaction data, the use of data, implicitly , automatically, and continuously collected from the whole user base, is not considered.
This is investigated by Orlovska.2018 (Orlovska.2018), showing how field user interaction data can support the evaluation of IVISs. They argue that field data-driven approaches enable more accurate feedback and enable testing of the whole user base. They also found that software platforms of Original Equipment Manufacturers are not designed for user behavior logging and that methods need to be developed that enable compliance with data protection regulations. Additionally, they present a case study (Orlovska.2018b) on how field user interaction data can benefit the overall usability assessment. However, this study exclusively deals with a specific system of a single manufacturer. Therefore, the results are hard to generalize.
3. Study Design
Despite the claimed potentials of using field user interaction data to improve the UX development lifecycle and its success in other fields, there are indications that these potentials are not (yet) leveraged in the development of automotive IVISs. We are interested in why this is the case. We want to answer the following research questions:
What is the current role of field user interaction data in the automotive UX development lifecycle?
What are the needs, challenges, and concerns in the context of data-driven UX Development?
How can the automotive UX development lifecycle benefit from field-data-driven approaches?
What is specific to the automotive UX development lifecycle and what can be generalized from digital companies?
|#||Age||Education||Job Title||Industry||# Employees||Professional Experience|
|P1||20-29||Master||User Researcher||Automotive (RH)||201-500||1-4 years|
|P2||40-49||PhD||Technical Specialist||Automotive (OEM)||10,001-100,000||10-19 years|
|P3||20-29||Bachelor||UX/UI Designer||Automotive (RH)||51-201||1-4 years|
|P4||30-39||Master||UX/UI Designer||Automotive (RH)||201-500||1-4 years|
|P5||30-39||Bachelor||UX/UI Designer||Automotive (RH)||201-500||10-19 years|
|P6||30-39||Master||UX Marketing Specialist||Automotive (OEM)||10,001-100,000||1-4 years|
|P7||40-49||Bachelor||Interaction Designer||Automotive (RH)||501-1,000||10-19 years|
|P8||40-49||Diploma||Project Manager UX||Automotive (OEM)||10,001-100,000||1-4 years|
|P9||20-29||Master||Interaction Designer||Internet of Things||501-1,000||5-9 years|
|P10||30-39||PhD||UX Manager||E-Commerce||10,001-100,000||5-9 years|
|P12||40-49||Diploma||Head of UX||IT Service Provider||1,001-10,000||10-19 years|
|P13||40-49||Bachelor||UX/UI Designer||Apps and Software||51-200||1-4 years|
|P14||30-39||Master||Design Manager||Digital Music Service||1001-10,000||10-19 years|
3.1. Research Method and Interview Design
To answer the research questions we followed a qualitative approach and conducted semi-structured interviews. Before conducting the interviews, we asked the participants to answer a questionnaire regarding their demographics, background, and experience. Although we prepared a list of questions111Interview Guideline: https://figshare.com/s/d9af6f2fa18f59e7e7eb, we varied the order of questions to unfold the interview conversationally. This exploratory approach allows open-ended questions and engages the participants to independently address the objectives they consider important. The interview itself was divided into three parts addressing the three usability engineering lifecycle phases introduced by Nielsen.1992 (Nielsen.1992): pre-design, design, and post-design. Regarding each phase, we asked the participants about the methods they currently apply, the challenges they face, and the potentials they see in data-driven approaches. Each interview lasted approximately one hour and was conducted by the first two authors with always one interviewee present. Of the 14 interviews, 5 were carried out in person, one via video call and 8 via phone.
3.2. Study Subjects
In total, we interviewed 14 UX professionals from 11 different companies, 8 working in the automotive industry, and 6 working for digital companies. We define a digital company as a company whose main product is a digital product or which has a digital product in its core business. The domains of these digital companies range from digital music services through e-commerce to telecommunications. However, we carefully selected candidates that are solely responsible for a digital product within their company. To get this broad range of perspectives inside each of the groups we applied purposive sampling (Etikan.2016). We approached companies of different sizes and domains and selected candidates of various backgrounds. The interviews were conducted between October 2019 and March 2020. Since all participants are kept anonymous, they are referred to with IDs P1-P14. All participants are currently employed in industry, with only 3 never having worked in a research context. Table 1 shows an overview of the demographics of the participants. In the automotive industry, it is very common that OEMs have multiple smaller research facilities or Research Hubs, where specialists work on a specific topic, decoupled from the main company. The participants did not receive any compensation.
3.3. Data Analysis
The first author transcribed and anonymized the audio recordings of the interviews. Afterward, the first and second authors applied a mixture of a priori and emergent coding (Saldana.2015) in a collaborative manner using ATLAS.ti222https://atlasti.com/.
For initial coding, both authors agreed on a set of codes based on the research questions. However, the authors were free to introduce new codes whenever they considered it to be necessary. For the coding, no special restrictions applied and each interview transcript was coded independently by the first two authors. To ensure the reliability of coding, the inter-coder agreement, according to Krippendorff.2018 (Krippendorff.2018), was calculated before the results of each interview were discussed and merged. The inter-coder agreement over all interviews is representing a satisfactory result (Krippendorff.2018). Newly introduced codes were reviewed by both authors and after mutual agreement, were added to the set of codes. This procedure was repeated for each interview and already coded transcripts were updated collectively by the authors. The changes introduced to the set of codes decreased after 6 interviews and no new codes emerged after 11 interviews. Therefore, we conclude further interviews will provide only a few (if any) new insights and we reached a point of theoretical saturation (Guest.2006).
The quotes from non-English speaking interviewees were translated into English and edited for readability. Colloquial expressions were not changed to reflect the informal setting of the interview.
3.4. Threats to Validity
The five threats to validity in qualitative research identified by Maxwell.2012 (Maxwell.2012) also apply to our study design. These threats describe the flaws that can occur while obtaining and interpreting the study observations. Further, the collected data might be manipulated to fit a specific theory, may it be deliberately or accidentally. Maxwell.2012 (Maxwell.2012) argues that the researchers must preclude those threats by developing a study design that provides evidence that no “alternative hypotheses” can be made (Lewis.2009).
Descriptive validity refers to the threat of incomplete and inaccurate recordings. To preclude this threat all interviews were recorded and transcribed. The transcripts are annotated with timestamps such that the original conversation can be traced back during analysis.
The threat of Interpretation validity addresses the challenge to capture the observations as intended by the participants. To avoid this threat, we used open-ended and non-directional questions. Additionally, all interviews were independently coded by two authors, and potentially ambiguous statements were discussed to identify the interpretation intended by the participant.
Theory validity and researcher bias refer to the threat that the researchers force the data to fit a certain theory they want to support or that they possess a deliberate bias regarding the participants or a certain outcome. Mitigating this threat is the fact that the study is constructed to be very exploratory, having the intention to reflect the current state-of-the-art in the industry and identify potentials. Additionally, we lowered the researcher bias by applying the introduced coding and reviewing concepts.
Reactivity describes the threat that the presence of the interviewers may influence the interviewees. This threat can hardly be mitigated but still, the authors payed attention to not influence the interviewees when conducting the interviews.
An additional threat is posed by the selection of the interviewees. We only interviewed employees of automotive OEMs, which might introduce some bias by excluding suppliers. The OEM research hubs usually act as company-internal suppliers, being solely responsible for whole systems within the car, which might add some similar perspectives.
We structured the identified codes into categories and illustrate their relations in Figure 1.
|State-of-the-Art:||Statements of phenomena in current practice, which reveals a challenge or leads to a need|
|Challenges:||Statements of problems that arise from current practices|
|Needs:||Statements of demands towards improving the UX development lifecycle|
|Concerns:||Statements of doubts that a challenge can be overcome or a need can be fulfilled|
|Potentials:||Statements of areas where data-driven approaches may address a challenge or fulfill a need|
The model shows that the reported State-of-the-Art reveals Challenges and leads to Needs of practitioners. Some of these challenges and needs can be addressed by analyzing field user interaction data (Potentials). These potentials are expressed explicitly and implicitly by the participants. Concerns were mentioned as hindering factors. The model applies to the pre-design, design, and post-design phase.
Figure 2 shows the distribution of codes within each category as bars, and the number of interviews the code occurred in as numbers on top. On average we introduced 80 codes per interview.
In the pre-design phase, the most frequent code in the state-of-the-art category is Requirements. Six of the eight interviewees from the automotive domain agree that the requirement and feature elicitation is not user-oriented. P1 states: “[…] at this point [the pre-design phase] we have no clue if the customer [user] is interested in this feature or not”. Only interviewee P2 from the automotive domain confirmed that they already use some form of field user interaction data in the pre-design stage by aggregating data from company cars in all markets in real-time. This data is then used, for example, to derive statements about the usage frequency of certain features to “prioritize what [the company] actually should spend money on and what […] the most important features based on usage [are]” (P2).
Another quarter of citations in the pre-design phase are tagged as User Feedback, being the most mentioned state-of-the-art code over all phases (see Figure 2). All participants from the automotive domain but P5 mention they receive some form of user feedback in the pre-design phase. This feedback, however, is usually in the form of general market research and not really focused on the users’ explicit needs or behavior. In contrast, P2 describes their rather elaborate process of analyzing user needs: “We do that [long-term ethnographic research] by observing, interviewing, participating with people in their life, being in their homes, trying to figure out what life people are living, what are their pleasure points and pain points”. In contrast, within digital companies, the elicitation of features and requirements seems to be more focused on the user. All digital domain participants report that the ideas in early development phases are created together with representative customers, are based on insights drawn from field user interaction data, or both.
In the design phase, two-thirds of state-of-the-art citations are coded with Design Decisions, Study Design, or User Feedback. Regarding the automotive state-of-the-art, 6 out of 8 practitioners confirm that they evaluate their designs mainly in-house in an informal, qualitative way with coworkers and other UX experts. In contrast, P2 and P6 from the automotive domain confirm that they recruit external people on a regular basis for early feedback on their designs and ideas. Interviewee P7 describes the current state-of-the-art: “Testing within [the company] is sort of ok if you just need to do something quickly, but if we want to verify things, it’s better to get people that are not familiar with what we do”. While all participants implement clickable prototypes for their products, these are only evaluated qualitatively. The automotive participants agree that their current process of gathering qualitative feedback on prototypes is quite advanced. At the same time, none of the automotive interviewees have direct access to or actively aggregate field user interaction data. These circumstances also show automotive UX experts mainly rely on explicit feedback from users and do not leverage implicit feedback through field user interaction data. As mentioned before, with the tracking of user interaction data on company vehicles, interviewee P2 reports the most advanced data-driven processes of the automotive participants.
In the post-design phase, 7 out of 8 automotive participants confirmed they do not get consistent and detailed feedback based on field user interaction on their product. Five of these practitioners, however, confirm that they do get feedback through market analyses and general customer surveys.
Interviewee P4 describes the feedback process in the post-design stage as follows: “At the moment, we only receive feedback through studies [the company] executes, that take weeks and months. They take the [product], test it in multiple markets with many people, and curate a [report] with the results.”
“At the moment, we only receive feedback through studies [the company] executes, that take weeks and months. They take the [product], test it in multiple markets with many people, and curate a [report] with the results.”In the digital domain, 4 out of 6 practitioners have implemented a process to receive user feedback for their products based on field user interaction data.
State-of-the-art in automotive (RQ1):
Requirements and feature elicitation is not user-centered and only supported by general market research.
Focus is on explicit feedback; implicit feedback through field user interaction data is not evident.
Prototypes are mainly evaluated qualitatively by co-workers and in-house UX experts.
Differences to digital companies (RQ4):
In digital companies, field user interaction data is considered for decisions throughout all design phases.
For feature elicitation and prioritization, digital companies use a mixture of explicit feedback from representative users and implicit insights from field user interaction data.
Within digital companies, insights from field user interaction data are very broad and range from feature usages to sophisticated hypotheses testing.
Figure 2 shows that Management Opinions are often considered a challenge in the automotive domain. In particular, in the pre-design phase, six out of eight participants from the automotive domain report that their findings from user research are not considered in the decision making process. They argue that their proposals are often overruled by higher management even though they provide evidence through their research. Practitioners from digital companies do not experience this challenge as often. This correlates with the challenge to back up the designer’s hypotheses toward user interaction with the product (Backup of Hypothesis). This challenge was expressed only by automotive participants. P7 states: “There are a lot of assumptions that people make about who is driving our cars, but none of them is actually backed up with any kind of information”. Data Accessibility and Data Collection are also mentioned frequently in the pre-design phase. Data accessibility refers to a generally insufficient availability, i.e. accessibility of user-related data within the company. P8 mentions that “[t]here is a very strong silo mentality in companies in the acquisition of information, but also in the distribution of information”. The fact that all citations tagged with Data Accessibility come from automotive participants highlights the significant deficits regarding data transparency. Considering the data collection challenges, all participants mainly refer to the challenge of collecting data as detailed as possible without violating legal restrictions. However, there are further technical peculiarities that complicate extensive data collection from users in the automotive domain. P7 states that for “the older systems none of this existed, so we have no way of understanding what people did with it”. Additionally challenging is the need for long-lasting architectures and the heterogeneous data processed by multiple different Electronic Control Units. P2 exemplifies that the current architecture of their vehicle platform is not yet prepared for the kind of interaction logging needed today.
Time Restriction and Design Decisions are the most often mentioned challenges by all participants regarding the design phase. Six participants describe that they often lack time to dive deep into user studies or interaction data. P1 explains: “The first priority is speed. We can’t work on data for two or three days”. Considering design decisions, 5 out of 8 automotive participants see a significant challenge in evaluating their designs and prototypes with regard to the context, i.e. the driving situation (Context Consideration). The participants further describe that the driving task itself and the influences from the environment affect how the driver interacts with the system. The difficulty of recreating this driving situation in a lab experiment is explained by P1: “The difference lies in the dual-task paradigm. When you are in the lab, the interaction with the HMI is the primary task, when you are driving it is only the secondary task”. P5 adds that dynamic driving simulators offer the possibility to model the driving situation to a certain degree but that due to high cost and low availability they are only used for very few studies. The participants from digital companies focus on Data Interpretability and what methods need to be applied to make reliable statements.
In the post-design phase, the challenge of Data Accessibility reoccurs. Three automotive participants argue that the biggest challenge after a product’s release is to get field user interaction data to evaluate how the product is accepted by the users. P2 states that “one of the main challenges is to make the right data available at the right time”. In addition to the data being available, the challenges of intuitive Data Presentation are discussed by the participants as well. Six participants express that, due to the amount of data, field user interaction data needs to be visualized in an intuitively understandable way. P4 underlines this challenge by saying that “an 80-page pdf with results […] doesn’t help that much because nobody wants to read through it and it doesn’t motivate designers to change anything”.
Challenges in automotive (RQ2):
Field user interaction data is often not available or accessible throughout the design process due to organizational, legal, or technical restrictions.
User research is not valued; Evidence-based design decisions are overruled by management.
The complexity of the driving context further affects the already challenging task to create insights from large amounts of field user interaction data.
Differences to digital companies (RQ4):
The disparity between user insights and management opinions is less challenging for digital companies.
Digital companies face more mature challenges in terms of integrating data in their design process rather than technical or organizational challenges.
The distribution of codes addressing the needs of the UX experts does not show major differences between the automotive and the digital domain (see Figure 2). Most mentioned for both groups are explicit demands for data-driven Features. The features are manifold and range from dashboards visualizing feature-specific clickstreams to the implementation of data-driven analyses in design tools.
In the pre-design phase, 5 out of 8 participants from the automotive domain mention the explicit need for data-driven solutions to support their hypotheses and proposals made in early phases of development (Empowerment). This is connected to the state-of-the-art and the resulting challenges, that personal opinions in higher management play an important role in feature elicitation and prioritization. However, participants from digital companies do not express this need in the pre-design phase. They rather emphasize the significance of Qualitative Feedback and the need for Triangulation of different data sources. The need for qualitative data is important in early ideation phases, especially for new products. P10 states: “For a comprehensive redesign of a product you can’t test A/B, you have to […] test them qualitatively to see if it makes sense to implement the hypothesis”
“For a comprehensive redesign of a product you can’t test A/B, you have to […] test them qualitatively to see if it makes sense to implement the hypothesis”.
Revisiting the challenges of time restrictions and decision-making, 5 out of 8 automotive participants express a need to automatically evaluate their designs based on data retrieved from field usage (Feature and Evaluation Metrics). P9 agrees that such a feature would facilitate their advances toward a user-centered design approach: “[I]t can really help to defend my decisions. I guess, honestly, I’m always trying to defend it, not for myself but for the user”. However, regarding automated analyses and models based on field user interaction data, especially the digital domain participants express a need for explainability. P10: “When you have some kind of magic box where I present a prototype and a magic score falls out, of course, people who are not so much into UX would ask: ’ok, but what does the box do? How does it get that number? Can I even trust it?’ ”.
The needs expressed most often in the post-design phase address how to measure the acceptance of a developed product or feature by field users. The participants indicate a need for Evaluation Metrics that quantify user acceptance and how it changes over time. Among conventional metrics like the number of clicks or conversion rates, 3 out of 6 participants from digital companies say that it is necessary to correlate these ratings with other Key Performance Indicators like profit or newsletter subscriptions.
Needs of automotive (RQ2):
Statistical support based on field user interaction data to leverage design hypotheses, feature elicitation, and prioritization.
Tool support to automatically evaluate designs.
Automated methods should offer explanations to facilitate interpretability.
Differences to digital companies (RQ4):
In digital companies, many of the needs toward hypothesis support, feature elicitation, and feature acceptance assessment are already satisfied.
In digital companies, there is a greater need to triangulate qualitative and quantitative data.
In the pre-design phase, the automotive participants are particularly interested in the potential of Feature Activation Analysis, i.e. the evaluation of usage frequencies and duration. Especially for arguing against management opinions, 6 out of 8 automotive participants made statements that those analyses can satisfy the expressed need to empower them in their decisions. They explain that feature activation analyses accompanied by appropriate metrics can offer valuable insights into the field usage of features. Therefore, they can facilitate feature elicitation and prioritization. The participants further indicate that User Flow Analysis based on field user interaction data can provide a deeper understanding of how the users behave in the current system. P6 states: “[W]e are very good at building solutions but not always good at identifying the right problems” and formulates the idea to “take the personas themselves from the market research and enrich them with certain usage data that are important to understand the user journey”.
To overcome challenges in the design phase, 11 out of 14 participants indicate that automated design evaluation methods based on field user interaction data could offer valuable Design Support. This design support could be manifested in automated quantitative usability analyses or the extraction of usage patterns for different user groups from extensive field data. 13 out of 14 participants indicate that the usage of field user interaction data for User Modeling could play an important role in their design process. P10 suggests using a “model that represents a persona to automatically evaluate a prototype”. Another recurring theme is the topic of context consideration. 7 out of 8 automotive participants see the potential to use field user interaction data to analyze how the driving context affects user interactions with the product (Context Analysis). The interviewees argue that the context plays an important role in the automotive domain since the interaction with the environment is bidirectional. P1 states that it would be necessary to not only evaluate a feature based on its interaction data but also on how its usage influence the driving behavior. The latter has a direct and potentially fatal impact on its environment. This critical correlation could be evaluated by matching user interactions with driving data like lane-keeping parameters.
In the post-design phase, the participants see the biggest potential of field user interaction data in monitoring how features and products are accepted in the field. They argue that instant monitoring after release and an easy to understand data presentation would offer interesting insights into how often features are used and how the interaction changes over time. P1 elaborates on the direct connection to the subsequent pre-design phase: “Requirement analysis would also mean looking at the data that was collected at the end of the last version again. This should ideally be a cycle and I see the methods data-driven analyses offer at every point in this development lifecycle”.
Potentials in automotive (RQ3):
Insights from field user interaction data can shift the elicitation and prioritization of features from personal best guesses to more user-centered decisions.
Automated evaluation methods and user modeling based on field user interaction data may offer valuable design support.
Field user interaction data can be triangulated with contextual data to investigate the impact of the driving situation on the interaction and vice versa, making evaluations less biased.
Differences to digital companies (RQ4):
Most identified potentials apply to both, automotive and digital domains, but digital companies are more advanced in unlocking these potentials.
In the pre-design phase, the participants express few concerns toward data-driven methods and the analysis of field user interaction data. P1 and P7 do not see any benefit of the discussed methods when it comes to the early ideation phases of a product. P7 states that “[t]hat’s an interesting insight that maybe all the data-driven stuff has a bigger impact on everything where you try to optimize something in contrast to the work where the creative process is the main part”.
The predominant concern in the design phase regards the Limitation of Creativity of the designers which might be caused by extensive use of data-driven analyses. This is strongly connected to the concerns in the pre-design stage, as participants from both groups see a risk to get stuck in small, iterative optimization processes. They anticipate that optimizing features based on historical data prevents thinking outside of the box to create something new. These concerns are related to the concerns toward User Models. P10 states that it is difficult to build a model without limiting creativity and describes it as an “overfitting problem: the model has seen too much old data and is therefore not able to generalize when it is applied to something new”. Further, 7 participants (4 automotive) are concerned about how to interpret the results produced by an automated evaluation method (Interpretability). They mainly argue that an explanation has to be provided to develop trust in automatic evaluation “because usage scores alone produce very little insight”, according to P12. P4 agrees by indicating: “A score might be ok, but there should be suggestions or information on how the score is calculated and influenced”.
There are very few concerns regarding the post-design phase. However, 5 out of 8 automotive participants communicate general doubts that legislation may prevent certain features and functions from being realized due to data collection restrictions. This especially holds for potentially person-related data, e.g. GPS coordinates of a vehicle.
Concerns in automotive (RQ2):
Insurmountable organizational, legal, or technical restrictions prevent that data can be collected.
Quantitative insights may not be useful in early ideation phases to evaluate volatile concepts.
Data-driven methods may limit creativity and shift the focus to small incremental changes.
Differences to digital companies (RQ4):
Participants from digital companies expressed more concerns.
A lack of interpretability can lead to disuse of data-driven methods.
In this section, we reflect on the needs, challenges, and concerns expressed by the practitioners and emphasize untapped potential in the evaluation and development of IVISs. We additionally relate our findings to prior published research and present methods that may benefit the automotive UX development lifecycle.
Leadership, culture, and the mindset within a company highly influence the usage of data-driven methods. Compared to digital-native companies, automotive OEMs find it difficult to keep up when it comes to the integration of data-driven methodologies in UX development. However, data-driven methods based on field user interaction data can benefit the development and evaluation of IVISs. In the pre-design phase, we see great potential in generating a deeper understanding of the users and their behavior through analyses of field user interaction data. Data-driven methods can facilitate decisions in early phases to prioritize features or products. Multiple approaches (Liu.2017; Wang.2016; Benevenuto.2009) that enhance the user understanding based on analyses of automatically collected field user interaction data can be leveraged to unlock this potential for IVISs. However, as participants also mentioned, these approaches should be considered as an additional source of user feedback and not as a replacement for already existing qualitative methods.
In the design phase, automated usability tests (Speicher.2015; Deka.2017; Ma.2012) can play an important role in making the design process more user-centered and efficient at the same time. The fact that the context of use, i.e. the driving situation, is inherently contained in field data is another key advantage. Additionally, the possibility to explicitly map field user interaction data with naturalistic driving data creates new opportunities in the design and evaluation of IVISs. One can, for example, predict driver distraction (Salis.2019), secondary task engagement (Risteska.2018), or identify drivers based on their driving behavior (Zhang.2016). This allows considering the complex interactions between driver, car, and environment without the costs and bias introduced by simulator experiments. This is in line with the findings made in earlier work on this topic (Orlovska.2018; Orlovska.2018b). However, to provide the biggest possible value for practitioners, all automated methods should provide an explanatory component and be triangulated with qualitative user feedback.
In the post-design phase, there is a need to monitor the acceptance and usage of IVISs after deployment. Here, data-driven methods offer insights that can then benefit the next development cycle.
However, in line with the findings of Lamm.2019 (Lamm.2019), automated and model-based approaches currently do not play an important role in the evaluation of IVISs. This originates from none of the interviewed OEMs having a system in place that is explicitly developed to record detailed user interactions. Current systems are yet built for different purposes and only modified to offer basic capabilities, while dedicated systems are only available for test fleets. Legacy car architectures and long product life cycles aggravate the difficulty to implement such new systems. Additionally, strong restrictions regarding privacy and security are challenging for OEMs. However, a dedicated system for interaction logging that provides detailed and high-quality data is the cornerstone of the potentials introduced by field data-driven methods.
Our results show that data-driven methods based on field user interaction data can have great value for the automotive UX development lifecycle and can play an essential role in making the development of IVISs more user-centered. All automotive domain experts in our study agree that there is a lack of implicit feedback through field user interaction data in UX development. These findings coincide with the work of Orlovska.2018 (Orlovska.2018; Orlovska.2018b). Additionally, our results support the disparity presented in Section 2.3 that in comparison to the automotive domain, digital domains are far ahead in exploring the potentials of field user interaction data. We conclude that the transition from predominately explicit and qualitative user feedback, e.g. through customer surveys or studies, to a combination of the former with implicit feedback through automatically collected field user interaction data is a necessary process for the development of an automotive UX fulfilling users’ diverse needs. Another important benefit is the statistical support of designers’ decisions to overcome the current opinionated guesswork often encountered in automotive UX development. Interestingly, in the automotive and the digital domain, we identified a high potential for the automated evaluation of field user interaction data and advanced user modeling based on interaction data for early prototype evaluation. These identified potentials will be subject to future work to unlock the benefits field user interaction data offers. Finally, the results of this work facilitate research on data-driven methods in the automotive UX development by directing the focus toward the needs, challenges, and concerns UX experts face today.