Log In Sign Up

Size matters? Or not: A/B testing with limited sample in automotive embedded software

by   Yuchu Liu, et al.

A/B testing is gaining attention in the automotive sector as a promising tool to measure casual effects from software changes. Different from the web-facing businesses, where A/B testing has been well-established, the automotive domain often suffers from limited eligible users to participate in online experiments. To address this shortcoming, we present a method for designing balanced control and treatment groups so that sound conclusions can be drawn from experiments with considerably small sample sizes. While the Balance Match Weighted method has been used in other domains such as medicine, this is the first paper to apply and evaluate it in the context of software development. Furthermore, we describe the Balance Match Weighted method in detail and we conduct a case study together with an automotive manufacturer to apply the group design method in a fleet of vehicles. Finally, we present our case study in the automotive software engineering domain, as well as a discussion on the benefits and limitations of the A/B group design method.


page 1

page 2

page 3

page 4


Bayesian propensity score matching in automotive embedded software engineering

Randomised field experiments, such as A/B testing, have long been the go...

Bayesian causal inference in automotive software engineering and online evaluation

Randomised field experiments, such as A/B testing, have long been the go...

Software Engineering in Civic Tech: A Case Study about Code for Ireland

Civic grassroots have proven their ability to create useful and scalable...

An architecture for enabling A/B experiments in automotive embedded software

A/B experimentation is a known technique for data-driven product develop...

Qualitative Research on Software Development: A Longitudinal Case Study Methodology

This paper reports the use of a qualitative methodology for conducting l...

Emotionalism within People-Oriented Software Design

In designing most software applications, much effort is placed upon the ...

A New Method for the High-Precision Assessment of Tumor Changes in Response to Treatment

Imaging demonstrates that preclinical and human tumors are heterogeneous...