Common Evaluation Pitfalls in Touch-Based Authentication Systems

by   Martin Georgiev, et al.

In this paper, we investigate common pitfalls affecting the evaluation of authentication systems based on touch dynamics. We consider different factors that lead to misrepresented performance, are incompatible with stated system and threat models or impede reproducibility and comparability with previous work. Specifically, we investigate the effects of (i) small sample sizes (both number of users and recording sessions), (ii) using different phone models in training data, (iii) selecting non-contiguous training data, (iv) inserting attacker samples in training data and (v) swipe aggregation. We perform a systematic review of 30 touch dynamics papers showing that all of them overlook at least one of these pitfalls. To quantify each pitfall's effect, we design a set of experiments and collect a new longitudinal dataset of touch dynamics from 470 users over 31 days comprised of 1,166,092 unique swipes. We make this dataset and our code available online. Our results show significant percentage-point changes in reported mean EER for several pitfalls: including attacker data (2.55 (3.2 of these evaluation choices result in a combined difference of 8.9 also largely observe these effects across the entire ROC curve. Furthermore, we validate the pitfalls on four distinct classifiers - SVM, Random Forest, Neural Network, and kNN. Based on these insights, we propose a set of best practices that, if followed, will lead to more realistic and comparable reporting of results in the field.


Continuous User Authentication Using Machine Learning and Multi-Finger Mobile Touch Dynamics with a Novel Dataset

As technology grows and evolves rapidly, it is increasingly clear that m...

Continuous User Authentication via Unlabeled Phone Movement Patterns

In this paper, we propose a novel continuous authentication system for s...

Adversarial Attacks on Remote User Authentication Using Behavioural Mouse Dynamics

Mouse dynamics is a potential means of authenticating users. Typically, ...

Techniques for Continuous Touch-Based Authentication Modeling

The field of touch-based authentication has been rapidly developing over...

Continuous Authentication Using Mouse Movements, Machine Learning, and Minecraft

Mouse dynamics has grown in popularity as a novel irreproducible behavio...

What's in Score for Website Users: A Data-driven Long-term Study on Risk-based Authentication Characteristics

Risk-based authentication (RBA) aims to strengthen password-based authen...

Please sign up or login with your details

Forgot password? Click here to reset