FMCode: A 3D In-the-Air Finger Motion Based User Login Framework for Gesture Interface

08/01/2018 ∙ by Duo Lu, et al. ∙ Arizona State University 0

Applications using gesture-based human-computer interface require a new user login method with gestures because it does not have a traditional input method to type a password. However, due to various challenges, existing gesture-based authentication systems are generally considered too weak to be useful in practice. In this paper, we propose a unified user login framework using 3D in-air-handwriting, called FMCode. We define new types of features critical to distinguish legitimate users from attackers and utilize Support Vector Machine (SVM) for user authentication. The features and data-driven models are specially designed to accommodate minor behavior variations that existing gesture authentication methods neglect. In addition, we use deep neural network approaches to efficiently identify the user based on his or her in-air-handwriting, which avoids expansive account database search methods employed by existing work. On a dataset collected by us with over 100 users, our prototype system achieves 0.1 user authentication, as well as 96.7 identification, using two types of gesture input devices. Compared to existing behavioral biometric systems using gesture and in-air-handwriting, our framework achieves the state-of-the-art performance. In addition, our experimental results show that FMCode is capable to defend against client-side spoofing attacks, and it performs persistently in the long run. These results and discoveries pave the way to practical usage of gesture-based user login over the gesture interface.



There are no comments yet.


page 11

page 13

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

I Introduction

Gesture interfaces are generally considered as the next generation of Human-Computer Interface (HCI) which can fundamentally change the way we interact with computer. Moreover, platforms equipped with gesture interfaces such as home entertainment consoles (e.g., Microsoft XBox with Kinect [1]), Virtual Reality (VR) headsets (e.g., HTC Vive [2]), and wearable computers (e.g., Google Soli [3]) are becoming popular in various application scenarios. On these platforms, user authentication and identification are basic functions for accessing resources, personalized services, and private data. For example, a user may be asked to verify the identity to unlock a wearable device similar as unlock a smartphone, or sign in a virtual site over a VR headset. However, in such an environment, it is usually inconvenient or impractical to access a keyboard or virtual keyboard on a touchscreen. Moreover, in certain scenarios such as clean room or operating theater, gesture interface is used to avoid physical touch due to high cleanliness. Therefore, research needs to answer how to login securely and conveniently with the gesture interface instead of using a keyboard or touchscreen. Current VR headsets (e.g., Oculus Rift and Vive) and wearable devices rely on the connected desktop computer or smartphone to complete the login procedure using either passwords or biometrics, which is inconvenient. A few standalone VR headsets (e.g., Microsoft Hololens [4]) present a virtual keyboard in the air and ask the user to type a password. However, it is slow and user unfriendly due to the lack of key stroke feedback and a limited recognition accuracy of the key type action in-the-air. Moreover, passwords have their own drawbacks due to the trade-off between the memory difficulty and the password strength requirement. Biometrics employ the information strongly linked to the person, which cannot be revoked upon leakage and may raise privacy concerns in online login. Therefore, the usage of feature-rich behavioral biometrics such as the in-air-handwriting is preferable.

During the past decades, identity verification methods involving writing the password in the air have been studied on different input interfaces such as hand-held devices [5, 6, 7], cameras [8, 9, 10, 11], and touchscreens [12, 13, 14]. However, the application of such a method encounters a few fundamental challenges. First, gesture input sensors have limited capabilities in capturing hand movements (e.g., limitation in accuracy, resolution, sampling speed, field of view, etc.

). Meanwhile the user’s writing behavior has uncertainty in posture and magnitude. These facts make signal processing and feature extraction difficult. Second, the captured handwriting contains minor variations in speed and shape even for the same user writing the same content. Unlike the password that does not tolerate a single bit difference, this ambiguity in the in-air-handwriting leads to difficulty in designing matching algorithms and limited discriminating capability. Hence, existing solutions rarely achieve an acceptable authentication performance. Third, user identification requires indexing a large amount of accounts using the ambiguous, inaccurate and noisy in-air-handwriting motion signals in order to efficiently locate the desired account upon a login request, which cannot be accomplished directly by current template matching based methods. As a result, existing solutions have to search the account database exhaustively to compare the signal in the login request with the template, which is impractical for real world usage. Fourth, for data-driven methods that train a model to recognize each account and classify the handwriting, the available signal samples at registration are scarce because of usability consideration. Since we can only ask the user to write the passcode a few times to sign up), the effectiveness of model training is significantly restricted.

Fig. 1: User login through gesture interface with hand motion capturing devices of wearable inertial sensor or 3D depth camera under two different scenarios: (left) VR applications with user mobility, (right) operating theater with touchless interface for doctors to maintain high clearliness once the hands are sterilized.

To address these problems, we propose FMCode (i.e., Finger Motion Passcode), a unified user login framework by writing an ID and a passcode in the air with the index finger, shown in Fig. 1. The finger motion is captured by either a wearable inertial sensor or a 3D depth camera and sent to the server as a login request. The user can write very fast, write in a language unable to be directly typed with a keyboard, or just doodle, as long as the finger motion can be easily reproduced in a stable way. Based on our datasets, usually the content of the ID or the passcode is a meaningful string or a shape that is easy to remember, while the writing can be just scribbles unreadable for anyone except the creator. This enables a much larger space for the ID and the passcode than the traditional password consisting of only typed characters. Additionally, because of the difference in writing convention of different people, memorable passcodes are not susceptible to attack as traditional password. For example, the user can draw a five-point star as an ID or a passcode which cannot be typed over the keyboard. Meanwhile, it has been proved that graphical memory is easier to remember and stays longer than passwords typed over the keyboard [15]. Due to this unique feature, we call the in-air-handwriting as a “passcode” instead of a “password”. Different from handwriting recognition, our framework does not try to understand each character, but identifies and authenticates users based on multiple factors including both the difference of passcode content and the fineness of handwriting convention. Hence, FMCode has both advantages of password such as revocability and advantages of biometrics such as non-repudiation.

The contributions of this paper are listed as follows.

  1. We present a unified login framework for gesture interface using the finger motion in-the-air, which can perform both user authentication and identification efficiently. Our framework supports gesture interface using either a wearable sensor or a contactless 3D camera.

  2. We design a type of feature named temporal local difference and train an ensemblement of Support Vector Machine (SVM) classifiers for each account to improve the discriminant capability. Our method can tolerate minor writing behavior variation and perform reasonably well in the long term.

  3. We design a deep Convolutional Neural Network (CNN) to index finger motion signals, which enables user identification with a constant time cost. Moreover, we invent special data augmentation methods to train this network with limited amount of data acquired at registration.

  4. We achieve state-of-the-art performance on datasets collected by us with two types of gesture input device - a custom data glove and a Leap Motion controller. The dataset contains more than 100 subjects from general public and over 20,000 data points, which is larger than most related work.

  5. We study the performance under active spoofing attacks and the long term stability with a duration of four weeks. These scenarios are usually neglected by existing works. Our results show that the FMCode framework has the capability to defend client side spoofing attack and perform persistently in the long term. This reveals great potential of this finger motion based user login method for practical usage.

The structure of the paper is organized as follows. In Section II, we describe the overall system architecture and in Section III we model the signal of the in-air-handwriting motion. Section IV and V explains details of the proposed authentication and identification methods. In section VI we report empirical evaluation results with our datasets, and in section VII, we present user evaluation results. Further discussions are given in section VIII, and the related work is provided in Section IX. Finally, we draw conclusions in section X. More detailed comparison with existing work and alternative technologies are shown in the Appendices.

Fig. 2: System Architecture and Procedures.

Ii System Model

Ii-a FMCode Architecture

Similar to password based login systems, FMCode requires an initial registration comparable to the password “sign up” step. At the registration step, the user is required to create an account with a unique ID and a passcode, and write the ID as well as the passcode a few times in the air. After that, the user can sign in to the created account by writing the same ID and passcode in the air.

There are two functional modules in the framework: (a) a gesture user interface device that is equipped with motion capture sensors, and (b) a login server that stores the account database, as shown in Fig. 2. On the user side, the finger motion of a piece of the in-air-handwriting of the ID and the passcode is obtained, and two corresponding signals containing the physical states of the hand are generated. Then, the gesture UI device preprocesses the signals and sends them to the login server. On the server side, each account in the database contains a (account number, id_template, passcode_template) tuple. The account number is a number or a string of characters usually allocated by the server to uniquely index each account. The two templates are generated by the in-air-handwriting signals of the ID and the passcode at registration in order to match with signals in the login request. Once the registration is complete, the login server has the following two main functions that can be used independently or together.

1) User Authentication: given the account number, verify the user identity, and “accept” or “reject” the login request using a piece of in-air-handwriting of the passcode in the request. The account number may be typed, remembered, recognized by face or other means, or obtained using the identification function detailed below. In this function, the server executes the following three steps: (a) retrieves the passcode_template of the user’s account according to the account number from the account database (step 3 in Fig. 2), (b) compares the template with the signal in the login request to extract features (step 4 in Fig. 2), (c) determines whether this login request is accepted or rejected using a binary classifier trained at registration (step 5 in Fig. 2).

2) User Identification: figure out the account number based on a piece of in-air-handwriting of the ID. As mentioned in the previous function, an account number is required, which is a number or a character string inconvenient to enter through the gesture user interface. Thus, we can ask the user to write the ID in the air. In this function, the server first obtains one or multiple candidate account numbers using a deep CNN (step 2 in Fig. 2). Then for each of them, the server runs the same three steps as the authentication to verify each of them by comparing the id_template and the signal of the in-air-handwriting of the ID. Finally, the best matched account number is returned. If all candidate IDs fail the verification, “unidentified” is returned.

Note that identifying the account number does not necessarily mean authenticating the user at the same time because an ID is usually not a secret unlike a passcode. The object of authentication is low error and high security level, while the objective of identification is fast speed with an acceptable accuracy. The login procedure is essentially performing both identification and authentication. Moreover, the user can explicitly update or revoke his or her FMCode just like updating or resetting a password at anytime. In addition, the server can generate fuzzy hash from the in-air-handwriting of the passcode using a similar deep CNN [16]. This fuzzy hash can be used to further generate cryptographical keys to enable more sophisticated authentication protocols or encrypt the template used in the login procedure in order to minimize the risk of server storage leakage (discussed in section VIII).

Ii-B System Requirements and Application Scenarios

FMCode is compatible with existing IT infrastructures using password based authentication. On the server side, only software changes are required, including the construction of templates, the implementation of feature extraction algorithms, the classifier and the deep CNN. The requirement of the network between the client and the server is the same as most password-based authentication systems, through the web. On the client side, a motion capture device such as a wearable device or a 3D depth camera is required. However, it should be noted that our login framework leverages the built-in gesture interface of the client machine rather than requires a dedicated device for the login purpose. As long as the finger motion can be captured for ordinary gesture based interaction with the computer, our framework can be deployed. Besides, login through our framework requires the active involvement of a conscious and willing user, rather than presenting a piece of static information such as password or fingerprint. This makes FMCode potentially more secure.

Our target application scenarios include VR headsets and wearable computers that already have a gesture interface but lack keyboard or touchscreen, as well as scenarios that provide gesture interface but inconvenient for typing such as operating theater or clean room. Our framework can be used for two different purposes. The first one is online user authentication or identification, where the server is remotely connected to the client via the Internet. For example, the user can sign into an online personal account through the gesture interface on a VR headset. The second one is local user authentication or identification, where the client and server reside on the same machine. For example, the user can unlock his or her wearable devices through the gesture interface. In addition, our framework can also be used as a supplementary authentication factor in a Multi-Factor Authentication (MFA) system together with traditional password or biometrics.

Ii-C Attack Model

FMCode has the same security assumptions as existing gesture-based authentication and identification systems as follows: (1) the device on the user side is secure(i.e., no sniffing backdoor); (2) the authentication server is secure (i.e., it will not leak the stored template to malicious attackers); and (3) the communication channel between the user and the server is secure (i.e, no man-in-the-middle attack). These security assumptions are also similar to traditional biometrics and password-based system. Attacks with relaxed security assumptions on (2) and (3) are further discussed in section VIII. Based on the assumptions, we are mainly interested in attacks on the user side, as listed below:

1) Random guess, i.e., the attacker tries to enter a user’s account by guessing a passcode and signs it on the same gesture interface, without any knowledge of the content of the passcode;

2) Spoofing attack, i.e., the attacker knows the content and broad strokes of the passcode of an account and tries to write it in the air through the same gesture interface. This is similar to the case that the attacker sign into the victim’s account with the password leaked.

For the spoofing attack, we assume that the attack source is a human attacker, and the attacker’s goal is to sign into the victim’s account or be identified as the victim. If the attack is successful, the account owner may suffer from loss of the account or leakage of private information. Though it is generally considered that ID is not a secret, in extreme cases, if the attacker is wrongly identified as the victim, he or she may launch further attacks e.g., phishing other sensitive personal information of the victim.

Iii In-Air-Handwriting Characterization

Iii-a Finger Motion Signal Model

Fig. 3: The In-Air-Handwriting Signal Model.
Fig. 4: Example of 3-dimensional finger motion signal (upper) when writing “FMCODE” and the correspondence of signal segments to letters (lower).

Handwriting is closely related to the cognitive process of humans [17] in different levels, and hence, we model the in-air-handwriting as a stochastic process in four levels: passcode level, character level, stroke level, and signal level (shown in Fig. 3). Usually each passcode is a string of meaningful symbols or characters in some language. The passcode is made of strokes defined by calligraphy, and the strokes further determine the hand motion of the writing through the muscle memory. The hand motion is captured as a series of samples of physical states of the hand and fingers. Here the passcode, characters, and strokes can be regarded as hidden states in a stochastic process, and only the signal samples are the observations. In general the in-air-handwriting process does not satisfy the Markov property (i.e., signal samples are correlated), and the mapping between signal samples and strokes are not fixed due to the minor variations of the writing speed and amplitude. However, the inherent process in the brain of generating hand motion by writing is acquired and reinforced when a person learns how to write [18], which indicates that the writing behavior is bound to individuals and persistent in the long term, as handwriting signature has been widely used for identity verification for a long time.

We use a vector series to denote the finger motion signal with samples, where represents an individual sample obtained by the sensor with axes. For example, if the signal is obtained from a wearable device with an inertial sensor, each sample may have three axes representing the acceleration of the fingertip of the right hand along the x, y and z-axes, and the whole signal may have 250 samples at 50 Hz. Assume the signal is aligned in a fashion that the writing speed variation is removed, it can be decomposed as

where is a constant vector determined by the content of the in-air-handwriting, and

is a vector of Gaussian random variables caused by the sensor noise and unintentional small hand movements. Since

is from different orthogonal sensor axes, we assume these axes are independent, i.e., , where . An approximation of and can be obtained by the signals at registration,


is the Maximum Likelihood Estimation (MLE) of

and is the unbiased MLE of . For each account, is stored as the or the , depending on whether is obtained by writing the ID or the passcode. is also stored together with the template to indicate the template uncertainty. The aligned signal set can be obtained by aligning each raw signal at registration to the first signal. An example is shown in Fig. 5. In our framework, alignment is made by the Dynamic Time Warping (DTW) algorithm.

Fig. 5: Example of 10 aligned finger motion signals (upper), 2 unaligned signals (middle), and the generated template (lower, where is shown as the blue line, is shown as the red line, and is enlarged by two to show significance). Only the first sensor axis is shown. Best viewed with color.

Iii-B In-Air-Handwriting Signal Preprocessing

As mentioned previously, there are also user behavior uncertainty in the writing posture and magnitude. To minimize the influence of such uncertainty, the following preprocessing steps are applied on the client side.

Step 1) State estimation: derive the indirect dynamic states from the raw sensor output and fix any missing signal samples due to the limited capability of the motion capture device. For the wearable device with inertial sensors, we derive the absolute orientation of the index finger relative to the starting position [19]. For the 3D camera device, we derive the velocity and acceleration for each sample from the position difference. The position trajectory and posture are estimated from the depth image frames by the 3D camera device itself.

Step 2) Trimming: throw away the sample at the start and the end of the signal where the hand does not move intensively. In practice, the signal can be trimmed in a more progressive way because we observed that the user behavior has larger uncertainty at the beginning and the end.

Step 3) Low-Pass filtering and resample: remove the high-frequency components above Hz because it is commonly believed that a person cannot generate finger movements faster than Hz. Then the signal is resampled at 50 Hz to remove influence on the variation of sampling rate.

Step 4) Posture normalization: translate the coordinate system to make the average pointing direction of the hand as the x-axis in order to remove the influence of posture variation respect to the motion capture device. For the data glove with inertial sensors, we also remove the influence of the gravity on the acceleration axes.

Step 5) Amplitude Normalization: normalize the data of each sensor axis individually, i.e., where ,

Iv User Authentication

The task of authentication is essentially a binary classification problem. Our design goal is to build a data-driven model that can optimally distinguish the signals from legitimate users and attackers. Given an account A and an authentication request with signal obtained from the in-air-handwriting of the passcode (referred as the passcode signal), we define the following classes.

  1. If is generated by the owner of account A writing the correct passcode, is from the “true-user” class;

  2. if is generated by any user writing an incorrect passcode, is from the “guessing” class (which means the writer does not know the passcode content);

  3. if is generated by an imposter writing the correct passcode of account A, we define that is from the “spoofing” class (which means the attacker knows the passcode content).

The “guessing” and “spoofing” classes are collectively called the “not-true-user” class. Our authentication method is based on the signal model explained previously – signals generated by the same user writing the same passcode have similar shape if they are aligned because they contain the same sequence of strokes. Hence, we define a temporal local distance feature that measures the difference of the signals locally in stroke level. Moreover, we also design a method to generate multiple feature vectors from just one pair of signal and template to overcome the shortage of training data at registration. Furthermore, we use an ensemblement of SVM classifiers for each account to distinguish signals from the “true-user” class and “not-true-user” class to maintain a stable long term performance.

Iv-a Feature Extraction

Given an account A, consider is the passcode signal in an authentication request, and is the of account A constructed at registration with uncertainty . The temporal local distance features are extracted as follows.

Step 1) Align to using DTW, so that the aligned will have the same length as .

Step 2) Calculate the distance , where is the element-wise absolute function.

Step 3) Segment into local windows, and each window has length , i.e., regroup as , where .

Step 4) Randomly pick different local windows as a window set , then randomly select a local distance feature from each window to form a feature vector , where each element is chosen from . Here is defined as the temporal local distance feature. For example, assume has 10 samples, segmented to five windows, and we can randomly pick two windows (i.e., ). Consider we pick the third window ( to ) and the fifth window ( to ), then we can form a feature vector by randomly choosing one sample from each window, such as .

Step 5) Given a certain window set, step 4 can be repeated multiple times to generate multiple feature vector from one pair of signal and template. Especially, we can regard as a Gaussian random variable and draw samples from the distribution in step 4. This technique allows us to augment the training data from the limited signals with “true-user” label at registration.

Fig. 6: Convolutional neural network for user identification.

Iv-B Binary Classification for Authentication

The SVM classifier is a binary classifier with linear decision boundary in the feature space that can be trained efficiently even with limited amount of data and high dimension of features. These characteristics are suitable for the authentication task. Given a training dataset with data points , where are the feature vectors and are binary class labels from

, SVM seeks a hyperplane

to maximize the separation of the data points of the two classes. Training SVM is equivalent to solving a quadratic programming problem, which can be done efficiently. However, since the decision boundary is very simple, naively applying SVM on

obtained from limited signals at registration would still suffer from the curse of dimensionality problem and lead to poor long term stability. Hence, we train an ensemble of SVM as follows.

Consider we are registering the account A and the template is constructed from signals . At registration, the server builds SVM classifiers for account A, with a distinct set of windows randomly picked for each classifier. To train a single classifier, first the server extracts feature vectors from those registration signals of account A, and assigns them the label (i.e., “true-user” class). Then the server extracts feature vectors from those registration signals of other accounts except the account A, and assigns them the label (i.e., “guessing” class). Usually there are more training data of the “guessing” class than necessary. Thus, only a portion is needed (usually around one thousand). After the feature vectors and lables of both classes are ready, an SVM is trained using Sequential Minimal Optimization (SMO). Finally, once all classifiers are trained, the server stores the model parameters , , and the set of windows of each classifier in the account database together with the template .

When signing into account A, given a signal in the authentication request, the server extracts feature vectors for each SVM classifier using the stored information, and predicts a score . Since multiple feature vectors can be extracted with one set of windows, the server can obtain multiple scores from a single and average them. Once the scores of all classifiers are ready, they are further averaged to produce a single distance score, i.e., . Finally, this score is compared with a pre-set decision threshold. If , the authentication request with signal is accepted, otherwise, it is rejected.

The aim of the feature extraction method and classifier ensemblement is to achieve a better separation of signals from different classes in the feature space and maintain a stable performance in the long term. If a signal is from the “true-user” class, it should have a small score because similar shape between the signal and the template leads to smaller . While signals from the “not-true-user” classes should have a larger score caused by large values of elements of due to shape differences, whose origin is the different content or different writing convention. However, the distance in sample level has uncertainties because of the minor variation of writing behavior for the same user writing the same content. Misclassification caused by such uncertainties may happen quite often if we blindly sum the sample level distance, as the plain DTW algorithm. Instead, we group local samples into segments which roughly map to the strokes, and hence, our method can tolerate the signal level uncertainties by comparing subset of strokes instead of samples. The final score of the ensemblement of classifiers is essentially a weighted sum of the sample-wise distance , where the trained weights help select those segments with less uncertainty and more consistency. In our framework, , and are a system-wide parameter. is usually chosen from 20 to 80 to approximate the number of strokes of a in-air-handwriting passcode. is usually empirically chosen based on the passcode length to avoid the curse of dimensionality. is determined as a trade-off between the computation time and authentication accuracy. In an extreme case, we can make and , which means only a single SVM is used to draw a decision boundary in a very high dimensional feature space (potentially this dimension can reach several hundred or even exceed one thousand). This may cause classifier stability issues in the long term because some local features may be wrongly considered to be consistent due to the limited amount of training data.

V User Identification

Different from authentication, the task of identification is essentially a multi-class classification problem, which must be done efficiently without query the account database linearly. Deep CNN is a type of neural network consisting of cascaded convolutional layers and pooling layers. The most attractive capability of deep CNN is that it can learn and detect features from low-level to high-level automatically by optimizing a loss function iteratively. This characteristic is crucial to solve very complicated pattern recognition problems, most notably the image classification

[20, 21, 22]. As illustrated previously in the signal model, a piece of in-air-handwriting contains hierarchical features in different abstract level, and hence, a deep CNN is suitable to our objective of mapping a signal of the in-air-handwriting of the ID (referred as the ID signal) to its corresponding account number. However, deep CNN has not been used in in-air-handwriting signal based user identification since the features expected to be learned in 3D handwriting signal are fundamentally different from 2D image, which requires the following special treatment.

The deep CNN in our framework contains five convolutional-pooling layers, one fully-connected layer, and one softmax layer, as shown in Fig.


. The raw signal is first preprocessed and stretched to a fixed length of 256 elements through linear interpolation, in order to be fed into the CNN. For example, if the number of sensor axes is 9, the input is a 256 by 9 matrix, where each sensor axis is regarded as an individual channel. For each convolutional-pooling layer, we apply a convolutional kernel of size 3 on all the channels of the previous layer, and a 2-by-1 maxpooling on the output of the convolutional layer for each channel. The first two convolutional layers utilize depthwise convolution

[22] which detects local features individually on each channel since these channels contain different physical states in orthogonal axes. The later three convolutional layers utilize separable convolution [22]

which associates low level features on all channels to construct high level features. For example, each neuron in the third conv-pool layer has a receptive field of 16 samples in the original signal, which is roughly corresponds to one stroke. There are 96 filters in this layer which can map to 96 different types of features in the stroke level. These features can capture different types of basic finger movement when writing a single stroke, including straight motion in different directions, and sharp turning between adjacent strokes. Hence, the output of this layer is a 32 by 96 matrix indicating the presence and intensity of a certain type of stroke (among all 96 types) at a certain place of the signal (among 32 slightly overlapping segments). Similarly, the fourth and fifth conv-pool layers are designed to detect the presence of certain characters and phrases. Finally, a fully connected layer runs classification on the flattened high level features and generates the embedding vectors, and the softmax layer maps the embedding vectors to probability distribution of class labels (

i.e., the account numbers).

A major challenge of training the CNN is the limited amount of data at registration, which leads to overfitting easily. To overcome this hurdle, we augment our training dataset with the following three steps. First, given signals obtained at registration, for each in this set, the server align all the other signals to to create new signals. Second the server randomly picks two aligned signals and exchanges a random segment. This can be done many times to further create many new signals. Third, for each newly created signal, we randomly perturb a segment both in time and amplitude. Besides of the data augmentation, we apply dropout [23] on the fully-connected layer.

To predict the account number of an input signal, the server simply chooses the most probable class or top-k most probable classes in the output probability distribution. However, blindly believing the predicted account number of the CNN may render the server extremely vulnerable to spoofing attacks because the decision is based on the presence of certain strokes detected by the feature layers, and a spoofed signal generating by writing the same content as the genuine signal naturally has the majority of the strokes. As a result, in our framework the server performs an additional step to verify the identity, using the same procedure as the authentication. Instead of passcode signal, here the server compares the ID signal and the id_template of the account corresponding to each candidate account number. Finally, the server returns the best matched account number or “unidentified” if the scores of all account are above the threshold.

Vi Experimental Evaluation

Vi-a Data Acquisition

Fig. 7: Distribution of passcode lengths of the signals in dataset 1.

To evaluate our framework, we build a prototype system with two types of gesture input device. The first device is a custom made data glove with an inertial measurement unit (IMU) on the tip of the index finger (referred as the glove device). Such an IMU has been widely used in the hand-held remote controller of current VR game console and smart TV, and it is relatively inexpensive ($10). The glove also has a battery-powered microcontroller on the wrist, which collects the IMU data at , and runs the state estimation preprocessing step. The output signal of this device contains a series of three Euler angles in , tri-axis acceleration in , and tri-axis angular speed in . The second device is the Leap Motion controller (referred as the camera device), which is an inexpensive ($80) commercial-off-the-shelf 3D camera. It contains a stereo infrared camera that can capture depth image at with an average Field of View (FOV) and a range of . The Leap Motion controller has its own processor that can estimate the hand skeleton of each frame and obtain the 3D coordinate of each joint of the hand. The client machine runs the state estimation and other preprocessing steps, and outputs signal of a series of 3D position, velocity, and acceleration.

Although our prototype system uses these two specific devices for proof of concept and evaluation, the proposed framework does not necessarily depend on these two devices. It should work with any similar device that can return samples of physical states of the fingers and the hand with a reasonable range, resolution and sampling rate. For example, there are other gloves [24] and rings [25] with inertial based motion tracking, and there are open source software [26] available that can estimate hand skeleton from 2D image or 3D depth image. We assume these devices are part of the gesture user interface for ordinary human-computer interaction, not specialized device dedicated for the login purpose.

We built our own datasets as follows:

1) We invited 105 and 109 users to participate in the data collection with the glove device and the camera device respectively. 56 users used both devices. Each user created two distinct strings. For each string, the user wrote it in the air for 5 times as registration, and after a rest, they wrote it 5 times again as login. The distribution of lengths of all strings are shown in Fig. 7 (users write a little slower with the camera device due to its limited FOV and range).

2) For each of the string in the first dataset, 7 impostors spoofed it for 5 times (due to the size of the first dataset there are more than 7 impostors in total). Here “spoof” means that the impostors know the content of the string, and try to write it using the same motion capture device as that used by the spoofing target user.

3) We asked 25 users participating in the first dataset to write the two created strings in multiple sessions. For each of the string, besides the 5 times at registration, the user wrote it 5 times as one session, two or three sessions a week on different days, for a period of 4 weeks (10 sessions in total).

4) Among the users who participate in the data collection of both devices in the first dataset, 28 of them filed a questionnaire on the usability of our prototype system (detailed in section VII)

The datasets and data acquisition method are based on our previous works [27, 28]. However, our datasets are larger with two types of devices. The participating users are from the general public including both office workers and non-office workers with an age range of 17 to 65.

At the beginning of the data collection, we briefly introduced our system to the users and informed them that the in-air-handwriting is for login purpose. The users are allowed to practice to write in the air for a few times. Most of them can understand the idea easily and get prepared within a minute. During the data collection, the user can voluntarily abort the writing and the incomplete data is discarded. Only one data glove and one Leap Motion controller are used and the data collection processes with the two devices are separate. All users write with the right hand and wear the glove on that hand. The Leap Motion controller is placed on a table or a side table. Both devices are connected to a laptop as the client machine. For the first dataset, there is no constraint on the content of the string created by the user except distinctiveness. Also there is no constraint on the writing convention. For example, the user can write in various direction, stack every character on the same place, write while standing or sitting, with elbow supported or not supported. Most users write very fast and their writing is illegible, like the way of signing a signature. Since the strings in the first dataset are distinct, we use them either as a passcode for authentication or an ID for identification.

Vi-B Authentication Experiments and Results

For authentication, each individual string is considered as the passcode of an account, and in total there are 210 and 218 accounts with the glove and camera device respectively. We run the following procedures with the 64 window (), 16 local features for each individual classifier (), and 32 classifiers as an ensemblement ().

1) We follow the registration process to create all the accounts, construct and train the classifier for each account. For the five signals for registration, all of them are used to construct the template and train the SVM classifier. Thus, For each account there are five training signals with “true-user” label, and or training signals with “not-true-user” label (i.e., the training signals of other accounts).

2) We use each of the five testing signals of the same account as an authentication request (i.e., the ground truth label of is “true-user”). If a signal from the “true-user” class is misclassified to “not-true-user” class, the result is a False Reject (FR); otherwise it is a True Accept (TA).

3) We use each of the five testing signals of an account as an authentication request to all other accounts (i.e., the ground truth label of is “guessing”). If a signal from the “guessing” class is misclassified to the “true-user” class, the result is defined as a False Accept (FA), otherwise it is a True Reject (TR).

4) We use each of the five spoofing signals in the dataset 2 as an authentication request to the spoofing target account (i.e., the ground truth label of is “spoofing”). If a signal from the “spoofing” class is misclassified to the “true-user” class, the result is defined as a Successful Spoof (SS), which is considered as a special case of FA.

Fig. 8: ROC with the glove device (left) and the camera device (right). “without spoof” means the plot of FAR against FRR, and the line of “spoof only” refers plotting FAR against FRR

The main evaluation metrics are False Reject Rate (FRR) and False Accept Rate (FAR), which are the the portions of false rejects and false accepts in all authentication requests respectively, formally defined as follows:

where is the total number of accounts, is the number of testing authentication requests for each account, is the number of impostors. is the number of FR of the account, is the number of FA for account using the signals of account as the authentication requests, and is the number of successful spoof for the account by the impostor. Equal Error Rate (EER) is defined as the rate where FRR is equal to FAR. FAR10K and FAR100K denoets the FRR when FAR is and , respectively. ZeroFAR denotes the FRR when FAR is zero. We varying the decision threshold to change the amount of FR and FA, and results are shown in the Receiver Operating Characteristic (ROC) curve in Fig. 8. The results are shown in Table I, with comparison to plain DTW method on the same dataset with the same signal preprocessing techniques using the same template. These evaluation metrics are widely used in traditional biometric authentication systems such as fingerprint [29]. For comparison, we mainly use the EER since it is a single number that captures the general performance, and for practical usage, FAR10K is more important.

Compared with plain DTW algorithm which is used in many related works, our method has a significant performance improvement. We believe DTW is a good alignment algorithm but not necessarily a good matching algorithm. Our method treats different local distance with different importance and considers different segments of signal individually, while DTW uses a sum of element-wise distance that may propagate locally miss matched sample pair to the final result even for signals in the “true-user” class. Besides, DTW treats every segment of the signal equally, while some segments can be more distinctive for different strings, e.g., the second half is more important for the signals generated by writing “PASSCODE” and “PASSWORD”. In general the data obtained by the glove device has higher consistency because it does not restrict the user to write in a certain area. Also the signal quality is better with the glove device, while signals with the camera device can contain missing samples or wrongly estimated hand postures. Hence, our prototype with the glove device has better performance.

Metric EER
Ours (glove) 0.1% 2.2% 0.4% 0.9% 1.0%
DTW (glove) 0.6% 3.9% 3.0% 9.2% 13.1%
Ours (camera) 0.5% 4.4% 1.9% 4.6% 5.6%
DTW (camera) 2.1% 7.7% 15.9% 28.5% 30.8%
TABLE I: Empirical results of authentication

Vi-C Identification Experiments and Results

Fig. 9: Identification performance with the glove device (left) and the camera device (right). The annotation shows results with identity verification.

In the identification experiments, each individual string is considered as the ID associated to an account. For the five signals at registration, we augment them to 125 signals to train the CNN. The activation function we use in the convolutional layer and fully-connected layers is a rectified linear unit (ReLU). The whole network is trained for 20 epochs with Adam algorithm

[30] with a learning rate of 0.001. We use a combination of cross-entropy loss and the center loss [31] as the optimization target. The experiment procedures are as follows.

1) First, we follow the registration process to create all the accounts, train the CNN, construct the template and train the SVM ensemblement.

2) Next, we use each of the five testing signals of each account as an identification request and run the identification process. If is from account A and the predicted account number is A, it is a correct identification; otherwise it is an incorrect identification.

3) After that, we use each of the five spoofing signals in dataset 2 as an identification request. If is a spoofing attack targeting account A and the predicted account number is A, it is a successful spoof; otherwise it is an unsuccessful spoof.

We also run these experiments without the identity verification. In this case, if the is a testing signal from account A or a spoofing attack targeting account A, and if the top-k candidate account numbers predicted by the CNN contains A, it is a correct identification or a successful spoof respectively.

It is commonly believed that ID is not a secret and the identity verification should not be too strict to hurt usability. Thus, we choose the threshold of the identity verification with a value that achieves the EER for spoofing only data. We vary from 1 to 7 and show the results in Fig. 9. In general, increasing the number of candidates helps identification accuracy, at a marginal cost of slightly increased spoof success rate. However, if identity verification is skipped, spoof success rate will have a significant increase, which renders the system vulnerable or even useless. The main cause is that the CNN learns features for distinguishing difference stroke combinations instead of distinguishing fine differences in the signal segments of the same stroke. Also in practice, there is no spoofing data available at registration time to train the CNN. Essentially, the CNN also serves as an index for all accounts in the database, which locates the probable accounts given a signal instead of search it exhaustively. With exhaustive search, our prototype system can achieve 99.9% accuracy with the glove device and 99.5% accuracy with the camera device. However, it takes more than one second to exhaustive search on a database containing about 200 accounts. The time consumption is mainly caused by accessing the stored template as well as aligning the signal, and it will grow linearly with the number of accounts in the database. More details on running time are shown in Appendix A.

Fig. 10: Authentication permanence performance with the glove device (left) and the camera device (right). Annotation shows results with update and retrain.
Fig. 11: Identification permanence performance with the glove device (left) and the camera device (right). Annotation shows results with update.

Vi-D Permanence Experiments and Results

We use the third dataset to study the long term performance of our prototype system by running the authentication procedure and the identification procedure for the five signals in each session. The change of authentication acceptance rate is shown in Fig. 10 (with the same setup as that in the authentication experiments described above, and a decision threshold at ). The figure shows slight performance degradation with time. We can update the template at the end of the session to prevent such a degradation for future sessions. The new template is updated as an affine combination of the old template and the new signal, i.e., , where is the update factor set to 0.1 in this experiment. Additionally, we can also both update the template and retrain the SVM classifers with both the old signals and new signals, which can further maintain or even improve performance.

The change of identification accuracy is shown in Fig. 11 (with the same setup as that in the identification experiments described above). The figure shows slight performance degradation with time. Similarly, the accuracy can be improved significantly if the CNN and the SVM classifiers are both retrained with the new signals at the end of the session. We believe that for some users merely 5 signals at registration cannot fully capture the uncertainty of the writing behavior, even with our data augmentation methods. In practice, typing a password can always be employed as a fallback. On the smartphone, if the user does not unlock it immediately with fingerprint or face, the smartphone will ask the user to type a password. If the password passes, it will update the fingerprint or face template accordingly. Such a strategy can also be utilized in our framework since showing a virtual keyboard to type a password can always be an backup option, though it is inconvenient.

Vi-E Comparison to Existing Works

A comparison to existing works which also use in-air-handwriting is shown in the Table II

. The major characteristics that differentiate our framework from them are as follows. First we use a data driven method by designing features and utilizing machine learning models, instead of crafting algorithms calculating a matching distance for authentication. Second, we avoid exhaustively searching and comparing the whole account database for user identification. Third, we evaluate performance of our framework under active spoofing attacks and with a time span of near a month, which is usually omitted by existing works. Fourth, our system has a significant performance improvement on a dataset with reasonable size. A more complete list and comparison of related work in this area is provided in Appendix C.

Vi-F Comparison to Password and Other Biometrics

In Table III we present a comparison of FMCode with password and biometrics based system. The results shown here are obtained from different publications with various datasets, which merely show limited information as an intuition about their performance, instead of the performance with serious and strongly supervised evaluation. First we show the classification accuracy in terms of EER. Here FMCode is comparable to fingerprint (on FVC2004 [29] among all the datasets), face, iris, and signature. Comparing to biometrics, a considerable portion of discrimination capability comes from the large content space of the in-air-handwriting. Next we show the equivalent key space in number of bits. For password used in device unlock, the commonly used 4-digit password (default setting on most smartphone) are considered, and for biometrics, the equivalent key space is defined by [32]. For FMCode we calculate key space with the corresponding FAR by setting the decision threshold at a place where the true user has 95% successful login rate (i.e., 5% FRR). The results show that FMCode is also comparable to password based login and authentication system. Due to the limited amount of data, we cannot achieve an FAR resolution lower than , i.e., more than 17.6 bit key space. For the glove device, at the 5% FRR decision threshold, the FAR is already 0 but we can only conclude that the equivalent key space is larger than 17.6 bits. In practice, recommended password key space is between 28 to 128 bits [33] while web users typically choose passwords with 20 to 90 bits key space and on average it is 40 to 41 bits [34]. However such large key space is underutilized because it is well known that people are not good at creating and memorizing strong password, and the actually entropy is much less than the whole key space [35]. Moreover, since password must contain letters that can be typed on keyboard, efficient password guessing strategies such as dictionary attack further weaken the calculated password quality in number of bits. A more detailed analysis on the comparison of usability, deployability, and security with password and biometrics is provided in Appendix B.

(w/o spoof)
(w/ spoof)
Device Algorithm
FMCode (glove) 105 (210) 0.1% 2.2% 96.7% (99.9%) data glove SVM / CNN
FMCode (camera) 109 (218) 0.5% 4.4% 94.3% (99.5%) Leap Motion SVM / CNN
Liu et al.[6] 20 25 3% 10% 88 98.4% Wii remote DTW
Bailador et al.[5] 96 1.8% 2.1% 5% N/A smartphone DTW, Bayes, HMM
Bashir et al.[36] 40 1.8% N/A 98.5% custom digital pen DTW
Chan et al.[9] 16 0.8% N/A 98% Leap Motion random forest
Tian et al.[11] 18 2% N/A N/A Kinect DTW
TABLE II: Comparison to existing works.
(online login)
(device unlock)
Fingerprint Face Iris Signature
EER (w/o spoof) 0.1% 0.5% N/A N/A 0.28%2% [29] 2.6%8.6% [37] 0.11% [32] 1%3% [38]
Key Space (bits) 17.6 16 2090 [34] 13.3 13.3 [32] N/A 19.9 [32] N/A
TABLE III: Comparison to password and biometrics.

Vii User Evaluation

We investigated the usability of our framework by taking questionnaire from 30 users with the experience of both the glove device and the camera device. First, the users evaluate various aspects of our in-air-handwriting based login framework with a score from 1 (strongly disagree) to 5 (strongly agree). The results are shown in Fig. 12.

Second, we ask the user to compare our framework with the password based systems and biometrics including fingerprint and face, on the easiness of usage, login speed and security. The user has three options: (a) our framework is better, (b) they are the same or difficult to decide which one is better or worse, (c) our framework is worse. The results are shown in Fig. 13. We can see that the user has a mixed attitude on the usability compared to traditional password, but clearly FMCode can not compete biometrics like fingerprint and face in usability. However, the majority of the users feel that FMCode is more secure than traditional password, and more than half of them feel it is more secure than fingerprint and face.

Third, we ask the following questions:

1) Compared to password, our framework fuses handwriting convention. Is this characteristic important?

2) Compared to biometrics, our framework allows change and revoke the in-air-handwriting passcode, which is unlinked to personal identity. Is this characteristic important?

Among the surveyed users, 89% and 82% of them answer “important” for the first and second characteristics respectively. Combined with the previous results, we can conclude that FMCode does not intend to replace existing password-based solution or biometrics. Instead, due to its unique characteristics that passwords and biometrics lack, FMCode is suitable in scenarios where such characteristics matter and where passwords and biometrics are not applicable, for example, login over gesture interface on VR headset or in operating theater.

At last, we ask the user which type of device is preferred between a wearable device and a contactless device for hand motion tracking. 21% of the users choose the wearable device and the other 79% choose the contactless device.

Viii Discussion

Viii-a Other Attacks

There are other potential attacks on our authentication system if some of our security assumptions do not hold. For example, the attacker may be able to access the server’s storage to steal the templates. Due to the inherent fuzziness in the authentication step, traditional hash is not able to protect the templates like the hashed password. One possible solution that we are working on is to adapt the deep CNN to generate a fuzzy hash and further a set of keys for each account using the signals at registration. The templates can be encrypted by each of the key to form a pool. At login time, the same neural network is applied on the signal in the login request to obtain a fuzzy hash and a key. It is highly possible that this key can decrypt an encrypted templates in the pool, if the signal is generated by the legitimate user performing the correct FMCode. After the template decryption, the authentication procedure can continue. If the templates can be successfully decrypted by the generated key from the signal in the login request, we can not conclude that the user is authenticated. As we have shown in the identification results, the ID is easier to spoof without a verification step, thus an impostor with the knowledge of the ID and the passcode may also be able to derive an acceptable key. However, the attacker that steals the templates does not have such knowledge.

Another possible threat is man-in-the-middle attacks, where the attacker intercepts and records the messages between the server and the client. We believe this is not critical because machine-to-machine authentication and message encryption can be used first to allow the client and the server to communicate securely. Existing technologies such as SSL/TLS and public key infrastructure (PKI) over the Internet can be leveraged to build a secure communication channel before user login, similar to the case of password-based login on most websites over the Internet nowadays.

On the user side, record and replay attacks must also be handled. The authentication request require to present a multi-dimensional signal about the physical states of the hand during the writing. In a preliminary experiment, we found that it is difficult to synthesis such a signal from a video recorded by monolens camera or depth camera placed 1 meter away from the user. The main reason is the drastic spatial resolution drop with distance, which cause limited hand skeleton estimation accuracy. A seconary reason is the posture uncertainty of the user and the motion blur due to insufficient frame rate of the camera. If the camera is close to the user, the user might get alerted, just like if someone within a proximity is watching a user typing his or her password, the user will be alerted and stop typing. More details about this type of attack will be presented in future work.

Ix Related Works

Traditional user authentication methods can be based on physiological traits (e.g., biometrics, like fingerprint and iris), knowledge (e.g., password), and possession (e.g., a card or a token). Unfortunately, none of them are perfect because physiological traits can be copied (e.g. iris spoofing by taking eye pictures), knowledge can be forgotten (e.g. forgotten password), and possession can be lost or faked (e.g. stolen or copied card). Currently, password and biometrics are the most widely adopted authentication approaches, but they both have shortcomings. On the password side, remembering more and more passwords is a problem for the user, because a memorable password is usually not strong, while a strong password is difficult to remember [34]. Most users just reuse the same password for many sites [39], and they rarely update it or only update it in certain predictable patterns [40]. As a result, password based system works far from optimal and there are numerous works which have made attempts to replace password [32, 41]. On the biometrics side, although identity verification using fingerprint [29] and iris [42] is convenient, inexpensive and more secure than password in some cases, biometrics are inherently not secret, and they are irrevocable after leakage. Moreover, biometric authentication is directly linked to a person and hence sensitive to privacy issues. FMCode fuses a secret with the handwriting behavior, which has the advantages of both password and biometrics. Besides, FMCode is able to defend spoofing attack, while such attacks are surprisingly effective for biometrics [43].

Early gesture and body motion based authentication methods use the accelerometer to track simple motions like shaking [44, 45, 46], walking [47, 48], or other predefined gesture patterns [49], which proves the feasibility of such methods. In-air-handwriting captured by a smartphone[50, 7, 5], handhold remote control devices [6] or a smartwatch [51] has better potentials because the complex motions contain more information unique to each user. However, the user behavior of writing with such devices different from pen is unnatural and inconsistent. In-air-handwriting with fingertip is also investigated with camera based gesture interface such as Kinect [11, 52, 53] and Leap Motion [10, 54, 9, 55]. The major drawback of such methods is strong constraints on the size, speed, and orientation of a user’s movement, because these cameras are set up at a fixed place. These constraints have a negative influence on both usability and authentication performance, as can be shown by the performance difference of our framework with the two different types of devices. Wearable monolens cameras such as Google Glass [8] would have certain advantages. However the vision understanding and pose estimation of hands in changing environment is difficult. For gesture based authentication, the critical technical challenge is developing good features and classifiers that can tolerate variations and long term behavior changes of the captured motions of the same user as well as distinguishing different users with high accuracy. For gesture based identification, the critical technical challenge is designing efficient way to index each account by the gesture signal or features for fast search.

Handwriting and signature has been studied extensively in the past, but FMCode is fundamentally different from handwriting [56, 57] or hand gesture recognition [58, 59, 60], i.e., the former identifies and verifies the writer while the latter recognizes the content of writing or meaning of the gesture regardless of the writer. Since a recognition system of handwriting or hand gesture tries to group similar motion patterns to the same class, if they are used as login systems, the recognized content is essentially another form of a password, which is vulnerable when the content of the writing is leaked. Besides, our framework is different from online signature verification [38]. We allow the user to draw arbitrary patterns instead of signatures (i.e., names), and we capture 3D motions of fingertip instead of movement of a pen or digital pen with sensors [36, 61] on a 2D surface. Our method targets the application scenarios with gesture interface but lacking tradition input interfaces like keyboard or mouse. For the same reason, we omit comparison to existing 2D free gesture based system on touchscreen [12, 13, 14].

Fig. 12: User evaluation scores.
Fig. 13: User evaluation with comparison.

X Conclusions

In this paper, we present FMCode, a user authentication and identification framework for applications using gesture input interface. FMCode does not completely replace password or biometrics in all use cases, but it can be an alternative way for user login over gesture interface where password and passive biometrics are inconvenient or not applicable. Our prototype system and experimental results show great potential of this method. However, FMCode also has limitations under certain security assumptions, which require further investigation, including methods of direct key generation from in-air-handwriting signals and performance under record and replay attack.


  • [1] “Microsoft Kinect,”, accessed: 2017-11-25.
  • [2] “HTC Vive,”, accessed: 2017-11-25.
  • [3] “Project Soli - Google ATAP,”, accessed: 2017-11-25.
  • [4] “Microsoft HoloLens,”, accessed: 2017-2-1.
  • [5] G. Bailador, C. Sanchez-Avila, J. Guerra-Casanova, and A. de Santos Sierra, “Analysis of pattern recognition techniques for in-air signature biometrics,” Pattern Recognition, vol. 44, no. 10, pp. 2468–2478, 2011.
  • [6] J. Liu, L. Zhong, J. Wickramasuriya, and V. Vasudevan, “uwave: Accelerometer-based personalized gesture recognition and its applications,” Pervasive and Mobile Computing, vol. 5, no. 6, pp. 657–675, 2009.
  • [7] A. Zaharis, A. Martini, P. Kikiras, and G. Stamoulis, “User authentication method and implementation using a three-axis accelerometer,” in International Conference on Mobile Lightweight Wireless Systems.   Springer, 2010, pp. 192–202.
  • [8] H. Sajid and S. C. Sen-ching, “Vsig: Hand-gestured signature recognition and authentication with wearable camera,” in Information Forensics and Security (WIFS), 2015 IEEE International Workshop on.   IEEE, 2015, pp. 1–6.
  • [9] A. Chan, T. Halevi, and N. Memon, “Leap motion controller for authentication via hand geometry and gestures,” in International Conference on Human Aspects of Information Security, Privacy, and Trust.   Springer, 2015.
  • [10] A. Chahar, S. Yadav, I. Nigam, R. Singh, and M. Vatsa, “A leap password based verification system,” in IEEE 7th International Conference on Biometrics Theory, Applications and Systems (BTAS).   IEEE, 2015.
  • [11] J. Tian, C. Qu, W. Xu, and S. Wang, “Kinwrite: Handwriting-based authentication using kinect.” in NDSS, 2013.
  • [12] M. Frank, R. Biedert, E. Ma, I. Martinovic, and D. Song, “Touchalytics: On the applicability of touchscreen input as a behavioral biometric for continuous authentication,” IEEE transactions on information forensics and security, vol. 8, no. 1, pp. 136–148, 2013.
  • [13] N. Z. Gong, M. Payer, R. Moazzezi, and M. Frank, “Forgery-resistant touch-based authentication on mobile devices,” in Proceedings of the 11th ACM on Asia Conference on Computer and Communications Security.   ACM, 2016, pp. 499–510.
  • [14] Y. Yang, G. D. Clark, J. Lindqvist, and A. Oulasvirta, “Free-form gesture authentication in the wild,” in Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems.   ACM, 2016, pp. 3722–3735.
  • [15] R. Biddle, S. Chiasson, and P. C. Van Oorschot, “Graphical passwords: Learning from the first twelve years,” ACM Computing Surveys (CSUR), vol. 44, no. 4, p. 19, 2012.
  • [16] D. Lu and D. Huang, “Fmhash: Deep hashing of in-air-handwriting for user identification,” arXiv preprint arXiv:1806.03574, 2018.
  • [17] Y. Itaguchi, C. Yamada, and K. Fukuzawa, “Writing in the air: Contributions of finger movement to cognitive processing,” PloS one, vol. 10, no. 6, p. e0128419, 2015.
  • [18] S. R. Klemmer, B. Hartmann, and L. Takayama, “How bodies matter: five themes for interaction design,” in Proceedings of the 6th conference on Designing Interactive systems.   ACM, 2006, pp. 140–149.
  • [19] E. M. Diaz, F. de Ponte Müller, A. R. Jiménez, and F. Zampella, “Evaluation of ahrs algorithms for inertial personal localization in industrial environments,” in Industrial Technology (ICIT), 2015 IEEE International Conference on.   IEEE, 2015, pp. 3412–3417.
  • [20]

    A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in

    Advances in neural information processing systems, 2012, pp. 1097–1105.
  • [21] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in

    Proceedings of the IEEE conference on computer vision and pattern recognition

    , 2016, pp. 770–778.
  • [22] F. Chollet, “Xception: Deep learning with depthwise separable convolutions,” arXiv preprint arXiv:1610.02357, 2016.
  • [23] N. Srivastava, G. E. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: a simple way to prevent neural networks from overfitting.” Journal of Machine Learning Research, vol. 15, no. 1, pp. 1929–1958, 2014.
  • [24] “Hi5 VR Glove,”, accessed: 2017-11-25.
  • [25] “Talon, the smart ring,”, accessed: 2017-11-25.
  • [26] “Openpose: Real-time multi-person keypoint detection library for body, face, and hands,” URl:
  • [27] D. Lu, K. Xu, and D. Huang, “A data driven in-air-handwriting biometric authentication system,” in International Joint Conf. on Biometrics (IJCB 2017).   IEEE, 2017.
  • [28] D. Lu, D. Huang, Y. Deng, and A. Alshamrani, “Multifactor user authentication with in-air-handwriting and hand geometry,” in 2018 International Conference on Biometrics (ICB).   IEEE, 2018.
  • [29] R. Cappelli, D. Maio, D. Maltoni, J. L. Wayman, and A. K. Jain, “Performance evaluation of fingerprint verification systems,” IEEE transactions on pattern analysis and machine intelligence, vol. 28, no. 1, pp. 3–18, 2006.
  • [30] D. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014.
  • [31]

    Y. Wen, K. Zhang, Z. Li, and Y. Qiao, “A discriminative feature learning approach for deep face recognition,” in

    European Conference on Computer Vision.   Springer, 2016, pp. 499–515.
  • [32] L. O’Gorman, “Comparing passwords, tokens, and biometrics for user authentication,” Proceedings of the IEEE, vol. 91, no. 12, pp. 2021–2040, 2003.
  • [33] S. Crocker and J. Schiller, “Rfc 4086-randomness requirements for security,” 2011.
  • [34] D. Florencio and C. Herley, “A large-scale study of web password habits,” in Proceedings of the 16th international conference on World Wide Web.   ACM, 2007, pp. 657–666.
  • [35] W. Ma, J. Campbell, D. Tran, and D. Kleeman, “Password entropy and password quality,” in Network and System Security (NSS), 2010 4th International Conference on.   IEEE, 2010, pp. 583–587.
  • [36] M. Bashir and J. Kempf, “Person authentication with rdtw based on handwritten pin and signature with a novel biometric smart pen device,” in Computational Intelligence in Biometrics: Theory, Algorithms, and Applications, 2009. CIB 2009. IEEE Workshop on.   IEEE, 2009, pp. 63–68.
  • [37] O. M. Parkhi, A. Vedaldi, A. Zisserman et al., “Deep face recognition.” in BMVC, vol. 1, no. 3, 2015, p. 6.
  • [38] D. Impedovo and G. Pirlo, “Automatic signature verification: the state of the art,” IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), vol. 38, no. 5, pp. 609–635, 2008.
  • [39] B. Ives, K. R. Walsh, and H. Schneider, “The domino effect of password reuse,” Communications of the ACM, vol. 47, no. 4, pp. 75–78, 2004.
  • [40] Y. Zhang, F. Monrose, and M. K. Reiter, “The security of modern password expiration: An algorithmic framework and empirical analysis,” in Proceedings of the 17th ACM conference on Computer and communications security.   ACM, 2010, pp. 176–186.
  • [41] J. Bonneau, C. Herley, P. C. Van Oorschot, and F. Stajano, “The quest to replace passwords: A framework for comparative evaluation of web authentication schemes,” in 2012 IEEE Symposium on Security and Privacy.   IEEE, 2012.
  • [42] J. Daugman, “New methods in iris recognition,” IEEE Transactions on Systems, Man, and Cybernetics, Part B, vol. 37, no. 5, pp. 1167–1175, 2007.
  • [43] “Chaos computer club breaks apple touchid,”, accessed: 2016-10-8.
  • [44] S. N. Patel, J. S. Pierce, and G. D. Abowd, “A gesture-based authentication scheme for untrusted public terminals,” in Proceedings of the 17th annual ACM symposium on User interface software and technology.   ACM, 2004.
  • [45] F. Okumura, A. Kubota, Y. Hatori, K. Matsuo, M. Hashimoto, and A. Koike, “A study on biometric authentication based on arm sweep action with acceleration sensor,” in 2006 International Symposium on Intelligent Signal Processing and Communications.   IEEE, 2006, pp. 219–222.
  • [46] R. Mayrhofer and H. Gellersen, “Shake well before use: Authentication based on accelerometer data,” in International Conference on Pervasive Computing.   Springer, 2007, pp. 144–161.
  • [47] D. Gafurov and E. Snekkkenes, “Arm swing as a weak biometric for unobtrusive user authentication,” in Intelligent Information Hiding and Multimedia Signal Processing, 2008. IIHMSP’08 International Conference on.   IEEE, 2008, pp. 1080–1087.
  • [48] J. Lester, B. Hannaford, and G. Borriello, ““are you with me?”–using accelerometers to determine if two devices are carried by the same person,” in International Conference on Pervasive Computing.   Springer, 2004.
  • [49] E. Farella, S. O’Modhrain, L. Benini, and B. Riccó, “Gesture signature for ambient intelligence applications: a feasibility study,” in International Conference on Pervasive Computing.   Springer, 2006, pp. 288–304.
  • [50] F. Hong, M. Wei, S. You, Y. Feng, and Z. Guo, “Waving authentication: your smartphone authenticate you on motion gesture,” in Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems.   ACM, 2015, pp. 263–266.
  • [51] B. Nassi, A. Levy, Y. Elovici, and E. Shmueli, “Handwritten signature verification using hand-worn devices,” arXiv preprint arXiv:1612.06305, 2016.
  • [52] J. Wu, J. Konrad, and P. Ishwar, “Dynamic time warping for gesture-based user identification and authentication with kinect,” in 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.   IEEE, 2013.
  • [53] E. Hayashi, M. Maas, and J. I. Hong, “Wave to me: user identification using body lengths and natural gestures,” in Proceedings of the 32nd annual ACM conference on Human factors in computing systems.   ACM, 2014, pp. 3453–3462.
  • [54] I. Aslan, A. Uhl, A. Meschtscherjakov, and M. Tscheligi, “Mid-air authentication gestures: an exploration of authentication based on palm and finger motions,” in Proceedings of the 16th International Conference on Multimodal Interaction.   ACM, 2014, pp. 311–318.
  • [55] M. Piekarczyk and M. R. Ogiela, “On using palm and finger movements as a gesture-based biometrics,” in Intelligent Networking and Collaborative Systems (INCOS), 2015 International Conference on.   IEEE, 2015, pp. 211–216.
  • [56] C. Amma, M. Georgi, and T. Schultz, “Airwriting: a wearable handwriting recognition system,” Personal and ubiquitous computing, vol. 18, no. 1, 2014.
  • [57] C. Qu, D. Zhang, and J. Tian, “Online kinect handwritten digit recognition based on dynamic time warping and support vector machine,” Journal Of Information &Computational Science, vol. 12, no. 1, 2015.
  • [58] D. Moazen, S. A. Sajjadi, and A. Nahapetian, “Airdraw: Leveraging smart watch motion sensors for mobile human computer interactions,” in Consumer Communications & Networking Conference (CCNC), 13th IEEE Annual.   IEEE, 2016.
  • [59] H. Wen, J. Ramos Rojas, and A. K. Dey, “Serendipity: Finger gesture recognition using an off-the-shelf smartwatch,” in Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems.   ACM, 2016, pp. 3847–3851.
  • [60] Y. Zhang and C. Harrison, “Tomo: Wearable, low-cost electrical impedance tomography for hand gesture recognition,” in Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology.   ACM, 2015.
  • [61] R. Renuka, V. Suganya, and A. Kumar, “Online hand written character recognition using digital pen for static authentication,” in Computer Communication and Informatics (ICCCI), 2014 International Conference on.   IEEE, 2014, pp. 1–5.
  • [62] M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard et al.

    , “Tensorflow: A system for large-scale machine learning,” in

    Proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI). Savannah, Georgia, USA, 2016.
  • [63] G. D. Clark and J. Lindqvist, “Engineering gesture-based authentication systems,” IEEE Pervasive Computing, vol. 14, no. 1, pp. 18–25, 2015.
  • [64] S. Lee, K. Song, and J. Choi, “Access to an automated security system using gesture-based passwords,” in 2012 15th International Conference on Network-Based Information Systems.   IEEE, 2012, pp. 760–765.


X-a Cost of Computing and Storage

We implemented our prototype system in Python with sklearn library and TensorFlow [62]. Preprocessing a single signal cost about 25 ms for the glove device (excluding the state estimation step since it is running on the device instead of the client machine) and 100 ms for the camera device, where filtering is the bottleneck. In authentication, generate template cost 2.1 ms and training the SVM cost 30 ms for each account; classification of each signal cost less than 1 ms, which is negligible compared to the time for writing the string. The time consumption measured here does not contain loading the data and model from disk to the main memory, because the disk speed varies significantly due to the page cache in the main memory and our dataset is small enough to be fully fit in the cache, which is quite different when used in real world scenarios. This measurement is conducted with a single threaded program on a workstation with Intel Xeon E3-1230 CPU (quad-core 3.2GHz, 8M cache) and 32 GB main memory. For identification, training the deep CNN for 20 epochs requires around 7 minutes using only CPU, while with a powerful GPU (Nvida GTX 1080 Ti in our case) this can drastically decrease to around 30 seconds; classification using the CNN costs less than 1 ms with only CPU. The space cost for template storage and the amount of data needed to be transferred between the server and the client is proportional to the signal length. If each sensor axis of a sample is represented in single precision floating point number (four bytes), the average space cost of a signal is 8.6 KB with our datasets for both devices (they both output data with 9 sensor axis). If all parameters are represented in single precision floating point number, storing the SVM classifiers costs 79 KB per account on average, and storing the CNN itself requires around 4 MB because of the 1 million weights and biases parameters.

X-B Comparison to Password and Biometrics in Usability, Deployability, Security

Usability FMCode Deployability FMCode Security FMCode
Memory effortless Maybe (+) Accessible Yes Resilient to physical observation Yes (+)
Scalable for users Maybe (+) Scalable Yes Resilient to targeted impersonation Maybe (+)
Nothing to carry Yes Server compatible Maybe (-) Resilient to throttled guessing Maybe
Physically effortless No Browser compatible Maybe (-) Resilient to unthrottled guessing No
Easy to learn Yes Mature No (-) Resilient to theft Yes (+)
Efficient to use Yes Non-proprietary Yes No trusted third party required Yes
Infrequent errors Maybe (-) Configurable[63] Yes Requiring explicit consent Yes
Easy to recovery Yes Developer friendly[63] Yes Unlinkable Yes
TABLE IV: Usability, Deployability, and Security Evaluation of FMCode.
Related Works
# of
Patel et al.[44] NA
Cellphone w/
NA NA NA shake
Okumura et al.[45] 12 22
Cellphone w/
6 weeks 4% NA shake DTW
Mayrhofer et al.[46] 51
Custom device
w/ accelerometer
NA shake
Farella et al.[49] 5 10
PDA w/
4 specified

+ kNN

Lester et al.[48] 6
Custom device
w/ accelerometer
NA NA 100% walk
Gafurov et al.[47] 30
Custom device
w/ accelerometer
NA 10% NA walk
Liu et al.[6] 20 25
Nintendo Wii
1 4 weeks
free writing DTW
Zaharis et al.[7] 4
Nintendo Wii
3 weeks NA 98.20% free writing
feature matching
Casanova et al.[5] 96 Smartphone 20 days 2.5% NA free writing
DTW, Bayes,
Lee et al.[64] 15 Smartphone NA NA 88.40% tap, flip, etc. decision tree
Bashir et al.[36] 10 40 Custom pen NA 1.8% 98.50%
alphabetic or
free writing
Renuka et al.[61] 12 Custom pen NA NA 95%
alphabetic or
free writing
Aslan, et al.[54] 13 Leap Motion NA 10% NA
2 specified
Nigam et al.[10] 150 Leap Motion NA NA 81% free writing
statistical feature
Chan et al.[9] 16 Leap Motion NA 0.80% >99% free writing random forest
Piekarczyk et al.[55] 4 5 Leap Motion NA NA
4 specified
+ kNN / LSH
Tian et al.[11] 18 Kinect 5 months 2% NA free writing DTW
Sajid et al.[8] 10 Google Glass NA NA 97.50% free writing
Clustering + DTW
Wu et al.[52] 20 Kinect NA 1.89% NA 8 gestures DTW
Hayashi et al.[53] 36 Kinect 2 weeks
NA hand waving SVM
TABLE V: Comparison of experimental results and algorithms of related works.

We evaluated the usability, deployability and security of FMCode using the criteria provided by [41], and the result is shown in Table IV. We added two aspects in deployability, “configurable” and “developer friendly” mentioned in [63]. Each usability, deployability and security item is evaluated by whether our FMCode method possess the characteristics, and a plus / minus sign means that our method is better / worse than password. We explain a few items here but the readers are recommended to refer to [41] and [63] for the definition of these characteristics. In general, compared with password and biometrics, FMCode achieves nearly all their usability and deployability characteristics, and it contains the security advantages from both password and biometrics.

On the usability side, nothing to carry means the user does not to present a physical item such as a card. Though our prototype system uses a glove, the glove is treated as a general gesture input device, just like password does not require the user to bring a keyboard everywhere. Arguably, FMCode has less memory burden because in section 6 we shows the discrimination capability comes from 3 layers of information instead of just the combination of characters in password, so memorable FMCode is not necessarily weak any more. However, there might be potentially more frequent rejection of legitimate user login than password because the internal fluctuation in the stability of finger motion.

On the deployability side, since the server only needs to store the template as a secret, FMCode is similar as those behavior biometrics without using special devices (e.g., keyboard or mouse dynamics), thus it can be compatible with most server with small software modification. FMCode can be supported by browsers on devices with gesture input interface, where the FMCode input of a user can be treated as a very long password and sent to the server over the Internet, similar as the case of logining in a website using password.

On the security side, FMCode can withstand spoof under semantic or visual disclosure and targeted impersonation to a certain extent, which is more like biometrics than password. But unlike passive biometrics, FMCode is changable and more difficult to steal. For example, an attacker can collect a user’s fingerprint from a cup after the user has touched it, but FMCode can only be recorded when the user performs it. FMCode suffers from server storage leakage and internal observer if the template is not properly protected, and such proper template protection might be difficult because of fussiness in alignment and matching. Also it shares all other security deficiencies of password and biometrics under attacks such as brute-force guessing/collision, dictionary guessing, phishing and cross-site leakage.

Compared with traditional password, FMCode uses handwriting, which makes it a behavior biometrics, and this provides certain protection under semantic or visual disclosure of the passcode. On the other side, compared with unchangeable biometrics like fingerprint, FMCode keeps most of the advantages of a password such as revocability and privacy preserving. This also allows one user to have more than one FMCode, different from traditional behavior biometrics like gait or voice in active speaker recognition. The most similar authentication techniques would be online signature verification (technically speaking, handwriting verification), or graphical password with stroke dynamics, but most of they assume a 2D pressure sensitive touchscreen instead of writing in the air.

X-C Comparison of Related Works

A comparison of related works is shown in Table V.