The task of animating a virtual puppet in real-time using live footage of a human is a well studied one. Broadly speaking, one can classify these techniques by their choice of input data (monocular video, multi-view, depth images) and the methodology (direct optimization, prediction using neural networks, heuristics). For instance, Ichimet al.  use monocular videos with predefined camera movements in order to obtain dense registration of person specific facial features and create a dynamic face model on the fly via optimization. Wu et al.  leverage multi-view data to align a 3D Face Morphable Model (3DMM ) using a bundle of neural networks and produce person specific blend shapes. We refer the reader to  for a review of related work on 3D face alignment and blend shape computation. In this paper, we focus on puppeteering on mobile devices, without the use of extra sensors or a person-specific calibration step.
Despite the success of these techniques, the resulting avatars tend to lack a certain liveliness or expressivity because they do not track the position of the pupils. For instance, prior approach  leverage 3DMM which does not have the pupils in its internal representation. We address this problem using a two stage pipeline that combines a neural network for predicting the position of the pupils (Section 2) and a displacement-based algorithm for estimating the pupil blend shapes (Section 2). We build this pipeline on top of a state-of-the-art face mesh prediction model , but our approach generalizes to other face meshes.
Our network detects 5 points of the pupil, outer iris circle, and eye contour for each eye. Based on the position of these points, we apply carefully devised heuristics to obtain blend shape coefficients in the range where 1 and -1 represent full blend shape activation (e.g. the eye looks up or down respectively) with 0 being the neutral position (the eye looks frontally). We follow this with post-processing to reduce jitter from the detection stage and make the final rendering smooth and appealing.
Our approach only requires a single frame at a time and does not rely on any additional sensors such as a depth camera. Figure 1 shows an example of a virtual puppet animated with our technique.
2 Neural network based eye landmarks
We start with a modern face mesh estimation pipeline that predicts a 468 vertex mesh for the human face . We then compute the bounding boxes for eye regions and pass the corresponding cropped image regions to a smaller landmark regression network that produces additional higher quality landmarks.
Specifically, we extract the corresponding region ( pixels) via cropping from the center of eye landmark of the face mesh estimator. This cropped region is fed into a tiny neural network that has a structure similar to that described in Bazarevsky et al. . This subsequent network predicts 5 locations in 2D (pupil center, 4 points of outer iris circle, and 16 points of eye contour) in the coordinate system of the image starting from the upper left corner. We combine the corresponding landmarks (16 points of eye contour) from the face estimation pipeline with those from the eye refinement network by replacing the coordinates of the former while leaving untouched. We extend the face mesh with 5 pupil landmarks (pupil center and 4 points of outer iris circle), with their coordinate set to the average of the coordinate of the eye corners. The final refined facial mesh contains 478 (468 landmarks + 5 left pupil landmarks + 5 right pupil landmarks) vertices and is used in the second stage of this pipeline.
To execute the neural network on mobile devices, we employ TensorFlow Lite with GPU backend coupled with MediaPipe  — a framework for building perception pipelines.
2.1 Model architecture
The neural network for predicting eye and iris landmarks contains a number of bottlenecks similar to recent work of Tan et al.  and Bazarevsky et al. . The model ends with a fully connected layer that outputs a tensor corresponding to coordinates for each landmark (defined in the cropped image coordinate system). This design allows the network to learn a rich feature representation that achieves low error rates per landmarks (see Section 5). Further, the proposed model architecture has a small memory foot print (due to the small input resolution) and low number of FLOPs (due to the compression and depthwise convolutions). This enables real-time performance of the network on CPU and super real-time performance using GPU capabilities on modern phones. The run-time measurements and error rates are shown in the Table 1.
3 Displacement-based pupil blend shape estimation
We use the refined mesh to predict 4 blend shapes for the pupils: pupils pointing outwards, inwards, upwards and downwards respectively. We compute the activation of these shapes using a simple yet powerful displacement based approach. Specifically, for every blend shape we choose a pair of vertices on the refined mesh that robustly captures the blend shape i.e. for the pupil pointing inwards, we use the vertex of the pupil and the vertex of eye corner. Next, we measure the displacement between these two vertices and compare it to two empirically derived displacements — the displacement with the minimum activation of the blend shape, and — the displacement measured using the maximum activation of the blend shape. Based on this comparison, we obtain a scalar value in the range of for each pupil blend shape.
Next, we merge pairs of opposite blend shapes into two aggregate blend shapes. Finally, we apply smoothing and couple the estimated blend shape values between both eyes.
An overview of the pipeline is presented on the Figure 2.
3.1 Real-time heuristics calibration
Heuristics play a vital part in the proposed blend shape pipeline. Algorithm described in the Section 2 requires two displacements: and to be defined. The initial displacements are empirically estimated based on the representative face mesh dataset. However, these initial values are unable to model all person-specific variations. For a visual reference, Figure 3 shows the variation in displacements over time for different subjects (drawn in different colors) and initial estimated value (drawn in black). The solid line represents measurements of the actual displacements on per frame basis while the dotted line indicates the smoothed trend for a specific subject.
To address the challenge of person-specific displacements and to make the system reliable, we propose to enhance the displacement estimation with a real-time calibration step. We employ the standard score 
calculation algorithm with a few modifications. The main idea of the filter is to check the displacement on every iteration and add it to a circular buffer of the trusted displacements if it falls within the specified confidence interval. Consequently, the calibrated displacement is calculated as an average of the trusted displacements. The standard deviation of these trusted displacements is used as the confidence interval in the next iteration. Details of the Standard Score Filter algorithm are presented inAlgorithm 1.
4 Datasets and training
In order to train the neural network to infer 2D positions of points around the eye, we use
manually annotated images from a globally sourced dataset. We applied a set of augmentations to these images such as affine (rotation, flip) and color transformations (hue, saturation, non-linear mapping, realistic camera noise injection). The network was trained for 250 epochs using the Adam optimizer. Similar to 
, we use the Mean Squared Distance normalized by the Inter-Eye Distance (MSE IED) as our loss function. This normalization avoids factoring in the scale of the eyes.
To quantitatively estimate the accuracy of the trained model, we use manually annotated images. The trained model achieved 7.16% MAD IED (Mean Absolute Distance normalized by the Inter-Eye Distance) on the collected dataset. The baseline error of the manual annotation is 5.73% for simple use cases (the face on the image is frontally rotated) and 7.04% - for hard cases. The error was measured on the same images annotated by different subjects. The inference speeds of the face mesh as well as eye refinement models on a number of phones are shown in the Table 1.
|Phone||Inference speed (ms)|
|Face mesh||Eye refinement|
|Pixel 2 XL||12||20||8|
|Pixel 3 XL||10||12||5|
We present a novel pipeline for real-time pupil tracking from live video on mobile devices at real-time speeds. The approach defines a full end-to-end pipeline for pupil blend shapes estimation from monocular images without any pre-calibration and can be combined with any existing blend shape implementation. It can be used as out-of-the box solution for accurate control of the eye movements for a virtual puppet.
BlazeFace: sub-millisecond neural face detection on mobile gpus. arXiv preprint arXiv:1907.05047. Cited by: §2.1, §2.
-  (1999) A morphable model for the synthesis of 3d faces.. In Siggraph, Vol. 99, pp. 187–194. Cited by: §1.
-  (2015) Dynamic 3d avatar creation from hand-held video input. ACM Transactions on Graphics (ToG) 34 (4), pp. 45. Cited by: §1.
-  (2019) Real-time facial surface geometry from monocular video on mobile gpus. arXiv preprint arXiv:1907.06724. Cited by: §1, §2, §4.
-  (2014) Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980. Cited by: §4.
-  (2019) On-device neural net inference with mobile gpus. arXiv preprint arXiv:1907.01989. Cited by: §2.
-  (2019) MediaPipe: a framework for building perception pipelines. arXiv preprint arXiv:1906.08172. Cited by: §2.
-  (2019) Mnasnet: platform-aware neural architecture search for mobile. In , pp. 2820–2828. Cited by: §2.1.
-  (2016) Face2face: real-time face capture and reenactment of rgb videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2387–2395. Cited by: §1.
-  (2019) MVF-net: multi-view 3d face morphable model regression. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 959–968. Cited by: §1.
-  (2017) Case study research and applications: design and methods. Sage publications. Cited by: §3.1.