Dynamic Drawing Guidance via Electromagnetic Haptic Feedback

06/27/2019 ∙ by Thomas Langerak, et al. ∙ 0

We propose a system to deliver dynamic guidance in drawing, sketching and handwriting tasks via an electromagnet moving underneath a high refresh rate pressure sensitive tablet. The system allows the user to move the pen at their own pace and style and does not take away control. The system continously and iteratively measures the pen motion and adjusts magnet position and power according to the user input in real-time via a receding horizon optimal control formulation. The optimization is based on a novel approximate electromagnet model that is fast enough for use in real-time methods, yet provides very good fit to experimental data. Using a closed-loop time-free approach allows for error-correcting behavior, gently pulling the user back to the desired trajectory rather than pushing or pulling the pen to a continuously advancing setpoint. Our experimental results show that the system can control the pen position with a very low dispersion of 2.8mm (+/-0.8mm). An initial user study indicates that it significantly increases accuracy of users drawing a variety of shapes and that this improvement increases with complexity of the shape.



There are no comments yet.


page 1

page 2

page 8

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1. Introduction

Sketches and handwritten text have been a primary form of communication for centuries. Given their importance for society at large and the arts and design in particular, it is not surprising that a large number of digital pen interfaces exist that aim to combine the expressiveness and flexibility of pens and pencils with the advantages of digital representations via standalone tablets (e.g., Apple iPad, Microsoft Surface) or specialized digitizers (e.g., Wacom). Several sketch (Xing et al., 2015; Simo-Serra et al., 2016; Limpaecher et al., 2013; Su et al., 2014) and digital ink (Aksan et al., 2018; Zitnick, 2013) beautification approaches exist. However, these typically improve results a-posteriori and hence do not provide real-time haptic feedback. Comparatively little attention has been devoted to improving sketching in-situ.

We propose a haptic feedback system (Fig. 2) to provide variable strength guidance onto the tip of a minimally instrumented ballpoint pen. The feedback is delivered via an electromagnet moving on a bi-axial linear stage below a touch and pressure sensitive digital tablet, governed by a closed-loop optimal control algorithm (see Figure 1). Our approach allows users to perceive different levels of feedback while drawing on the tablet. Importantly, the variable strength of force feedback is crucial in allowing user autonomy, while maintaining the ability to provide guidance. We show several use casses where this is desirable such as stylization of drawings.

Previous approaches to pen-based haptic feedback rely on permanent Neodymium magnets (e.g., (Yamaoka and Kakehi, 2013)). However, due to the steep increase in magnetic force as the pen approaches the magnet, this fully controls the pen, removing agency of the user. In contrast we propose a novel optimization scheme, inspired by model predictive contouring control (MPCC) (Lam et al., 2010), to position and regulate an electromagnet such that it provides dynamically adjustable in-plane magnetic forces to the pen tip. This is challenging due to: i) the quadratic increase in magnetic force as a function of magnet-pen distance (i.e. a small spatial offset can significantly increase the perceived force), ii) the fast pen motion compared to the speed of the linear stage, and iii) the hard to predict behavior of the user. These challenges enforce a very tight computational budget from pen motion to magnet actuation, that is, we need an efficient numerical solution to design a system with a low overall latency.

To this end we propose an approximate, yet accurate, model of the electromagnetic force field that can be evaluated analytically and is hence suitable for online control. This model is combined with an MPCC-like optimization scheme that iteratively predicts the pen motion and adjusts magnet position and power accordingly. In contrast to simpler control schemes such as MPC (Faulwasser et al., 2009) or PID (Åström and Hägglund, 1995) control, our approach does not require a timed reference and hence allows users to draw at their desired speed. Furthermore, the optimization scheme allows for error-correcting force feedback, gently pulling the user back to the desired trajectory rather than pushing or pulling the pen to a continuously advancing setpoint on the trajectory.

To assess the proposed feedback mechanism and control algorithm, we performed an user-study with 12 participants. In our experiment we focus on drawing primitives such as circles, spirals, and more complex curves. Our results indicate that the haptic feedback increases both accuracy and precision quantitatively, reduces drift (i.e., error over time), and that users qualitatively appreciate the system. We furthermore, illustrate a number of potential use-cases for the proposed method such as a support tool for hand drawn sketching and writing, or as in-situ feedback tool for inking, under- and overpainting.

To foster adoption of our technique, and to encourage industrial miniaturization of our device, we will release our reference software implementation and hardware blueprints.

Figure 2. Hardware overview.

2. Related Work

Haptics in touch and pen-based interaction is widely researched. We provide a brief overview of the most closely related work, covering sketching, pen-based interfaces, and magnetic actuation.

Sketching Guidance and Beautification

A number of systems have been proposed to provide varying levels of support during sketching, ranging from basic visual guidance to automatic stroke-refinement and vectorization. ShadowDraw guides users by continually updating a shadow underneath their sketch

(Lee et al., 2011). The shadow is generated by first edge-extracting images from a large database, and then analyzing and combining the closest matches to the current sketch. Limpaecher et al. (2013) improve sketches by replacing strokes automatically by strokes extracted from drawings stemming from a large database. The authors report that this reduces undo and other correction operations. When tracing images using a technique called underpainting, Su et al. (2014) leverage the underlying image to perform stroke-refinement in real-time to better fit the gradient information underneath. They perform optimization both locally on a single stroke, and semi and fully globally by considering the interaction between multiple strokes when no close matches are found. Xing et al. (2015) perform global similarity analysis across multiple frames to assist during animation tasks and to suggest strokes that beautify single frames based on past frames.

Inking refers to the process of simplifying sketches and line drawings for improved clarity. Several approaches leverage vectorization (Hilaire and Tombre, 2006; Favreau et al., 2016), stroke aggregation (Liu et al., 2018)

, and more recently, convolutional neural networks

(Simo-Serra et al., 2016; Simo-Serra et al., 2018b) and generative adversarial networks (Simo-Serra et al., 2018a) for this task. Our system is orthogonal and complementary to these approaches, providing active physical support during sketching. We argue that our system is a prime candidate to be combined with such existing systems to re-enforce the intended or corrected stroke of the user, so that the user not only attains a beautified stroke, but also feels the dynamics of creating it.

Haptic Feedback for Pen-Based Interfaces

Haptic feedback has been studied in the domain of stylus and tablet interfaces. Poupyrev et al. (2004) used a display instrumented with piezoelectric actuators to provide tactile feedback and report that users were significantly more precise in continuous interaction tasks such as sketching. Lee et al. (2004) propose an active stylus containing embedded actuators to provide personalized haptic feedback in collaborative settings. Withana et al. (2010) embed a linear actuator inside the pen to convey a sense of depth of the screen. Digital rubbing employs a similar system, however, for the purpose of tracing over digital images on real paper (Kim et al., 2008). The technique requires a solenoid to activate when the pen passes over a part of the underlying sketch. Because of the delay between measurement and actuation, a simple dead-reckoning movement predictor was applied resulting in significantly better alignment between digital and analog tracings. Some combined haptic feedback with visual guidance (Portillo et al., 2005). This also promoted research into evaluating visuohaptic systems (Yang et al., 2008).

Haptic interfaces have also been used to increase the accessibility of GUIs by providing Braille-like feedback on the side of the pen during interaction tasks (Kyung et al., 2008, 2009). RealPen (Cho et al., 2016) increases the realism of sliding over surfaces by mimicking audio-tactile properties of materials like paper. Pen based input was also combined with a 3-DOF device (Phantom Omni) as a handwriting aid for stroke patients (Mullins et al., 2005). However, since the user writes in air, the realism of writing on paper is lost. Similarly, there has been some work on tools (Zoran and Paradiso, 2013; Zoran et al., 2014; Peng et al., 2015).

While the benefits of haptic feedback in tablet and pen-based interfaces have been demonstrated (Cho et al., 2016; Poupyrev et al., 2004; Kyung et al., 2009), we argue that a tight adaptive control loop is necessary to improve perception, accuracy and utility of such approaches (cf. (Kim et al., 2008)). We therefore propose a MPCC-like optimization approach to provide programmable real-time haptic feedback to the user with the aim of increasing accuracy and aesthetic output, without removing user agency.

Magnetic Actuation

Providing magnetically-driven haptic feedback on tabletops is desirable as the force can be exerted through the surface without affecting the display. A common approach is using arrays of controllable electromagnets, combined with permanent magnets embedded in objects on the surface (Pangaro et al., 2002; Yoshida et al., 2006; Weiss et al., 2011). Fingerflux by Weiss et al.  (2011) provide near-surface haptic feedback before the finger touches the screen to guide users to appropriate screen locations. Pangaro et al. (Pangaro et al., 2002) model the force-field of each electromagnet and combine these using standard aliasing techniques, allowing directed movement of multiple objects on the surface. However, moving objects smoothly across the surface is problematic due to the low resolution of the grid, and the interaction of forces between multiple electromagnets. Furthermore magnets are modeled using a simplification of the Gilbert model, considering attraction between single point poles. In sketching and writing tasks, accuracy is of the utmost importance, and thus, we employ a printer-like setup, allowing for smooth movement across a 2D plane and contribute a more accurate EM model based on oriented dipoles.

dePENd by Yamaoka and Kakehi (2013) is perhaps the closest prior research to our system. They move a permanent Neodymium magnet on a two-axis setup to control the pen of a user. They make use of the ferromagnetic feature of the metal tip of regular ballpoint pens to attract it. Our work differs significantly in the type of magnet, its mathematical model and the control strategy, resulting in a different user experience and system capabilities. The neodymium magnet in their work “forces”, rather than guides, the user to follow a predefined path. In Yamaoka and Kakehi (2013) the pen is not tracked which results necessitates an open-loop control strategy and little to no user autonomy. The user is allowed to deviate from the stroke only slightly by lifting or moving the pen. However, this input does not alter the behavior of the magnet. In comparison, our system allows the user to move at their own pace through a drawing, and reacts in real-time to user input by altering the position and strength of the magnet to compensate for user input, thus providing an haptic guidance of the location of the reference path, without prescribing a set velocity along the guidance path and not entirely controlling the user motion.

Online Path Following

Optimal reference following given real world influences is studied in depth in the control theory literature. Methods like MPC (Faulwasser et al., 2009) optimize the reference path and the actuator inputs simultaneously based on the system state. MPC is wildly applied to many robotics (e.g., (Mueller and D’Andrea, 2013)) and graphics applications (Da Silva et al., 2008). However, (Aguiar et al., 2008) show that the tracking error for following timed-trajectories can be larger than if following a geometric path only. To address this issue Model Predictive Contouring Control (MPCC) (Lam et al., 2013) has been proposed to follow a time-free reference, optimizing system control inputs for time-optimal progress. MPCC approaches have been successfully applied in industrial contouring (Lam et al., 2013) and RC racing (Liniger et al., 2014) and in drone cinematography (Nägeli et al., 2017). We also pose our optimization problem in the MPCC framework. However, to the best of our knowledge we are the first to do so in the context of haptic feedback systems where one has to consider both a controllable (i.e., the linear stage) and non-controllable (i.e., the user) system. Furthermore, we contribute a fast approximate electromagnet (EM) model, that gives good experimental fit, for use in iterative optimization schemes.

3. Overview

The goal of our work is to provide an integrated software-hardware solution that can provide dynamically adjustable force feedback to a regular ballpoint or digital pen. Importantly, we argue that user agency is crucial. This is defined as the user staying in control of their actions and the system playing only a supportive role. Hence the system should never control the user’s motion but only provide feedback. In particular, the user may maintain personal speed and style of drawing.

Function MPCC()
       compute lag and contour error Sec 5.2 compute force error Sec 4 & Sec 5.3 sum() minimize() Sec 5.4
return , initialize while drawing not finished do
       Measure pen position Sec 5.1 update system states, from sensor data update MPCC parameters Sec 5 apply(
end while
Algorithm 1 Closed Loop Haptic Feedback Control.

We propose the MagPen system, shown in Figure 2. MagPen consists of a high speed pressure sensitive tablet under which an electromagnet is moved on a bi-axial linear stage. The electromagnet delivers adjustable force feedback to a mostly unmodified pen – we only attach a small permanent magnet to an otherwise passive ballpoint pen. Our proposed optimization scheme (summarized in Alg. 1) allows us to adjust the magnet position and strength such that it gently pulls the pen tip towards a desired stroke, while allowing users to draw at their desired speed and without fully taking over control. Here we assume that the user traces a known trajectory. This already enables a number of applications such as practice support in writing of characters or sketching, or as active support in inking or underpainting. We leave integration with a full predictive model (e.g., (Aksan et al., 2018)) for future work.

At each time step, we minimize a cost functional over a receding time horizon in order to find optimized values for system states and inputs . The cost function, here given as high-level abstraction,


serves three main purposes: 1) ensuring that the user stays close to the desired path, 2) makes progress along it and 3) the user perceives haptic feedback of dynamically adjustable force.

We first introduce the hardware platform in Sec. 4, including a novel approximate model of the electromagnet that can be evaluated in closed form and hence is usable within an iterative optimization method. We then introduce our closed-loop control formulation in Sec. 5 that leverages the force behavior model to compute control inputs for the bi-axial linear stage and for the electromagnet.

4. Hardware

In designing the MagPen hardware (Figure 2) we balanced several important aspects. First, the magnetic force must be controlled in a fine-grained manner to allow for user agency, essentially ruling out the use of permanent Neodynium magnets. Second, very small pen displacement can lead to undesired snapping of the pen tip unless the magnet is adjusted almost instantaneously, due to the quadratic rise in magnet attraction as a function of the pen magnet distance (see accompanying video for an illustration). This requires low-latency sensing hardware and control software. We chose a display-less pressure-sensitive tablet since it provides a high framerate. Finally, modeling electromagnetic fields is highly involved and typically requires FEM simulation which would not lend itself for use in iterative optimization schemes. This problem is intensified if EM fields overlap and interact, such as in a grid of electromagnets (cf. (Pangaro et al., 2002)). Therefore we move a single electromagnet on a bi-axial linear stage and contribute an approximate model of the EM field. We leave miniaturization into a tablet form-factor for future work but argue that the current form-factor already is an interesting solution for tracing tables or digital whiteboards.

4.1. Sensing and Actuation

One of our design goals is to provide users with an as unencumbered as possible experience, staying close to the experience of sketching on pen and paper but allowing for in-situ haptic force feedback.

In MagPen users draw on normal paper with a minimally modified ballpoint pen. The strokes are recorded by a Sensel Morph (https://sensel.com/

) pressure sensitive touch pad. We chose the Sensel board for it’s high spatial resolution (6502 DPI), high speed (500 Hz) and low latency (2 ms). Since the board is designed to be used in combination with different overlays, the sketching surface does not interfere with the input recognition. Users draw with a 3D printed ballpoint pen with a permanent ring magnet mounted in the shaft (see Figure 


To deliver haptic feedback to the pen, we move a programmatically controlled electromagnet on a bi-axial linear stage directly underneath the input sensor. Our implementation leverages the bi-axial system and motors from a Makerbot Replicator 2X 3D printer. We replaced the printer head with an electromagnet (Intertec ITS-MS-5030-12VDC, diameter = 5 cm, height=3 cm, 12 V 11 W) mounted on a heatsink and cooled by a fan. The choice of the electromagnet is non-trivial: we need a strong enough electromagnet, fitting in our hardware setup which limits size and heat dissipation, while not being too heavy for the linear stage. We employ FEM analysis to make an informed choice with regard to the electromagnetic characteristics. Our system allows up to 488 mN of lateral force (at 11 W), with only 2.8 mm of point dispersion (see Sec 6.3)

The stepper motors are controlled via a Sparkfun EasyDriver motor-shield and an Arduino Uno, allowing for micro stepping (a full step is 2.mm, we use quarter stepping) of the motors which increases the smoothness and accuracy of the magnet motion. The electromagnet is controlled via pulse-width modulation (PWM) and an H-Bridge. While allowing for both repulsive and attractive forces, we only use attraction for simplicity.

4.2. Electromagnetic Force Model

With the hardware platform in place, we require a mathematical model of the electromagnet to compute the attraction force between magnet and pen as a function of their distance. Here we face a significant challenge: modeling the full EM field, with the help of a FEM analysis, is not computational feasible in real-time; yet we require a physically accurate description of the force behavior at every iteration of our optimization procedure. One of our main contributions is an approximate model that is efficient to evaluate and provides a very good fit to empirical data. Here we briefly discuss its derivation and we refer to Appendix B and C for additional details and for a full validation of the model.

The electromagnet core is made of a non-linear ferromagnetic material. Thus, calculating the physical correct force behavior would involve pre-computation of a volumetric map of the EM field via FEM simulation for all levels of the electrical current. The actuation force on the pen is then given by integrating over the volume of the permanent magnet in the pen:


where is the magnetization of pen magnet and is the EM field evaluated at the pen position. Intuitively this can be read as the force response corresponding to the gradient of the EM field evaluated within the volume of the pen magnet. Since this is too costly to evaluate in real-time, we propose an approximate model that is consistent with the underlying physical phenomena.

We make the following two assumptions in our derivations: 1) the electromagnet and the permanent magnet can be approximated as dipoles (i.e., oriented point magnets), and 2) for the smaller dipole (the permanent magnet in the pen) the out-of-plane vector component is much larger than the in-plane counterpart. This allows us to use only the vertical component in the calculation of the force. Note that these simplifications lead only to a small approximation error. Compared to an angle dependent formulation (see Appendix C Eq. 55), a tilt of up to leads to a max error in our model (Eq. 10) equivalent to shifting the distance by [mm] (see Figure 4). This uncertainty in is comparable with the in-plane positioning error of our overall system (Sec. 6.3). Furthermore, we found that users can not perceive a difference in strength when tilting the pen in-place.

Figure 3. Illustration of the model to compute the force on dipole due to dipole (see Eq. 3). In Eq. 10 we describe the actuation force under the upright pen approximation (), while in Appendix C we report a more general model for small tilt angles ().

The first approximation allows us to follow the formulation in (Yung et al., 1998) and to model the force exerted by the electromagnet dipole onto the pen dipole (see Figure 3) as:


where is a constant (see Table 1) and is the 3D vector between the centers of the electromagnet and pen dipoles. The electromagnet can be seen as magnetic dipole with variable strength, controlled by the dimensionless scalar .

The three vectors needed to compute Eq. 3 can be expressed in the coordinate system of Figure 3 as


Figure 4. In-plane magnetic force as function of position. The horizontal displacement between curves (each denoting a different pen-tilt) is the approximation error induced by the upright pen (purple) assumption (angles defined in Figure 3).

While providing us with an analytic and differentiable expression, this leads to an equation for the actuation force that depends on the tilt of the pen through the angles and . With our hardware these data are non-trivial to attain. We now leverage our second assumption by rewriting Eq. 3 with an equivalent pen dipole , obtained by applying the small tilting angle approximation ( and ) to Eq. 5,


where the scalar magnetization is given by . is the residual magnetization of the permanent magnet and its volume and is the -unit vector. This approximation removes the requirement for tracking the pen tilt. More importantly it drastically simplifies the force equation since both dipoles now only have a component and thus the actuation only depends on the distance between pen and magnet (not on nor ). This provides a simplified version of the 3D distance vector,


where the vertical distance, , is constant. Note that the in-plane distance is one of the variables we seek to control, given the projections of the pen position () and the electromagnet position () onto the paper plane.

The electromagnet dipole () is mounted in a fixed upright position. Therefore it can be expressed via Eq. 4, without incurring any approximation error. The magnetization value of the full-strength dipole , which approximates the electromagnet, can be derived experimentally. For this purpose we scan the magnetic field generated by the electromagnet, setting and using a hall sensor and adjust the parameters of EM field equation to give a good fit. The full procedure and derivations can be found in Appendix B. Table 1 reports the values of , and that were used in our experiments.

Name Value Description
[H/m] Vacuum permeability
1.3 [T] Pen magnet type (NIB N42)
0.66 [cm] Pen magnet volume
0.683 [A m] pen dipole ()
1.286 [A m] electromagnet dipole (see App. B)
2.71 [cm] z-distance to (see App. B)
1.40 [cm] height pen-tip to magnet (see Fig. 3)
0.488 [N] force factor in Eq. 10),
Table 1. List of electromagnet model and hardware parameters

The total force acting on the pen (Eq. 3) can now be decomposed into the in-plane and vertical force components:


Here represents the quantity we seek to control. By substituting the results form Eq. 4, 7 and 8 into Eq. 3 and maintaining only the in-plane contributions ( direction), we obtain the expression for the actuation force as function of pen-magnet separation:


where is a constant force parameter given by the expression,


Fig. 4 illustrates how the dimensionless ratio within parentheses in Eq. 10 governs the force strength as function of distance . The actuation force is zero if the two magnets are aligned with one another (), it has a maximum at , and we can assume there is no more attraction for distances . In Table 1 we report the value of we obtained for our setup.

Figure 5. Illustration of actuation force , desired force , and the force cost-term associate with the difference between those two forces.

We note that the vertical force component from Eq. 9 pulls the pen downwards. However, during our experiments there was no significant change in ink thickness or perceived friction when comparing the drawings with and without electromagnet (i.e., with or without ). For this reason we do not actively optimize for in our optimization. Finally, we provide an angle dependent formulation of our model in Appendix C for future use in cases where the pen angle is tracked.

5. Closed-Loop Control

With the force behavior model from Eq. 10 we can now derive our control strategy, such that a force of desired strength is exerted onto the pen, in order to keep the user close to the desired path. The known path is parametrized by , where is the length of the path. Note that we do not want to prescribe how fast the user draws the shape and hence for each given pen position we first need to establish the closest position on the path parametrized by . Furthermore, we seek to find optimized values for the electromagnet intensity and the in-plane electromagnet position . We phrase this problem in the MPCC framework (Lam et al., 2013). Solving the error functional given in Eq. (5.4) at each timestep, yields optimized values for system states and inputs .

Figure 6. Overview of different control strategies and their theoretical behavior. For sake of simplicity the user is kept at a constant position for all time-steps. In the case of Open-Loop, the position of the electromagnet is identical to the setpoint. The setpoint is defined per timestep. For MPC the setpoint is still defined per timestep, however the EM is at an optimized position between the setpoint and the pen. MPCC, on the other hand, also optimizes the setpoint based on the pen position (hence, in this stationary case, the setpoint is also stationary). In a time-dependent setting the electromagnet might not be close to the pen (open-loop) or in the wrong direction (time-dependant closed loop). These problems are not present in a time-independent closed loop approach.

From a high level perspective, Model Predictive Contour Control is a closed-loop time-independent control strategy that minimizes a cost-function over a fixed receding horizon. As is commonly done in MPC(C), the system is initialized from measurements at . The system state is then propagated over the horizon with the help of the system dynamics (Eq 12). The state vector x contains only variables that are controlled by the algorithm (cf. Eq. 13). Only the first of the optimized inputs () is then applied to the physical system, transitioning the system state to , before iteratively repeating the process. This allows for iterative correction of noisy predictions due to modelling errors.

There are several advantages in using a time-independent closed-loop controls strategy, such as MPCC, over open-loop or time-dependent strategies. First, closed-loop control allows the system to react to user-input, whereas open-loop control removes all user agency. Both MPC and MPCC are closed-loop control strategies. The main difference is that MPC tracks a timed reference, prescribing a fixed velocity, whereas MPCC follows a time-free trajectory, which allows the user to progress at the desired speed. Figure 6 illustrates the expected behavior for a closed-loop versus a timed and time-free strategy respectively, given that the user decides to slow down or stop moving the pen.

5.1. Dynamics Model

To control linear stage and electromagnet we require a model describing the system dynamics given its states and inputs .


We model the electromagnet with its position and magnet intensity and include a path progress .


The inputs to the system consist of the in-plane electromagnet accelerations , and velocities and for magnet intensity and the spline progress respectively:


Note that we empirically found that magnet accelerations yield smoother motion than using velocities. The state and inputs in the model are in SI-units. However, the acceleration is converted to stepper motor increments before they are send to the EM controller.

The system model is given by the non-linear ordinary differential equations using first and second derivatives as inputs:


where are the external inputs. The continuous dynamics model is discretized using a standard forward Euler approach: (Gibbs, 2011).

We derive the sets of admissible states and inputs empirically to conform to the physical hardware constraints of the linear stage and EM specifications (e.g. max voltage). These will be used in the contrained optimization problem solved in Eq. 27.

The pen position is propagated via a standard linear Kalman filter

(Gibbs, 2011). While not an accurate user model, it works well in practice since the states are recalculated at every timestep.

5.2. Path Following

We continuously optimize the EM parameters with the goal of keeping the distance between the desired path and the pen minimal. However, we cannot rely on a timed trajectory as is commonly done for example in MPC formulations (e.g., (Faulwasser et al., 2009)), since we want to give the user freedom in deciding their drawing speed. To achieve this trade-off, we first need to find the reference point itself. Finding the closest point on the path is an optimization problem itself and hence can not be used within our optimization. Similar to recent work in MAV trajectory generation (Nägeli et al., 2017; Gebhardt et al., 2018) we decompose the distance to the closest point into a contouring and lag error.

Figure 7. Illustration of lag- and contouring error decomposition.
(a) On-path behavior.
(b) Sudden jump in pen position.
(c) Smooth correction.
Figure 8. Illustration of error correcting behavior. Left-to-right: (fig:error_correction_a) a simulated user is close to the desired path. (fig:error_correction_b) A sudden jump in pen-position causes the magnet to move towards the pen and to increase the magnet strength, (fig:error_correction_c) which pulls the pen smoothly back to the trajectory . See Sec. 6.2. The background is a plot of the cost function; in which purple is the lowest.

We define as the distance between the pen and a point on the spline, and as the normalized tangent vector to the spline at that point:


with . The vector can now be decomposed into a lag error and a contour error (see Figure 7). The lag-error is computed as the projection of on the tangent of , while the contour-error is the component of orthogonal to the normal:


Separating lag from contouring error allows us, for example, to differentiate how we penalize a deviation outside the path (), from encouraging the user to progress forward (). We furthermore include cost terms to ensure the magnet stays ahead of the pen () and to encourage smooth progress ():


Finally, we optimize the progress variable so that is a combination between the closest point and ensuring that the magnet eventually does progress to indicate to the user in which direction to continue.

5.3. EM Force Control

Leveraging our physical model for the electromagnetic force (Eq. 10), we propose a residual for the desired force (see Figure 5). The desired force, pulls the pen towards the target point . The cost is modeled with a spring-like behaviour, so that it linearly increases with the distance :


where is a scalar that regulates the stiffness of the spring, a scaling of the EM force and the distance between dipoles in (set manually). Although simple, this formulation ensures that the haptic guidance is strong under large deviation from the path while vanishing as the user approaches the target path (). Note that the EM force saturates at . Using the expressions for the actuation force (Eq. 10) and desired force (Eq 22) we formulate a quadratic cost term to penalize the difference:


Since the actuation force declines rapidly with distance, the gradient of goes to 0 for large values of causing the optimization to become unstable. To counterbalance this issue we encourage the electromagnet to stay close to the pen:


Finally we encourage an interplay between magnet position and intensity by penalizing excessive use of magnetic intensity :


5.4. Problem Formulation

We combine the cost terms Eq. 1821 and Eq. 2325 to form the final stage cost:


where the scalar weights control the influence of the different cost terms. The values used in our experiments can be found in Table 6 in Appendix A. The states and inputs to the bi-axial linear stage and to the electromagnet are then computed by solving the -step finite horizon constrained non-linear optimization problem at time instance .

Subject to: (System Model)
(Initial State)
(Initial Progress)
(Progress along path)
(Path Length)
(State Constraints)
(Input Constraints)

Here indicates the horizon stage and the additional weight reduces over the horizon, so that the current timestep has relatively more importance than later timesteps. is a positive definite penalty matrix avoiding excessive use of the control inputs. In our implementation we use a horizon length of . Experimentally we found that this is sufficient to yield robust solutions to problem instances and longer horizons did not improve results.

6. Technical Evaluation

We evaluate our system both from a technical and performance perspective. Furthermore we report quantitative and qualitative findings from an initial user study.

6.1. Implementation Details

We use a standard desktop computer (Intel Core i7-4770 CPU 4 cores at 3.40 GHz) running Ubuntu 17.10 in our experiments. The solver is implemented in FORCES Pro (Domahidi and Jerez, 2014), which produces efficient C-code. The communication with the Sensel Morph tablet and the Arduino board controlling the stepper motors happens asynchronously over a serial connection.

6.2. Error-correcting behavior

An important design goal is to provide guidance to the user, especially under deviation from the path. Here we analyze how the algorithm responds to various types of erroneous situations. To this end we simulate a cooperative user, always following the guidance, but initialize the pen position at different off-path positions. We update the pen position via a simple constant velocity model:


where the pen-position at timestep is updated via the previous position and a weighted sum of the normalized velocity , the exerted force direction and a constant velocity factor .

We analyze a typical but difficult to handle path deviation: a y-offset in combination with an offset in the direction of negative (backwards). Figure 8 illustrates that our algorithm handles the error gracefully. In particular the magnet is controlled such that it gently pulls the pen back to the path, rather than pulling directly towards either the closest point on the path or a steadily advancing setpoint, as would be the case in traditional control methods.

6.3. Positional dispersion

We now analyze the positional accuracy of the system while isolating the user contribution. Note that in the context of drawing the interesting aspect is not the accuracy of the position of the electromagnet (stepper motors are discrete and accurate), but the influence of the magnet on the pen. To assess this we moved the electromagnet to random locations and then always back to the center of the drawing surface. With a pen being held fully upright and the magnet at full strength (

) and the user following the magnet passively, this allows us to measure the system’s positional dispersion. We collected data from 300 repetitions, resulting in a mean offset from the target of 2.8 mm with a standard deviation (SD) of 0.8 mm, indicating that the system can control the pen well. One of the factors that contribute to this dispersion is the vanishing of the actuation force

as . This can lead to the pen motion stopping slightly before it reaches the target.

6.4. Latency

Due to the steepness of the electromagnetic force and the potentially fast pen motion, runtime and latency are crucial performance metrics. The optimization algorithm contributes to both, whereas latency is dominated by the hardware and I/O. The mean solve time for a problem instance is 7.4 ms ( 3.0 ms). Since we do not manipulate the system state space and pen input is the only measurement input this can be expected to be mostly constant. The hardware and overall system latency adds to the solve time. We employ a high-speed camera (1000 fps) to establish the motion (pen) to motion (magnet) latency. This yields an approximate latency of ~10 ms. Given the combined latency envelope of ~20 ms, we did not experience any abrupt pen snapping in our experiments.

7. User Evaluation - Control Strategies

We first conduct a preliminary experiment to validate the choice of a time-free closed-loop optimization strategy. For this purpose we compare our implementation to a simpler MPC variant and an open-loop strategy (our implementation of dePENd (Yamaoka and Kakehi, 2013)). The main difference in ours is that is an optimization variable itself, whereas MPC used a setpoint that progresses along the reference at a predetermined speed.

7.1. Procedure

In this experiment we asked participantes () to draw one complex shape (Fig. 10(e)) in three different conditions: Open-Loop (OL), time-dependent closed loop (MPC), and time-free closed loop (ours). The order of the trials was counterbalanced via a latin-square design.

Figure 9. Overview of the different metrics for the preliminary user evaluation.

7.2. Measures

We analyze three measures; i) The mean distance from the pen to the path, ii) the mean distance from the pen position projected onto the path and along the path and iii) the mean distance from the pen to the electromagnet. Taking the mean of the error terms over subjects we ensure equal numbers of datapoints, accounting for differences in speed. Note that here we assume that the user roughly maintains a constant speed.111In our full implementation and haptic feedback experiments this assumption is not necessary. However, the metric used here assumes time-independent datapoints. We also gathered qualitative feedback in the form of a semi-structured interview.

A one-way ANOVA with Kruskal-Wallis test was performed for data analysis.

7.3. Results

(a) Path distance pen-
(b) The euclidean distance from pen-magnet
Figure 10. Comparison of error over time for a single participant (P1). In 9(a) there is the inverse u-shape that illustrates that the ) moves at a different speed than the user for OL and MPC. Sub-figure 9(b) show how the EM is relatively close and constant to the pen with MPC and MPCC. With OL the magnet moves away from the pen. The data is smoothed to increase readability.
—pen-path— d(pen, ) —pen-em—
MPCC 2.0 6.2 4.6
Table 2. Mean distance in mm from 1) the pen to the closest point on the path, 2) distance from pen to along the path, 3) the pen to the electromagnet and 4) the fraction of the measurements where the electromagnet is more than 15 mm away from the pen (based on Figure 4). All units are given in mm.


Table 2 summarizes our quantitative findings. Not surprisingly, the distance from the electromagnet to the pen for (OL) is significant. Since the force exerted on the pen falls off quadratically with distance, participants often lost any haptic guidance early on, confirmed via user comments such as “I don’t feel anything” (P3) and “Is the system on?” (P6). Similarly, we see that is larger compared to ours by a factor of six.

While MPC reduces the distance from the pen to the magnet (and hence always provides haptic feedback), it does not optimize for the progress along the path and hence may pull the pen into undesired directions. For example, MPC produces extreme corner cutting behavior to catch up to the setpoint.

Finally, ours has the highest accuracy (H(2)=20.76, p¡.001). Furthermore, the setpoint (H(2)=7.362, p ¡.05) and the electromagnet (H(2)=27.12, p ¡.001) are closest to the pen. Thus our time-free formulation overcoms both problems of wrong setpoints (MPC) and a run-away electromagnet, as can happen with strategies proposed in prior-work (Yamaoka and Kakehi, 2013).

Figure 10 shows that both the distance along the path and the pen-magnet distance accumulate over time if OL or MPC control strategies are employed. Note that this is not the case with our implementation (MPCC) and both errors remain low over the entire path length.

Standard Deviation

Note that for OL and MPC the standard deviation is high. This is likely due to the absence of direct coupling between user feedback and path progress, which makes it possible for the user to lag behind the setpoint significantly (albeit at the cost of reduced force feedback). In our implementation the path progress is adjusted to the user’s drawing speed, drastically reducing the standard deviation and in consequence ensuring delivery of force feedback throughout the drawn path.


From our observations we saw that was either in front or behind the user for MPC. This was also confirmed in our interview, where people especially commented on the MPC strategy: “The system tries to push me in the wrong direction” (P2) and “It is counteracting me” (P11). This also resulted in the MPC being the least preferred option. In contrast with our formulation the magnet remains always slightly ahead of the pen, resulting in users rating the MPCC as the most preferred condition. In the words of one subject this is: “since I still had control” (P9).

Taking both the quantitative and qualitative results into account, we see that our MPCC formulation performs best overall. Open-Loop causes numerous problems, including users not perceiving any haptic feedback. This is especially troublesome in settings where autonomy is desired. In MPC the haptic feedback is perceived, but can be erroneous. This is especially evident when users do not conform the expected behavior. We therefore only report results from the MPCC formulation in all further evaluations.

8. User Evaluation - Haptic Feedback

The impact on task performance and user perception is potentially the most important aspect of any haptic feedback system. To assess these factors we ran an initial controlled user study. In this study, we investigate the overall performance of our MPCC formulation in more detail. To this end we conduct experiments with several different shapes, and compare to a no-feedback baseline. A first impression of the results can be found in Figure 12.

(a) Circle
(b) Line
(c) Spiral
(d) Sinusoidal
(e) Dog
(f) Ellipse
Figure 11. Shapes used in our user tests. Note that the drawing surface only contained sparse visual references (shown in blue) and starting positions (orange).
(a) Line
(b) Ellipse
(c) Circle
(d) Spiral
(e) Sinusoidal
(f) Dog
Figure 12. Selected experimental results. Each shape drawn by different participants with (orange) and without (blue) guidance, compared to the reference (dotted). Sinusoidal is different from the one in Figure 12(a)

8.1. Procedure

We invited 12 participants (=8; =4, Age=28.2 2.2) into our lab. All subjects were right-handed and did not have any professional drawing experience. Before commencing the experiment, users were given an introduction to the system functionality and got to experience the system in a self-timed training phase. Only once participants were reasonably confident in the system we continued with the experiment.

During the experiment we asked each participant to draw six basic shapes, illustrated in Figure 11. Each participant drew each shape with and without haptic feedback once. The presentation order of shapes and interface condition was counterbalanced. The drawing surface only contained a starting point and, in the case of more complex shapes, additional visual guidance (shown in red in Figure 11). Furthermore, the participants were shown a scaled version during task execution (scaled to prevent 1:1 copying).

(a) Sinusoidal
(b) Error over time
(c) Pen to reference error histogram.
Figure 13.

Accuracy comparison for a single participant. fig:single_user_a reference shape (dotted line) overlaid by path drawn by the same user with (orange) and without (blue) haptic guidance. The absolute error increases over time without error correction fig:single_user_b. Error histogram reveals more compact distribution skewed towards low errors.

8.2. Results

Quantitative Results

We first analyze the results quantitatively. We use a Hausdorff-like distance (Rockafellar and Wets, 2009) as error metric. To make the metric robust to drawing speed, we re-sample the drawn path and the reference equidistantly. We then compute the distance from all re-sampled points to the closest point on the reference. To ensure fairness we also compute the distance from reference to the drawn path. A Kolmogorov-Smirnov test (Kolmogorov, 1933) indicates that both sets are the same and we report uni-directional distances.

Figure 12(a) compares the reference (dotted-line) to a sinusoidal drawn without (blue) and with haptic feedback (orange) by one of our participants. The path drawn with guidance clearly stays closer to the reference and drifts less. Plotting the error over time confirms this observation (Figure 12(b)), where the error with guidance stays more or less constant and the guidance-free error continuously increases. Figure 12(c), plots the error histogram for both conditions showing a longer tail without haptic assistance. This trend holds for all users as can be seen in 14, plotting the pen-reference distances for all users.

We conducted a two-way ANOVA on the mean error (computed over all users), as metric for accuracy. Results show a main effect for the feedback type (F=46.187, p¡.001) and for the shapes (F=11.771, p¡.001). Post-hoc analysis reveals that the line is statistically significantly different from all other shapes and we report its results separately. Mean accuracy per shape and significance levels are summarized in in Table 3, showing that haptic feedback significantly improves accuracy across shapes to 1.871 mm (p¡.001). While there is no significant difference for the line, even when including it, we attain a significant accuracy improvement by 1.537 mm (p¡.001).

Figure 14. Distribution of pen to reference distances. All subjects combined. The entries have been trimmed for 10% on both the upper and lower limit in order to increase readability.
With Without
Scenario Mean SD Mean SD Err
Circle* 2.19 0.90 4.26 2.39 0.51
Line 1.18 0.80 1.03 0.84 1.15
Spiral*** 2.55 0.75 4.38 1.64 0.58
Sinus*** 2.53 0.70 5.08 2.19 0.50
Dog*** 2.31 0.54 3.81 1.32 0.60
Ellipse*** 2.40 0.56 3.84 1.22 0.62
Table 3. Mean accuracy in mm. Percentage of error: avg(with)/avg(without) – lower is better. Significance values set to: *p¡0.05, **p¡0.01, ***p¡0.005

Qualitative Results

A brief exit interview (see Table 4) shows that users subjectively rate the system favourably in terms of accuracy, speed, force and overall performance on a 5-point Likert scale.

Finally, we qualitatively show the effect of using haptic feedback in Figure 12 using different shapes drawn by different participants. The more complex the shape, the more pronounced the difference between the conditions (cf. circle, spiral, dog).

Question Accuracy Speed Force Improvement
Mean 4.33 4.00 3.50 4.50
SD 0.62 0.91 0.86 0.90
Table 4. Survey results. Likert scale: 1=Not Accurate/Slow/Weak/No Improvement. 5=Very Accurate/Fast/Strong/Much Improvement.

9. Additional Results

Figure 15. Overview of use cases: calligraphy (14(b) and 14(d)) and drawing exercises (14(f)). Figures 14(a), 14(c), 14(e) show the guidance given during the experiment.

To further demonstrate the capabilities of our system we illustrate potential use-cases including applications in learning to draw, in outlining and in inking.


Figure 14(b) and Figure 14(d) illustrate writing of flourished characters, with only minimal visual guidance (single starting point). Although an offset from the reference path remains, the lines are smooth and the overall shape is close to the desired characters.

Drawing teaching aid:

Connect-the-dots exercises are often used to teach children motor skills as well as stroke ordering. Figure 14(f) shows results from a similar exercise performed with our system, yet using much fewer dots as visual guidance than a paper version.

Outlining & inking:

Figure 17 illustrates the effect of two core capabilities of the proposed approach. Here we first outline the proportions of the dragon head and then use different pens to ink-in the details. Note that the system provides haptic guidance but allows the user to draw the shape in different styles and with varying high-frequency detail, while maintaining similarity to the reference shape. This is a direct consequence of using time-free closed loop control approach, as is alluded to in Sec. 7. In this case, all four variants were drawn without changes to the system or desired path.

10. Discussion

From a technical perspective we can we can conclude our time-independent closed-loop control formulation, provides qualitatively and quantitatively better results than simpler approaches. Open-loop control might lead to complete loss of feedback due to differences in speed. This is solved in a closed-loop setting. However, time-dependent implementations can maintain haptic feedback but the perceived direction of the feedback maybe wrong. This is solved by our time-independent implementation, where is part of the optimization problem.

Our haptic feedback experiments indicate that the proposed approach indeed increases accuracy in drawing tasks and that users rate the system favorably. We did not find a significant difference for the straight line. This may be explained by user feedback, that the maximum speed of the linear stage is a limiting factor in the current implementation. We leave faster magnet positioning for future work.

Two aspects from the exit interviews are noteworthy. First, there is a high standard deviation in how users rated the perceived force. We hypothesize that this is due to the way users operate the pen, with some leveraging the full arm and others rely more on the wrist. We note that our palm rejection implementation is simple and may have contributed to this. Furthermore, some users indicated that they had the feeling that their drawings without feedback were more accurate once they experienced the haptic guidance, indicating the possibility of short-term muscle memory. Long-term learning however is difficult to evaluate experimentally and goes beyond the scope of this paper.

(a) 36.9 
(b) 46.1
(c) 57.1
(d) 73.3
Figure 16. Curvature dependent error. Insets subfig:angle_error_a-subfig:angle_error_c show increasingly sharp cornered references (dotted lines) and the resulting pen trajectory (in simulation). x-axis is normalized for cord length, so that all angles can be directly compared. Bottom: error over 9 different levels of curvature (in degrees).

Finally, during our experiments we noticed a tendency to cut corners, well illustrated in the case of the sinusoidal (see Figure 12(a), 11(e)). To unpack this issue further, we performed an experiment in simulation, with the user model from Eq. 28, tracing references with increasingly sharp angles (see Figure 16). The plot clearly shows an increase in error with increase in curvature. It has been shown that humans trade-off speed and accuracy in tracing tasks and that they slow down when tracing high-curvature paths (Accot and Zhai, 1997). Currently our implementation does not take curvature of the reference into account but it would be straightforward to penalize the progress along the reference according to its curvature.

11. Limitations & Future Work

There are, of course, some limitations to our current approach. First, the maximum speed of the linear stage we used is relatively slow. This causes users to slow down their drawing speed. This could be overcome via a faster bi-axial linear stage, which are commercially available but expensive. A potentially more interesting and scalable direction would be to extend the proposed magnet model towards a grid of electromagnets. However, the interaction between several overlapping EM fields with a moving permanent magnet are non-trivial to model and would require significant research.

Once the hardware-induced speed limitation is overcome, efficient closed-loop control approaches become an interesting direction for future work, since faster pen motion would also tighten the latency and accuracy budget. In the context of sensing it would be interesting to incorporate a mechanism to reconstruct the tilt of the pen. This could be achieved for example via an accelerometer built into the pen or via a grid of hall-sensors underneath the surface. Information on the pen tilt could then be combined with the angle dependent formulation of our EM model (see Appendix B). Furthermore, we believe there are many research opportunities in combining our approach with other sketch and ink beautification approaches (e.g., (Simo-Serra et al., 2016; Simo-Serra et al., 2018a; Xing et al., 2015)). Particularly interesting would be to leverage fully predictive models (e.g., (Aksan et al., 2018)) in order to overcome the need for a known reference.

Another interesting direction of research is connected to the observation that different users perceive the feedback at different strength levels. This could be due to grip strength, or movement from the shoulder rather than wrist. A stronger (i.e., bigger) electromagnet could increase the dynamic range. However, this would have to be carefully counterbalanced with weight and heat dissipation concerns as well as with a loss in accuracy towards the center of the electromagnet (the force goes to zero as ).

We believe that it could be an interesting direction for future work to combine our approach with different types of haptic feedback, either environment mounted or body-worn. Moreover, we have so far focused our attention towards drawing applications. However, electromagnetic feedback in combination with spatial actuation maybe interesting in other settings. For example, a magnet mounted to a robotic arm could deliver contact-less feedback in VR scenarios. It would also be interesting to investigate how to best exploit the system capabilities in the context of motor memory and learning.

Figure 17. Different variants of the same dragon, drawn with identical system settings by a novice. Each pair of drawings used with different tools. First a pencil for proportions and a fine-liner (top) or pencil (bottom) to ink-in details. Multi-stroke lines are achieved by approaching each seperate instance as a new figure, the system is trigger by a pen lift.

12. Conclusions

We have proposed MagPen, a system that delivers dynamically adjustable guidance in drawing and sketching tasks. We have detailed our hardware setup and discussed a novel model of the electromagnetic interactions in the system. The proposed model can be evaluated analytically and is hence suitable for iterative, real-time optimization approaches. We have furthermore demonstrated that the assumptions of dipole magnets and an upright pen lead only to a small approximation error. However, we have included a angle dependent formulation that maybe used in future work, where pen-tilt information is available.

We have also discussed a novel optimization scheme based on the MPCC framework that leverages the EM model, in order to optimize the system states and its inputs over a receding horizon via solving a stochastic optimal control problem at each timestep. Our formulation has been designed to provide dynamically adjustable forces and automatically adjusts magnet position and strength.

Our experiments have shown that the proposed hardware-software solution is effective in improving accuracy and in guiding users in a variety of drawing tasks, without taking away agency and control from the user. We believe this is an interesting first step towards many exciting applications of electromagnetic haptic feedback. In order to foster future research we will release all hardware schematics and software source code to the public.

This work was supported in part the Hasler Foundation (Switzerland), ERC (OPTINT StG-2016-717054) NSF (IIS-CAREER 1652515 and OAC:1835712), a gift from nTopology, and a gift from Adobe Research. We thank all participants for taking part in our experiments.


  • (1)
  • Accot and Zhai (1997) Johnny Accot and Shumin Zhai. 1997. Beyond Fitts’ law: models for trajectory-based HCI tasks. In Proceedings of the ACM SIGCHI Conference on Human factors in computing systems. ACM, 295–302.
  • Aguiar et al. (2008) A. Pedro Aguiar, Joao P. Hespanha, and Petar V. Kokotovic. 2008. Performance limitations in reference tracking and path following for nonlinear systems. Automatica 44, 3 (2008), 598 – 610. https://doi.org/10.1016/j.automatica.2007.06.030
  • Aksan et al. (2018) Emre Aksan, Fabrizio Pece, and Otmar Hilliges. 2018. DeepWriting: Making Digital Ink Editable via Deep Generative Modeling. In SIGCHI Conference on Human Factors in Computing Systems (CHI ’18). ACM, New York, NY, USA.
  • Åström and Hägglund (1995) Karl Johan Åström and Tore Hägglund. 1995. PID controllers: theory, design, and tuning. Vol. 2. Instrument society of America Research Triangle Park, NC.
  • Cho et al. (2016) Youngjun Cho, Andrea Bianchi, Nicolai Marquardt, and Nadia Bianchi-Berthouze. 2016. RealPen: Providing Realism in Handwriting Tasks on Touch Surfaces using Auditory-Tactile Feedback. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology. ACM, 195–205.
  • Da Silva et al. (2008) M. Da Silva, Y. Abe, and J. Popović. 2008. Simulation of Human Motion Data using Short-Horizon Model-Predictive Control. Computer Graphics Forum 27, 2 (2008), 371–380. https://doi.org/10.1111/j.1467-8659.2008.01134.x arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/j.1467-8659.2008.01134.x
  • Domahidi and Jerez (2014) Alexander Domahidi and Juan Jerez. 2014. FORCES Professional. embotech GmbH (http://embotech. com/FORCES-Pro).
  • Faulwasser et al. (2009) T. Faulwasser, B. Kern, and R. Findeisen. 2009. Model predictive path-following for constrained nonlinear systems. In Proceedings of the 48h IEEE Conference on Decision and Control (CDC). 8642–8647. https://doi.org/10.1109/CDC.2009.5399744
  • Favreau et al. (2016) Jean-Dominique Favreau, Florent Lafarge, and Adrien Bousseau. 2016. Fidelity vs. simplicity: a global approach to line drawing vectorization. ACM Transactions on Graphics (TOG) 35, 4 (2016), 120.
  • Gebhardt et al. (2018) Christoph Gebhardt, Stefan Stevsic, and Otmar Hilliges. 2018. Optimizing for Aesthetically Pleasing Quadrotor Camera Motion. ACM Transactions on Graphics (Proceedings of ACM SIGGRAPH) 37, 4, Article 90 (2018), 11 pages.
  • Gibbs (2011) Bruce P Gibbs. 2011. Advanced Kalman filtering, least-squares and modeling: a practical handbook. John Wiley & Sons.
  • Hilaire and Tombre (2006) Xavier Hilaire and Karl Tombre. 2006. Robust and accurate vectorization of line drawings. IEEE Transactions on Pattern Analysis and Machine Intelligence 28, 6 (2006), 890–904.
  • Kim et al. (2008) Hyunjung Kim, Seoktae Kim, Boram Lee, Jinhee Pak, Minjung Sohn, Geehyuk Lee, and Woohun Lee. 2008. Digital rubbing: playful and intuitive interaction technique for transferring a graphic image onto paper with pen-based computing. In CHI’08 Extended Abstracts on Human Factors in Computing Systems. ACM, 2337–2342.
  • Kolmogorov (1933) Andrey Kolmogorov. 1933. Sulla determinazione empirica di una lgge di distribuzione. Inst. Ital. Attuari, Giorn. 4 (1933), 83–91.
  • Kyung et al. (2008) Ki-Uk Kyung, Jun-Young Lee, and Junseok Park. 2008. Haptic stylus and empirical studies on braille, button, and texture display. BioMed Research International 2008 (2008).
  • Kyung et al. (2009) Ki-Uk Kyung, Jun-Young Lee, and Mandayam A Srinivasan. 2009. Precise manipulation of GUI on a touch screen with haptic cues. In EuroHaptics conference, 2009 and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems. World Haptics 2009. Third Joint. IEEE, 202–207.
  • Lam et al. (2010) Denise Lam, Chris Manzie, and Malcolm Good. 2010. Model predictive contouring control. In Decision and Control (CDC), 2010 49th IEEE Conference on. IEEE, 6137–6142.
  • Lam et al. (2013) Denise Lam, Chris Manzie, and Malcolm C Good. 2013. Model predictive contouring control for biaxial systems. IEEE Transactions on Control Systems Technology 21, 2 (2013), 552–559.
  • Lee et al. (2004) Johnny C Lee, Paul H Dietz, Darren Leigh, William S Yerazunis, and Scott E Hudson. 2004. Haptic pen: a tactile feedback stylus for touch screens. In Proceedings of the 17th annual ACM symposium on User interface software and technology. ACM, 291–294.
  • Lee et al. (2011) Yong Jae Lee, C Lawrence Zitnick, and Michael F Cohen. 2011. Shadowdraw: real-time user guidance for freehand drawing. In ACM Transactions on Graphics (TOG), Vol. 30. ACM, 27.
  • Limpaecher et al. (2013) Alex Limpaecher, Nicolas Feltman, Adrien Treuille, and Michael Cohen. 2013. Real-time drawing assistance through crowdsourcing. ACM Transactions on Graphics (TOG) 32, 4 (2013), 54.
  • Liniger et al. (2014) Alexander Liniger, Alexander Domahidi, and Manfred Morari. 2014. Optimization-based autonomous racing of 1:43 scale RC cars. Optimal Control Applications and Methods (2014). https://doi.org/10.1002/oca.2123
  • Liu et al. (2018) Chenxi Liu, Enrique Rosales, and Alla Sheffer. 2018. StrokeAggregator: consolidating raw sketches into artist-intended curve drawings. ACM Transactions on Graphics (TOG) 37, 4 (2018), 97.
  • Mueller and D’Andrea (2013) M.W. Mueller and R. D’Andrea. 2013. A model predictive controller for quadrocopter state interception. In Proceedings of the European Control Conference (ECC), 2013. 1383–1389. http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=6669415
  • Mullins et al. (2005) James Mullins, Christopher Mawson, and Saeid Nahavandi. 2005. Haptic handwriting aid for training and rehabilitation. In Systems, Man and Cybernetics, 2005 IEEE International Conference on, Vol. 3. IEEE, 2690–2694.
  • Nägeli et al. (2017) Tobias Nägeli, Lukas Meier, Alexander Domahidi, Javier Alonso-Mora, and Otmar Hilliges. 2017. Real-time Planning for Automated Multi-View Drone Cinematography. ACM Transactions on Graphics (Proceedings of ACM SIGGRAPH).
  • Pangaro et al. (2002) Gian Pangaro, Dan Maynes-Aminzade, and Hiroshi Ishii. 2002. The actuated workbench: computer-controlled actuation in tabletop tangible interfaces. In Proceedings of the 15th annual ACM symposium on User interface software and technology. ACM, 181–190.
  • Peng et al. (2015) Huaishu Peng, Amit Zoran, and François V Guimbretière. 2015. D-Coil: A Hands-on Approach to Digital 3D Models Design. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. ACM, 1807–1815.
  • Portillo et al. (2005) O Portillo, Carlo Alberto Avizzano, Mirko Raspolli, and Massimo Bergamasco. 2005. Haptic desktop for assisted handwriting and drawing. In ROMAN 2005. IEEE International Workshop on Robot and Human Interactive Communication, 2005. IEEE, 512–517.
  • Poupyrev et al. (2004) Ivan Poupyrev, Makoto Okabe, and Shigeaki Maruyama. 2004. Haptic feedback for pen computing: directions and strategies. In CHI’04 extended abstracts on Human factors in computing systems. ACM, 1309–1312.
  • Rockafellar and Wets (2009) R Tyrrell Rockafellar and Roger J-B Wets. 2009. Variational analysis. Vol. 317. Springer Science & Business Media.
  • Simo-Serra et al. (2018a) Edgar Simo-Serra, Satoshi Iizuka, and Hiroshi Ishikawa. 2018a. Mastering sketching: adversarial augmentation for structured prediction. ACM Transactions on Graphics (TOG) 37, 1 (2018), 11.
  • Simo-Serra et al. (2018b) Edgar Simo-Serra, Satoshi Iizuka, and Hiroshi Ishikawa. 2018b. Real-time data-driven interactive rough sketch inking. ACM Transactions on Graphics (TOG) 37, 4 (2018), 98.
  • Simo-Serra et al. (2016) Edgar Simo-Serra, Satoshi Iizuka, Kazuma Sasaki, and Hiroshi Ishikawa. 2016. Learning to simplify: fully convolutional networks for rough sketch cleanup. ACM Transactions on Graphics (TOG) 35, 4 (2016), 121.
  • Su et al. (2014) Qingkun Su, Wing Ho Andy Li, Jue Wang, and Hongbo Fu. 2014. EZ-sketching: three-level optimization for error-tolerant image tracing. ACM Transactions on Graphics 33, 4 (2014).
  • Weiss et al. (2011) Malte Weiss, Chat Wacharamanotham, Simon Voelker, and Jan Borchers. 2011. FingerFlux: near-surface haptic feedback on tabletops. In Proceedings of the 24th annual ACM symposium on User interface software and technology. ACM, 615–620.
  • Withana et al. (2010) Anusha Withana, Makoto Kondo, Yasutoshi Makino, Gota Kakehi, Maki Sugimoto, and Masahiko Inami. 2010. ImpAct: Immersive haptic stylus to enable direct touch and manipulation for surface computing. Computers in Entertainment (CIE) 8, 2 (2010), 9.
  • Xing et al. (2015) Jun Xing, Li-Yi Wei, Takaaki Shiratori, and Koji Yatani. 2015. Autocomplete hand-drawn animations. ACM Transactions on Graphics (TOG) 34, 6 (2015), 169.
  • Yamaoka and Kakehi (2013) Junichi Yamaoka and Yasuaki Kakehi. 2013. dePENd: augmented handwriting system using ferromagnetism of a ballpoint pen. In Proceedings of the 26th annual ACM symposium on User interface software and technology. ACM, 203–210.
  • Yang et al. (2008) Xing-Dong Yang, Walter F Bischof, and Pierre Boulanger. 2008. Validating the performance of haptic motor skill training. In 2008 Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems. IEEE, 129–135.
  • Yoshida et al. (2006) Shunsuke Yoshida, Haruo Noma, and Kenichi Hosaka. 2006. Proactive desk II: Development of a new multi-object haptic display using a linear induction motor. In Virtual Reality Conference, 2006. IEEE, 269–272.
  • Yung et al. (1998) Kar W Yung, Peter B Landecker, and Daniel D Villani. 1998. An analytic solution for the force between two magnetic dipoles. Physical Separation in Science and Engineering 9, 1 (1998), 39–52.
  • Zitnick (2013) C Lawrence Zitnick. 2013. Handwriting beautification using token means. ACM Transactions on Graphics (TOG) 32, 4 (2013), 53.
  • Zoran and Paradiso (2013) Amit Zoran and Joseph A Paradiso. 2013. FreeD: a freehand digital sculpting tool. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 2613–2616.
  • Zoran et al. (2014) Amit Zoran, Roy Shilkrot, Pragun Goyal, Pattie Maes, and Joseph A Paradiso. 2014. The wise chisel: The rise of the smart handheld tool. IEEE Pervasive Computing 13, 3 (2014), 48–57.

Appendix A Notation and Variables

We summarize the notations used in this paper in Table 5. Please refer to Table 6 for all weights, constants and parameters used in our implementation.

Symbol Description
State of the system at time
Input to the system at time
Cost term for optimization problem
Weight for the optimization cost terms
Cost function () for the -time prediction
Set of state constraints
Set of input constraints
Sampling time
Magnetic dipole of the electromagnet
Magnetic dipole of the pen magnet
Approx. of , with only component and
3D Vector distance from to
3D Vector distance from to
Pen tip position
Center of electromagnet, projected into paper plane
in-plane vector
unity vector that goes from
in-plane separation between and , i.e
, i.e vertical distance between and
intensity of the electromagnet (PWM input)
Actuation force on the pen, from to
Control parameter to progress along the desired path
Current desired point in the path
in-plane vector
Desired force on the pen, from to
The normalized tangent at
Table 5. List of notations used on this work.
Value 10 1.5 1.5 10. 0.1 10.

Variable Value 0.05 7. 5. 1. 1. 0.1

Table 6. List of variables and optimization weights used in this work.

Appendix B Electromagnet dipole equivalent

Here we describe the experimental validation of the dipole model for our electromagnet, that allows us to compute the force that the electromagnet exerts onto the permanent magnet on the pen. This interaction is minimal at distance , when the electromagnet and pen are located directly above each other (See Figure 5). The magnetic field generated by a dipole (electromagnet) at the position of dipole (pen) can be written as,


where the vector is the vector that goes from to (see Figure 3). The magnetic field is well described in a cylindrical system centered on the dipole and with the z-axis aligned on the direction of . Taking only the component on Eq. 29 and using the definitions of (Eq. 8) and (Eq. LABEL:eq:m2) we arrive at,


We measure the z-component of the magnetic field generated by the electromagnet to compare it with the dipole prediction of Eq. 30. We use a hall sensor (Allegro A1324, sensitivity is 5 mV/G)222http://www.farnell.com/datasheets/1538021.pdf to measure the z-magnetic flux at a fix height , where the magnet of the pen would be. Setting the electromagnet to and moving it in a grid we attain multiple readings of the hall sensor for different electromagnet positions . We present the obtained magnetic field plotted in Figure 18, top.

Figure 18. 3D overview of all data points. The x,y axis are position and the z-axis is

Due to symmetry over the z-axis we expect for , we re-plot all points as a function of distance , with the in-plane position of the hall sensor. In turn, Eq. 30 can be expressed in the form,


where we have defined two parameters used for the fitting,


The bottom plot of Figure 18 shows the measured data for magnetic flux and the fitting to Eq. 31, from which we obtained and . By replacing these values in equations 32 and 33, we observe that our system can be described by the values [A m] and [cm]. We want to emphasize the excellent agreement in Figure 18 between the experimental values and the proposed dipole model for the electromagnet. However, we should note that the experimental points show a flattening of for values of [mm], that translates into smaller values of forces in that region, as . This experimental behaviour may explain the position dispersion results we report in Section 6.3.

Appendix C Angle Aware Dipole-dipole Model

In this section we describe the derivation of the dipole-dipole model for the in-plane actuation force, in the case of considering a pen tilt of the pen. Please refer to the schematic Figure 3 for vector notations we use in this section. The coordinate system is given by,


with the in-paper-plane distance from the pen contact point to the electromagnet center projection, the vertical out-of-plane direction and the orthogonal vector to the former two.

The dipole-dipole expression for the force acting on due to and separated by is given by Eq. 3), repeated here:


The two dipoles and the vector distance between them can be expressed in the proposed coordinate system as,


and the three scalar products of equation 37,


We continue the deduction of by substituting Eq. 3843 into the main expression Eq. 37. However, by following that path we wouldn’t necessarily attain information on how strong the actuation force depends on the tilting angles and . Here we take a different path. Based on the geometry of our system, we consider the cases where the pen is tilted by only a small angle (). We introduce this small-angle approximation by keeping only the first order terms in ,


As an indication of what this approximation means, for an angle , the difference between using or or their approximations forms (Eq. 44 and 45) is 5% and 15 %, respectively. Under the small- approximation, the dipoles’ vectors are,


and the distance between dipoles,


with the length of that distance, at first order on ,


In turn, the scalar products (Eq. 4143) can be written as,


We can now substitute these expressions into the main force equation 37. As we do in Section 4.2, we consider only the terms that contribute to the component of the force. Keeping only these terms that contain up to the first order,


where we define,


Note that by considering the case in Eq. 55, we recover what we obtain in the Sec. 4.2 for . That means that the equation for we obtained in this Appendix subsumes the cases of the pen being tilted by a small angles, and it can be used in future EM actuated systems which may be able to track and .