Sensor-based control encompasses a family of methods that exploit feedback information from (typically external) sensors for controlling the robot’s motion, and in general, its behaviour. On its most fundamental form, it can trace back its origins to the servomechanism problem . Some common examples are visual servoing , tactile/force servoing [3, 4], proximity servoing , aural servoing , deformation/shape servoing [7, 8], to name a few cases.
To effectively execute these types of motion tasks, sensor-based controls invariably require some knowledge (at least approximated) of how the robot’s motor commands transform into sensory signals. This information is captured by the sensorimotor model of the robotic system , which besides coordinating action with perception, it can also be used to anticipate the effect that an input motor control will produce on the output sensor measurements . Note, however, that if this information is not known (or is highly uncertain), the robot cannot properly coordinate its motions with sensory feedback.
(which might not be known) or only compute instantaneous local estimations of it (therefore, they cannot globally describe and control the system). Compared to these computational approaches, the human brain has a remarkable degree of adaptability that allows it to learn new sensorimotor relations from birth through death and under multiple morphological and perceptual conditions (see e.g. the pioneering study ). Humans can easily coordinate hand motions through a mirror, position unknown tools attached to the body, and even recover (some) mobility after strokes.
Our aim in this paper is precisely to address the design of computational methods that efficiently provide sensor-guided robots with robust adaptation capabilities. For that, we first formulate the sensorimotor control problem of robots using uncertain perceptual/motor models. Next, we formulate various structure-based and structure-free methods to adaptively compute these unknown relations. Finally, the presented methodology is exemplified with two cases of study and discussions about its implementation are given.
The contribution of this work is that it presents a general and intuitive methodology that can be used as a guideline or even a tutorial for researchers working on adaptive sensor-based control of robots with uncertain models. The proposed control approach can be used to guide the motion of different robotic platforms (e.g. manipulators, omnidirectional robots, active robot heads) with various sensing modalities (e.g. vision, audio, thermal, attitude, proximity).
Along this note we use very standard notation. Column vectors are denoted with bold small lettersand matrices with bold capital letters . Time evolving variables are represented as , where the subscript denotes the discrete time instant, or, the iteration step. Gradients of functions are denoted as .
2.2 Control Architecture
Consider a class of fully-actuated robotic systems whose configuration (e.g. modelling the end-effector pose in a manipulator) is denoted by the vector . In our formulation of the problem, it will be assumed that the motion of robotic system is commanded via a standard position/velocity controlled interface [15, 16] (which is typically found in the large majority of commercial robotic platforms). With position interfaces, the control commands are given in terms of differential displacement motions as follows:
All methods presented in this note are formulated using the above described position controls, yet, these can be easily transformed into its velocity control equivalent by dividing both sides of (1) by the time step of the servo-loop as
2.3 Configuration Dependant Feedback
To perform a sensorimotor task, the robot is equipped with a set of sensors (not necessarily of the same modality) that continuously measure physical quantities whose values depend on the robot’s configuration. This situation means that relative robot motions produce relative sensory changes. Some examples of configuration dependent signals (measured using either external or internal/on-board sensors) are: geometric visual features, observed end-effector poses, forces applied onto a surface, proximity to an object, intensity of an audio source, temperature from a heat source, ultrasound images from probe, etc.
The feedback signal from the th sensor is denoted by the vector , where the function represents the analytical sensor model that statically relates the instantaneous configuration of the robot with the feedback signal. All these signals can be conveniently grouped into a single vector . Sensorimotor controls often require to construct a vector of meaningful features to quantify and guide the task . To this end, let us introduce the (possibly nonlinear) vectorial functional
for as the number of feedback feature coordinates (along this note, we assume that is a smooth functional). There are three cases with this configuration-dependent structure: (more controls than features), (more features than controls), and (same number of features and controls). These cases have different properties that determine the controllability of by the robot.
2.4 Sensorimotor Control Problem
The differential expression that describes how the motor actions result in changes of feedback features is represented by the first-order difference model:
for as the Jacobian matrix of the system (also known as the interaction matrix in the sensor servoing literature ), whose elements depend on the instantaneous configuration .
The sensorimotor control problem consists in coordinating the motor actions with the feedback signals such that a desired sensory behaviour is achieved. Without loss of generality, such behaviour is characterised as the set-point regulation of towards a constant sensory target . The necessary actions to approach the target can be computed by minimising the quadratic cost function111The rational behind the minimisation of (5) is to find a motor command that projects into the sensory space as a vector pointing towards .:
for as feedback gain, and as a vectorial saturation function to ensure that satisfies the differential motion condition in (1). By computing the extremum , we obtain the normal equation
which exposes the different properties of the three cases relating the relative dimensions of and .
Note that the above motor action will globally minimise (5) (i.e. ), as long as the feature coordinates in are linearly independent with respect to . This ensures that the matrix can be inverted.
For , the solution is obtained by solving the normal equation (6) for , which yields:
Substituting (8) into (5) shows that the cost function can only be locally minimised (i.e. , for ). The use of redundant features is useful in practice to cope with intermittent feedback from sensors, such as in the case of camera occlusions or malfunctions.
For the trivial case of , the matrix is square, therefore, the solution is simply obtained via standard matrix inversion .
3 Continuous Model Adaptation
3.1 Uncertain Sensorimotor Models
Computing any of the above motor actions requires some knowledge (at least coarse) of the transformation matrix , which in turn, depends on the sensor and the feature models and . However, if the estimated model is corrupted at some point in time, the robot may no longer properly coordinate action with perception. This situation may happen when the mechanical structure of the robot is altered (e.g. due to bendings or damage of links) or when the configuration of the perceptual system is changed (e.g. due to relocation of external sensors).
The capability to dynamically estimate sensorimotor models is needed to use robots in many growing (and economically important) fields such as domestic/social robotics, field robotics, autonomous systems, etc, where the sensorimotor conditions are highly uncertain. Several methods have been proposed to compute or approximate these models (see 
for a comprehensive survey on the topic). In this paper, we coarsely classify these methods into the following two approaches: structure-based estimation and structure-free estimation. In the following sections, we present the model adaptation problem and provide various solutions to it.
3.2 Structure-Based Model Adaptation
These types of algorithms represent calibration-like techniques that aim to estimate the parameters of the uncertain model. Its implementation requires exact knowledge of the analytical structure of the model , which for ease of presentation, we assume it can be linearly parametrisable with respect to as follows:222For non-linear model parametrisations, other types of optimisation algorithms (namely non-convex) must be used, whose details are beyond the scope of this expository note.:
where represents a known regression-like matrix that captures the properties of the analytical model, and whose elements depend on the configuration vector .
To compute the vector of estimated parameters , structure-based methods require to first collect a set of input-output observation points , for (see e.g. ). Standard methods use this data for computing a quadratic cost function of the following form:
for as a learning gain. By using (9) and after some algebraic operations, it is easy to show that the above function is convex with respect to the estimation error vector . Therefore, can be adaptively minimised with the gradient descent rule333Along this note, the detailed expressions of function gradients are omitted, yet, these can be easily computed after simple analytical calculations.:
which in the absence of measurement noise and for a sufficient number of linearly independent observations, it globally minimises the cost (i.e. , yet, a small estimation error is typically expected in practice). The adaptive transformation matrix is then simply computed as:
Structure-based approaches have one major disadvantage: its dependency on fixed analytical models. Note that since the model’s structure (9) is explicitly used within the adaptation algorithm (11), these methods are not robust to unknown changes in the mechanical and perceptual conditions. Furthermore, in many situations, the analytical model might not be available or subject to large uncertainties. In the case of advanced robots whose morphology is constantly evolving, fixed analytical models will clearly fail to capture the system’s properties. These issues limit the applicability of structure-based approaches.
3.3 Structure-Free Model Adaptation
These types of algorithms have the capability to compute the unknown sensorimotor model in the following manner: (i) entirely from scratch (i.e. without requiring any a-priori knowledge of the model’s analytical structure), (ii) on-demand (i.e. they can modify its acquired structure at any time instant so as to identify new relations), and (iii) from data observations only (i.e. by using information from controls and measurements only).
Based on its computation, we coarsely classify these algorithms into the following two general categories: instantaneous estimation, and distributed estimation.
As the name suggests, these techniques compute a matrix that is only valid at the current (instantaneous) configuration . The Broyden rule  is one possible example of such technique. It iteratively computes with the following update rule:
for , and as a tuning gain. With “high” gains , by right-multiplying (13) by (namely, projecting the motor action into sensory space), we can see that is satisfied. However, using high gains results in a noisy and rapidly changing matrix . For slow robot motions, the Jacobian matrix is expected to change slowly, therefore, using “small” gain values for can help to make the computation less responsive, i.e. as well as to filter out noisy measurements.
Another example of these techniques can be derived from the following (convex) cost function :
which provides a metric of the accuracy of at the current configuration . The terms of this unknown matrix are continuously adapted with the rule:
where the scalar denotes th-row th-column term in .
With instantaneous estimation techniques, the matrix must be continuously recalculated with new sensor observations as the robot moves into other configurations. These types of adaptation techniques do not provide a mechanism for preserving knowledge (i.e. remembering models) of previous configurations.
Note that since the feedback feature functional (3) is smooth (i.e. differentiable), its Jacobian matrix is expected to smoothly change along the configuration space . This means that a local Jacobian matrix estimated at a particular configuration point is also valid around the neighbourhood surrounding it. This simple, yet powerful, idea forms the working principle of distributed estimation techniques. With these adaptive algorithms, the estimation problem is shared amongst multiple computing units that specialise in a local transformation.
There are many possible implementations of distributed estimation algorithms. Along this note, we briefly describe a variety based on self-organising maps (see  for details). Consider a network of computing units spread around the robot’s configuration space . The following data structure is associated with each computing unit:
where stands for a local approximation of estimated at the configuration point . There are various methods for establishing the distribution of these units around the robot’s workspace, e.g. based on self-organising rules, evenly distributed locations, random point distributions, etc. . For ease of presentation, we assume that the location of these configuration points associated with the units has already been established.
Estimation of local transformation matrices is performed by first collecting a data set of observation points , for . Then, the following local cost function for the th unit is computed:
for as a local ball centred at the th unit, and as its Gaussian neighbourhood function computed as
where determines the ball’s radius. In this method, the idea is to make use of neighbouring data (whose contribution decreases with its distance to the centre unit ) for approximating the local transformation matrix. The update rule to adaptively compute the th row th column term of is as follows:
Once the local cost functions (17) have been minimised, the network is trained to perform sensorimotor transformations with each of its units. In order to implement motor commands as the ones derived in Sec. 2.4, a local transformation matrix that best matches the current position must be retrieved from the units. This can be done by solving the search problem
Note that this adaptation approach can be combined with the previous instantaneous estimation technique (or possibly others) by defining a cost function that exploits the current feedback measurements. This allows to also quantify the accuracy of the model based on new sensory data; the cost function can then be minimised with similar gradient descent tools as before.
Compared to the previous estimation approaches, distributed estimation requires more data to approximate the sensorimotor model. However, these methods can effectively preserve local knowledge within its computing units and does not require any prior analytical representation of the robot’s action-to-perception relations.
4 Cases of Study
4.1 Visual Shape Servoing of Elastic Objects
To exemplify our methodology, consider the setup depicted in Fig. 1, where a 3D camera captures point clouds of a beam-like elastic object manipulated by a robot. Let us denote the captured 3D points by , for (note that the number is generally in the order of hundreds). The task to be performed is to automatically deform the object into a desired shape. We can use the point cloud to approximate a the object’s backbone (represented as the blue curve in Fig. 1). With this geometric information, we compute the feature vector defined as follows:
for as the object’s curvature, and as the angle of the object’s bending with respect to it’s frame, see  for details. For this task, model adaptation can be performed with distributed estimation algorithms. These approaches provide an efficient solution to the highly nonlinear transformation problem of relating robot poses to object deformations (note that deformation models are hard to compute analytically). For that, several computing units must be first defined at key end-effector poses, e.g. ranging from fully stretched to varying bending configurations; local sensory observations can then be collected for approximating the model. Since , the motor command is computed as in (7), with a target shape defined as .
4.2 Multi-Modal Scanning with Ultrasound Probes
Consider the setup in Fig. 2, which depicts a robot performing automatic scanning of tissues with an ultrasound probe  (we assume the robot has 6-DOF). This system is instrumented with a force/torque sensor and a 3D camera. Let us denote by the (normal) force applied onto tissues, by the image location of the ultrasound feature of interest, and by the probe’s 3D orientation. The task to be performed is conveniently described with respect to the body’s 3D frame. It consists in positioning at the centre of the ultrasound image, while applying a desired normal force and controlling the probe’s pose over the tissues. Note that this relative orientation can be computed from the 3D point clouds . The task’s feature vector is defined as
The models for the above feature coordinates are simple to analytically derive, namely using Hooke’s law for , horizontal image displacements for , and affine/homogeneous transformations for . Therefore, model adaptation can be performed using structure-based algorithms as in (9). With these approaches, we can robustify the sensor-guided task by continuously calculating unknown task parameters such as: stiffness of soft tissues, relative location of ultrasound features, and robot-camera-body transformations. Since , the motor command is also computed as in (7), with a set-point feature defined as .
In this expository paper, we addressed the problem of computing adaptive sensorimotor models for robots with uncalibrated sensory feedback and/or uncertain morphology. A general sensor servoing approach was first formulated using an energy minimisation approach. Then, we derived various methods for providing these controllers with continuous model adaptation capabilities. Two cases of study we presented to illustrate the proposed methodology.
The presented sensorimotor controls are formulated based on the assumption that the feedback signals dependent on the robot’s configuration only. Although this condition can be fairly used to represent many sensor-guided applications, it may not be the most accurate model for describing tasks where the measurements also depend additional variables (e.g. manipulating fabrics with infinite dimensional configurations) or even time-varying states (e.g. controlling the effect of cosmetic lasers stimulating skin tissues). The development of more general sensor models is still an open research problem.
The presented model adaptation methods allow robots to perform sensor-guided tasks even when its sensorimotor model is not known or might suddenly change. For example, robots can adapt to unknown sensor configurations and/or morphologies. By understanding the principle of how sensors models can be effectively created from scratch and adapted on-the-fly, we hope to build machines with more resilient properties that allow them to perform long-term operations with minimal supervision. These advanced capabilities are needed to advance towards building truly autonomous robots.
-  E. Davison and A. Goldenberg, “Robust control of a general servomechanism problem: The servo compensator,” IFAC Proceedings Volumes, vol. 8, no. 1, pp. 231 – 239, 1975.
-  F. Chaumette and S. Hutchinson, “Visual servo control. Part I: Basic approaches,” IEEE Robot. Autom. Mag., vol. 13, no. 4, pp. 82–90, 2006.
-  Q. Li, C. Schurmann, and R. H. R, “A control framework for tactile servoing,” in Proc. Robotics: Science and Systems, vol. 1, 2013, pp. 1–8.
-  M. Prats, P. J. Sanz, and A. P. del Pobil, “Reliable non-prehensile door opening through the combination of vision, tactile and force feedback,” Auton. Robots, vol. 29, no. 2, pp. 201–218, 2010.
-  S. E. Navarro, B. Hein, and H. Wörn, “Capacitive tactile proximity sensing: From signal processing to applications in manipulation and safe HRI,” Soft Robot., pp. 54–65, 2015.
-  A. Magassouba, N. Bertin, and F. Chaumette, “Audio-based robot control from interchannel level difference and absolute sound energy,” in Proc. IEEE Int. Conf. Intelligent Robots and Systems, 2016, pp. 1992–1999.
-  D. Navarro-Alarcon, Y.-H. Liu, J. G. Romero, and P. Li, “On the visual deformation servoing of compliant objects: Uncalibrated control methods and experiments,” Int. J. Robot. Res., vol. 33, no. 11, pp. 1462–1480, 2014.
-  D. Navarro-Alarcon and Y.-H. Liu, “Fourier-based shape servoing: A new feedback method to actively deform soft objects into desired 2D image shapes,” IEEE Trans. Robot., vol. 34, no. 1, pp. 272–1279, 2018.
-  M. Haruno, D. M. Wolpert, and M. Kawato, “Mosaic model for sensorimotor learning and control,” Neural Computation, vol. 13, no. 10, pp. 2201–2220, 2001.
-  R. Pfeifer and C. Scheier, “Sensory-motor coordination: The metaphor and beyond,” Robot. and Auton. Syst., vol. 20, pp. 157–178, 1997.
-  G.-Q. Wei, K. Arbter, and G. Hirzinger, “Active self-calibration of robotic eyes and hand-eye relationships with model identification,” vol. 14, no. 1, pp. 158–166, 1986.
-  H. Wang, Y.-H. Liu, and D. Zhou, “Adaptive visual servoing using point and line features with an uncalibrated eye-in-hand camera,” IEEE Trans. Robot., vol. 24, no. 4, pp. 843–857, Aug. 2008.
-  K. Hosoda and M. Asada, “Versatile visual servoing without knowledge of true Jacobian,” in Proc. IEEE/RSJ Int. Conf. Intelligent Robots and Systems, vol. 1, 1994, pp. 186–193.
-  I. Kohler, “Experiments with goggles,” Scient. American, vol. 206, no. 5, pp. 62–73, 1962.
-  D. Whitney, “Resolved motion rate control of manipulators and human prostheses,” IEEE Trans. Man-Mach. Syst., vol. 10, no. 2, pp. 47–53, Jun. 1969.
-  B. Siciliano, “Kinematic control of redundant robot manipulators: A tutorial,” Journal of Intelligent and Robotic Systems, vol. 3, no. 3, pp. 201–212, 1990.
-  A. Cherubini, R. Passama, P. Fraisse, and A. Crosnier, “A unified multimodal control framework for human-robot interaction,” Robot. and Auton. Syst., vol. 70, pp. 106 – 115, 2015.
-  Y. Nakamura, Advanced robotics: redundancy and optimization. Boston, MA: Addison-Wesley Longman, 1991.
-  O. Sigaud, C. Salaün, and V. Padois, “On-line regression algorithms for learning mechanical models of robots: A survey,” Rob. Auton. Syst., vol. 59, no. 12, pp. 1115 – 1129, 2011.
-  D. Navarro-Alarcon, H. M. Yip, Z. Wang, Y.-H. Liu, W. Lin, and P. Li, “Adaptive image-based positioning of RCM mechanisms using angle and distance features,” in IEEE/RSJ Int. Conf. on Intelligent Robots and Systems, 2015, pp. 5403–5409.
-  T. Kohonen, Self-Organizing Maps. Springer Berlin Heidelberg, 2001.
S. Haykin, Neural Networks and Learning Machines
, ser. Neural networks and learning machines. Prentice Hall, 2009, no. 10.
-  D. Navarro-Alarcon, O. Zahra, C. Trejo, E. Olguin-Diaz, and V. Parra-Vega, “Computing pressure-deformation maps for braided continuum robots,” Frontiers in Robotics and AI, vol. 6, p. 4, 2019.
-  M. Victorova, D. Navarro-Alarcon, and Y. Zheng, “3D ultrasound imaging of scoliosis with force-sensitive robotic scanning,” in IEEE Int. Conf. Robotic Computing, Feb 2019, pp. 262–265.