Transparent surfaces, such as glass, transmit most of the visible light that falls on them. For a common plate of glass, only about 4% of the incident light are reflected from each surface while the other 92% are transmitted across. This non-opaque characteristic prevents common vision sensors from locating transparent objects with high accuracy. Given the prevalence of transparent objects in industrial and home environments, such a limitation hinders the application of many vision-based robotics systems.
Most robotics research into the detection/recognition of transparent objects is only concerned with coarse localisation – typically, centimetre-accuracy in indoor navigation or tens-of-centimetre-accuracy in outdoor navigation. By contrast, industrial tasks (transport, assembly, glue dispensing on car glasses, etc.) require a much higher level of accuracy, typically of several millimetres or submillimetre. Furthermore, most of the existing methods in transparent objects shape and pose estimation require elaborate apparatus setup (e.g. specific and fixed configurations of multiple sensors).
Here, we propose a method to locate glass objects to millimetre accuracy using a simple LRF attached to the robot end-effector. The proposed method can locate both planar and non-planar objects for subsequent manipulation. Briefly, the proposed localisation and manipulation pipeline is as follows:
Registration: Perform Iterative Closest Point (ICP) registration of the points against an available 3D model;
Manipulation: Based on the object pose estimate, compute the robot trajectory to perform the manipulation (here: following the object contour mimicking a glue dispensing task).
The remainder of the paper is organized as follows. In Section II, we discuss several works done in localisation of transparent objects. In Section III, we survey the background knowledge on laser-glass interactions. In Section IV, we present our glass localisation method. Sections V and VI report experimental results on respectively point and full-object localisation. Finally, in Section VII, we conclude and sketch directions for future work.
Ii Previous work on localisation of transparent objects
Most research into transparent object recognition does not require the estimated object to be manipulated upon. As such, the estimated models are often simplified and lack the necessary detail for accurate manipulation. Furthermore, most of the methods require elaborate apparatus setup.
, Hata used a structured light setup to project stripe patterns into transparent object with one flat side. The refracted pattern is then inspected and extracted using imaging sensor. Genetic Algorithm is then iteratively employed to construct a 3D model until the calculated error is within a threshold. The author achieved shape accuracy below 20% after running 50 generations for six hours.
In , the authors tracked features from different frames of a moving camera to estimate shapes and poses of transparent objects. The shape reconstruction assumes parametric form of the transparent object. They showed that the problem can be highly non-linear even for object with simple shape. Achieved shape parameters error range from submillimetre to tens-of-centimetre while there was no data on real experimental pose parameters.
In , Narita proposed a non-contact optical method for measuring the 3D shape of transparent objects and their refractive indices using LRF. The method relies on observing and triangulating the two strongest reflected light rays. The author could measure thickness of two transparent object of simple shapes to submillimetre error, obtained through non-trivial arrangement of multiple sensors.
In , Kutulakos calculated depth maps of opaque and transparent specular objects by reconstructing light paths from a light source. The number of viewpoints required depends on the number of times the light get redirected. An experiment to reconstruct a diamond-shaped glass object with eight faces and from five viewpoints was done. The average RMS distance from the obtained points to the estimated plane is 1.33 mm.
In the robotics field, transparent object detection is of special interest for mobile robot localisation especially to avoid collision. Developments in non-flat object recognition through robotic arm haptic-based approach are of interest as well, which can be generalized to transparent objects. These approaches hold promise with their non-reliance on external vision sensors, which tend to fail transparent objects.
In , Foster tackled the problem of mobile robot localisation in an environment with many glass obstacles using a LIDAR sensor. The author looked into the behaviour and working of LIDAR sensors to recognise the visibility of glass at certain view angles and conditions. These view angles are later experimentally identified and used in modifying the standard occupancy grid algorithm. The algorithm looks for evidence that a transparent object occupies a cell but does not estimate the pose of occupying object.
, Allen employed Utah-MIT hand fitted with tactile sensor pads that was attached to PUMA 560 manipulator. Borrowing previous research on human haptic system, they extended human hand movement strategies in discovering 3D objects attributes to robotic domain. The idea was to generate a rough initial shape estimate using Grasping by Containment before performing additional exploratory procedures (EPs) to refine the structure.
In , Ibrayev developed an invariant-based method to recover surface geometry with few data collected from touch sensors. Firstly, curve class recognition is done by calculating differential invariants for few classes of quadratic curves and special cubic curves. From there, actual curve is estimated based on the shape parameterisation as well as contact locations on the curve. The author claimed an average relative error of 1% depending on the curvature and its derivative estimation performance.
Iii Laser-glass interactions
Iii-a Triangulation-based laser scanners
The triangulation-based laser scanner works by projecting a laser line beam onto the object surface to be determined. The reflected light is then subsequently captured by the scanner’s receiver. In our application, the scanner projects laser line onto the object at right angle while the receiver is pointed towards the reflected line from an angle as seen in Figure 2 (left). In the sensor’s receiver, the reflected light will fall on an array of Charge-Coupled Device (CCD), from which the object position is calculated.
Most of the light that falls onto the transparent material either gets reflected or transmitted. The reflected light consists of two different types: specular and diffuse, as shown in Figure 2 (right). In the former, the incident and reflected rays’ angles are equal but located on opposite sides of the reflected surface normal. In the latter, the incident ray is scattered to reflected rays that cover many angles. Meanwhile the transmitted ray might be transmitted again through the rear surface or get refracted multiple times internally before being transmitted out from either surfaces.
Iii-B Laser reflection from glass surfaces
The laser scanner’s ability to register points from glass surfaces depends on several main factors, two of which are: laser reflection type and presence of opaque object in the background.
Specular reflections that reach the receiver sensor tend to saturate the CCD, causing blooming effect  as shown in Figure 3. The saturated pixels will lead to inaccurate reading and hence false points being registered. Avoiding specular reflections can be challenging especially when dealing with non-planar glass with varying surface normal angles.
On the other hand, the scanner’s receiver relies on diffuse reflection of the laser line to capture the surface contour. However, the diffuse reflections from the glass surface tend to be very low. As previously mentioned, only about 8% of the laser light get reflected back from both surfaces. Only part of this is reflected diffusely due to glass surface imperfection or small particles. Depending on the laser scanner, some setting adjustment on exposure time or minimum intensity threshold might be required to pick up these low intensity points.
Presence of opaque surfaces in the background can also cause the scanner to register false reflections. This problem is sometimes referred to as the Mixed Pixels problem in literature . In some cases involving LIDARs, the range measurement with higher intensity has a higher chance of being selected . The author argued that the intensity of return from glass can overpower the opaque background object only for certain critical angle near the glass normal for a constant distance between the two objects. An example given was that when a diffuse wall was placed 1 m behind the glass, the glass will only be visible from a small 0.25 degree angle range.
In Figure 4, we show the obtained reading from the glass surface taking the above factors into account. Figure 3(a) shows the sparse 3D points constructed by moving the robotic arm to scan along a flat glass surface marked in Figure 3(b). The LRF scanning properties used are listed in Table I.
|Angle (from normal)||2|
|Exposure time||5 ms|
|Selection method||highest intensity|
|Distance from glass to opaque surface||57 cm|
. The point distribution is characterised by distinct inclined spikes which mask the true surface flatness. Outlier points are also registered along the thickness of the glass at different parts of the glass. In Figure4(b), we show the noisy reading obtained when specular reflection is involved. The glass panel spans the horizontal middle section of the image, with irregular points distributed above and below it. All of these observations point towards the unpracticality of the scanned output.
In section IV, we propose an alternate method to realize our glue-dispensing robotic system that does not rely on such irregular output.
Iii-C Laser reflection from glass edges
While the transparent surface of the car glass window gives mostly unreliable readings, the edge offers a better compromise. Most glass panels, such as the car window, have higher surface irregularities along the border that can be attributed to applied glass edge finishing which is often required for aesthetics and safety reasons. This physical property allows lasers that fall on the less transparent glass edge to give out more diffuse reflection. This is desirable as it provides cleaner and hence more accurate range measurement.
There is, however, the uncertainty whether laser line projection could return points from the edge line reliably. To understand this, we devised an experiment to assess the distribution of laser line on the transparent-opaque surface intersection as seen in the bottom of Figure 6:
Tape is laid along the border and onto the glass surface. The laser reading obtained this way reflects the true contour of the glass.
A border cover object is placed on the glass surface that coincides with the edge line. Another laser reading is then obtained.
After removing the tape and cover object, another laser reading is obtained.
Repeat the experiment for various scan angles.
In both Figures 6(a) and 6(b), the laser reading from Step 1 and 2 are traced by the blue crosses and green squares respectively. The first intersection from the left between the blue and green plots corresponds to the point on the edge line and the laser line readings obtained from Step 3 are visualised as red plots. It was found that the selection of scanning angle affects the degree at which the laser points coincide with the edge line. In Figure 6(b), laser points from a 40 degrees scanning angle above the horizontal plane display closer proximity to the edge line compared to laser points from a 10 degrees scanning angle in Figure 6(a).
Iv Glass pose estimation pipeline
Based on the observations of the previous section, we propose the following pipeline:
Registration: Perform Iterative Closest Point (ICP) registration between the points against an available 3D model;
Manipulation: Based on the object pose estimate, compute the robot trajectory to perform the manipulation (here: following the object contour mimicking a glue dispensing task).
We position the laser scanner at different poses along the glass border with the laser line intersecting the edge. At each pose, the scanner receives a set of points from the laser line being reflected off the glass edge. We then proceed to filter these collinear points to obtain a single point which we call the Border Identifier Point (BIP). The BIP filtering method should ensure consistent point location among the collinear points for all the poses. For our application, we ensure that the BIP selected is the highest point among the collinear points above the ground plane. This point corresponds to the point on the intersection between the transparent glass surface and diffuse glass edge (refer to the illustration on top of Figure 6). Hence, in the end we obtain set containing BIPs from scanner configurations.
Our proposed method requires a 3D model of the glass object to be available. If unavailable, pre-measures can be employed to augment the 3D model construction. For instance, coating materials such as talcum powder can be applied on the surface to obtain accurate scan measurement. Construction of the 3D model need only be done once and hence will not affect subsequent glue-dispensing operation time.
ICP registration  is then performed to obtain the estimated pose of the car window. The ICP algorithm works in principle by matching two cloud points, in this case the cloud points formed from against the constructed 3D model, . Through minimizing the distance distribution, the algorithm returns a transformation that aligns the two models as seen in Figure 8.
V Experiment 1: point localisation
The experiment can be briefly summarised as follows: The robot aligns the laser line with the point-of-interest before moving the tooltip to touch it. The resulting distance from the point-of-interest to the final tooltip position is then recorded.
We perform this experiment on two different object profiles for comparison: Firstly on glass edge and later on an opaque object’s edge with diffuse reflection. Through comparison of the two results, we can obtain the relative accuracy of scanning on glass edge against opaque object edge as ground truth.
The laser scanner used is a 2D LRF, Microepsilon Scancontrol 2600-100, with 658 nm red semiconductor laser as light source. The LRF transformation parameters relative to the robot’s flange is calibrated based on method described in SCALAR .
The experiment steps are as follows:
Position the laser beam line to intersect the object edge at multiple locations spanning the perimeter and calculate the BIP point at each location, where represents the robot’s base coordinate frame.
At each location, mark the point on the object under the laser beam line which is supposed to coincide with the respective and move the tooltip to touch it while keeping a predefined orientation of the tooltip, . Record the joint states when the tooltip touches the desired point as .
Program the robot to move towards each with as approach orientation. Record the resulting joints states as .
Let the Forward Kinematics mapping from the configuration-space to task-space be and the tooltip transformation relative to the robot’s flange be .
The localisation error can be defined as follows
One can then define the mean and maximum errors across all tested points.
V-B Results and discussion
The table below compares the performance result for the two object profiles:
|Mean error ( std) (mm)||1.01 ( 0.14)||0.94 ( 0.28)|
|Max error (mm)||1.27||1.35|
The two experimental results show minimal discrepancy suggesting similar performance in point localisation. Such an outcome provides validation that scanning on glass edge offers comparable reliability to scanning on opaque objects. Henceforth, we can make use of glass edge measurement reading as input to glass pose estimation.
Vi Experiment 2: full pipeline
To validate the accuracy of the full task pipeline, an experiment is devised to measure the accuracy of the transparent object pose estimation. For visual guidance, we refer the reader to Figure 9.
The following validation steps are performed following the steps in Section V:
Perform ICP on the previously obtained points to obtain transformation matrix that transform to the estimated pose in the workspace.
For every , we calculate the nearest point in as .
Program the robot to move towards with as approach orientation. Record the resulting joint states as .
The localisation error is defined as
One can then define the mean and maximum errors across all discretised points along the glass border.
Vi-B Results and discussion
In the following Table III, we list down the average operation time for each stage making up the full glue-dispensing operation with =4:
|Stage||Average time (s)|
While scanning and execution stages contribute significantly to the total operation time, the values presented in the table do not reflect the best time achievable as they are highly dependent on the robot’s velocity. In our experiment, we limit the velocity and acceleration of the robot for safety reason.
The table below presents the pose estimation result based on the validation metric mentioned previously with =12:
|Mean error ( std) (mm)||0.88 ( 0.39)|
|Max error (mm)||1.65|
In this paper we have looked into the feasibility of accurately locating transparent object using a 2D LRF with the ultimate goal of having a 6 DOF robot manipulating on it. Based on a physical understanding of laser-glass interactions, we introduced a method that combines 6 DOF robotic arm with 2D LRF to estimate a transparent object pose in the workspace by looking at the object edge. The method was tested in an industrial scenario which emphasises the need for fast and accurate online operation. Experiments showed that our method could achieve errors in the order of the millimetre, making it suitable for many industrial scenarios, such as manipulation or glue dispensing.
Future work will include integration with upstream coarse localisation. The accuracy can also be further improved by performing a more complete calibration of laser-in-hand robotic system.
This work was supported in part by NTUitive Gap Fund NGF-2016-01-028 and SMART Innovation Grant NG000074-ENG.
-  S. Hata, Y. Saitoh, S. Kumamura, and K. Kaida, “Shape extraction of transparent object using genetic algorithm,” in icpr, p. 684, IEEE, 1996.
M. Ben-Ezra and S. K. Nayar, “What does motion reveal about transparency?,”
Proceedings Ninth IEEE International Conference on Computer Vision, pp. 1025–1032 vol.2, Oct 2003.
-  D. Narita and M. Baba, “Measurement of 3-d shape and refractive index of a transparent object using laser rangefinder,” in Instrumentation and Measurement Technology Conference, 2005. IMTC 2005. Proceedings of the IEEE, vol. 3, pp. 2247–2252, IEEE, 2005.
-  K. N. Kutulakos and E. Steger, “A theory of refractive and specular 3d shape by light-path triangulation,” International Journal of Computer Vision, vol. 76, no. 1, pp. 13–29, 2008.
-  P. Foster, Z. Sun, J. J. Park, and B. Kuipers, “Visagge: Visible angle grid for glass environments,” in 2013 IEEE International Conference on Robotics and Automation, pp. 2213–2220, May 2013.
-  P. K. Allen and P. Michelman, “Acquisition and interpretation of 3-D sensor data from touch,” IEEE Transactions on Robotics and Automation, vol. 6, no. 4, pp. 397–404, 1990.
-  R. L. Klatzky and S. J. Lederman, “The intelligent hand,” in Psychology of learning and motivation, vol. 21, pp. 121–151, Elsevier, 1988.
-  R. Ibrayev and Y.-B. Jia, “Tactile recognition of algebraic shapes using differential invariants,” in 2004 IEEE International Conference on Robotics and Automation, vol. 2, pp. 1548–1553 Vol.2, April 2004.
-  Y. M. Amir and B. Thörnberg, “High Precision Laser Scanning of Metallic Surfaces,” International Journal of Optics, vol. 2017, 2017.
-  C. Ye, “Mixed pixels removal of a laser rangefinder for mobile robot 3-d terrain mapping,” in 2008 International Conference on Information and Automation, pp. 1153–1158, June 2008.
-  D. Holz, A. E. Ichim, F. Tombari, R. B. Rusu, and S. Behnke, “Registration with the point cloud library: A modular framework for aligning in 3-D,” IEEE Robotics & Automation Magazine, vol. 22, no. 4, pp. 110–124, 2015.
-  R. Diankov, Automated Construction of Robotic Manipulation Programs. PhD thesis, Carnegie Mellon University, Robotics Institute, August 2010.
-  T. S. Lembono, F. Suárez-Ruiz, and Q. Pham, “Scalar - simultaneous calibration of 2d laser and robot’s kinematic parameters using three planar constraints,” in 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 5570–5575, Oct 2018.