A Machine learning approach for Shape From Shading

I Introduction

Several researchers are working to solve the Shape From Shading(SFS) problem [1, 2, 3], but there are no solutions that give good results on real images. Even with complex synthetic images, several constraints must be defined[4].

The methods of SFS can be classified into three categories: The first category concerns the local resolution methods (pontland

[5][6], lee and Ronselfd[6], tsai and shah[6]), in which, the computing of the surface orientation of each pixel is principally given by the gray level information of its neighbors. The second category is the global resolution methods in which the resolution is calculated using all the pixels of the image[4, 7], by passing over each pixel several times. the third category is Mixed methods.

In this paper we are interested to the needle map integration methods [8, 9], in which the resolution is given by two steps: the generation and then the integration of the needle map for the surface reconstruction (each step can be local global or mixed.

the needle map represents the set of normals corresponding to all the pixels of image. We propose a method for generating the needle map by using a Machine learning method. This method is composed of three phases: The first phase is the generation of the 3D object. The second phase is the preparation of the database examples (offline).In the third phase we use the database examples to generate the needle map of each pixel (online), it is belong to the local resolution methods.

Among the methods dealing with Shape from shading, we mention the Puntlands method [6, 5], he proposed the first method to solve SFS problem using a local method. He chooses to direct all points of the image using the angles SLANT and TILT. There is also the method of lee and Ronselfd that follows the Puntlands method [10, 6, 11], but using a perspective camera and light source at the infinity. A.sethian[8] is the first who applied the level set method to solve the SFS problem. His method uses the depth function (Z(x,y)) to generate the levels curves, and also jean Denis [12] used this category where he suggested a perspective camera and light source at the infinity.

Like most methods proposed in SFS problem, we suppose that: the surface is smooth, the image taken by a parallel projection camera, light source at the infinity and regular surface (lambertian).

The paper is structured as follows: After the introduction section that offers a range of methods for solving the SFS problem with their classification. Section two summarizes the basic concepts for the image formation and explains the mathematical equations used. The third section details the proposed technique based on the Machine learning method. The last experimenting section gives the obtained results after applying our approach on synthetic and real images.

Ii Basic notions of image formation

SFS is the process of generating a three-dimensional shape using a single two-dimensional image, which is the reverse process of the image formation.

Ii-a notations and definitions

In this section we will define some basic notion and notations (see fig.1):

Normal vector $N (N_{x}, N_{y}, N_{z})$ : Is the perpendicular vector to the surface in a point (x,y).
light source vector $S (S_{x}, S_{y}, S_{z})$ : Is the vector which represents the direction of the light source and its intensity, the direction of S is towards the light source.
needle map: Represents the set of the normals corresponding to all the pixels of image.
SLANT ( $θ$ ): is the angle between the origin (the X-axis) and the projection of the normal on the plane (x,y).
TILT ( $ϕ$ ): Is the angle between the normal vector and Z-axis (see figure 1).
Boundary Condition: The characteristic (solution) of the pixels located at the boundaries of the object is known.
Neumann boundary condition: The solution in Neumann Boundary Condition is the gradient (in our case the gradient of the depth Z).
Albedo: Reflectance factor (ratio of light emitted and light received).
Lambert surface: Surface that reflects radiation uniformly in all directions.
Gradient of depth (Z):depth variation (the derivative of Z with respect x and y).
Singular point: pixel has a maximum illuminance.

Fig. 1: TILT and SLANT of normal and light source vectors

Ii-B Image formation

The image formation consist to study the generation of images from objects (see fig.2). This process is used in cameras. We can generate an image form a 3D object by using the basic equation of the images formation for more details see [4]. The illumination E is:

E = \frac{α}{4} (\frac{p}{f})^{2} I {cos}^{4} α L

(1)

Fig. 2: Perspective model of the camera

The luminance (brightness) of a Lambert surface can be expressed as follows:

L = \frac{ρ}{π} (\to N . \to S)

(2)

The illumination (E) can expressed using the angle $(α)$ between the two vectors(N and S) [8]:

E = K (c o s (α))

(3)

K is a constant $(K = \frac{ρ}{4} (\frac{p}{f})^{2} I)$ , $ρ$ is the albedo of the surface. In a singular point both vector N and S are equal, $(α = 0)$ , the illumination in this point is maximum $E_{m a x} = K$ . The equation ( 3) can be written as [8]:

c o s (α) = \frac{E}{E_{m a x}}

(4)

Ii-C Problem modeling

Our method belongs to the category of integration methods which consists of reconstructing the scene in two steps. Generate the needle map then construct the scene from it. The following diagram in figure 3 shows the different Steps for the reconstruction.

Fig. 3: Steps for the 3D reconstruction

Ii-D Relation between normal vector, SLANT and TILT angles

In our approach, we propose the generation of the needle map using the SLANT and TILT angles. We can compute the normal vector from the two angles (TILT and SLANT) using the formula: $N = (s i n (ϕ) c o s (θ), s i n (ϕ) s i n (θ), c o s (ϕ))$

As explained previously, the TILT angle is equal to the angle between the normal N and the light source. S, $T I L T = α = a r c c o s (\frac{E}{E_{m a x}})$ . We propose a method to compute the SLANT (the angle between the project vector of N on image plane and the vector N) using Machine learning under some constraints. We assume that the surface is Lambertian, differentiable, continuous with a punctual light source located at infinity, taken by a camera with a parallel projection.

Iii 3D reconstruction using A Machine learning approach

Iii-a Machine learning approach

Our method is divided into three phases: the first phase is the generation of the 3D shape from mathematic functions. The second phase is the preparation of the examples Database (offline). Third phase uses the database examples to generate the needle map of the test images (online).

Iii-A1 Generation of 3D objects for learning

In this phase we will generate the training data, these data contain the inputs (gray levels) and outputs (SLANT). In order to generate the data we need the 3D objects and their gray level images.

In this work we will create 3D objects by two methods: The first method is based on the generation of 3D surface from mathematic functions. Figures 6, 7 and 8 show three 3D objects generated by the functions $f 1 (x, y) = - x^{2} - y^{2}, f 2 (x, y) = x * e x p (x^{2} - y^{2})$ and $f 3 (x, y) = s i n (x) + s i n (y)$ respectively. The second method uses 3D surfaces defined by their depth (Z) for example the silt (Figure.9) Mozart (Figure10) and penny (Figure11). After the generation of the depths (Z of each pixels) we will calculate the TILT and SLANT angles, and then we will generate the image corresponding to each 3D object. We will finally get results as a set of pixels and corresponding angles.

Iii-A2 Offline phase

The purpose of the offline phase is to create a database containing several examples. Each example contains:

Input:
- The gray level of the pixel (i,j)
- The gray level and the SLANT of three adjacent neighbors of pixel (i,j)
Output: The SLANT of pixel (i,j)

For example in figure 4, we have a pixel (i,j) and three adjacent neighbors, The database contains eight fields, seven input and one output, the inputs are the Gray level of ${(i, j); (i, j + 1); (i + 1, j); (i + 1, j + 1)}$ and the SLANT of ${(i, j + 1); (i + 1, j); (i + 1, j + 1)}$ . The SLANT of(i,j) is the output.

Fig. 4: Database example

Iii-A3 Online phase

In our approach we will use the Neumann boundary condition, so it is assumed that the normal of edge around the treated area in the image is known (figure 5). All pixels with three adjacent neighbors known SLANT are ready to find its SLANT. The search is done by the Euclidean distance between the ready pixels and each element of examples database, and choose the minimum distances.

$D (P (i, j), E (i, j)) = \sqrt{3 \sum k = 1 (G L n P (k) - (G L n E (k))^{2} + (S n P (k) - (S n E (k))^{2}}$

such as:

$P (i, j)$ : is a ready Pixel.
$E (i, j)$ : is a database example .
$G L n P (k)$ : is the gray level of $k^{t h}$ neighbor of P(i,j)
$G L n E (k)$ : is the gray level of $k^{t h}$ neighbor of E(i,j)
$S n P (k)$ : is the SLANT of $k^{t h}$ neighbor of P(i,j)
$S n E (k)$ : is the SLANT of $k^{t h}$ neighbor of E(i,j)
$G P$ is the gray level of P(i,j)

Fig. 5: Boundary condition

Iii-B Integration of the needle map for the scene reconstruction

There are several methods to integrate the normal field, J.Denis[9, 13] shows some iterative and no-iterative methods of the normal field integration. Iterative methods are slow but give better results. We use in the following the method of Horn and Brooks[9], it is simple, easy to implement and gives a good results. The equation of integration is follows:
$Z_{i, j}^{k} = \frac{Z_{i + 1, j}^{k} + Z_{i, j + 1}^{k} + Z_{i - 1, j}^{k} + Z_{i, j - 1}^{k}}{4} - \frac{δ}{8 g} (p_{i + 1, j} - p_{i - 1, j} + q_{i, j + 1} - q_{i, j - 1})$

p and q are calculated using SLANT( $θ$ ) and TILT( $ϕ$ ).

$(p, q) = (\frac{s i n (ϕ) c o s (θ)}{c o s (ϕ)}, \frac{s i n (ϕ) s i n (θ)}{c o s (ϕ)})$

Iv Experiments

The results of our approach is depend to the learning phase (offline), we will test on many examples database generated from different functions. Figures 6, 7 and 8 show four objects and corresponding images, these objects are generated using mathematics functions, $f 1 (x, y) = - x^{2} - y^{2}, f 2 (x, y) = x * e x p (x^{2} - y^{2})$ and $f 3 (x, y) = s i n (x) + s i n (y)$ . Figures 9, 10 and 11 shows three objects generated from the depths matrix and corresponding images. In all this cases the images are generated from objects using Equation 3.

Fig. 6:

f 1 (x, y) = - x^{2} - y^{2} / {x, y} \in [- 1, 1]

Fig. 7:

f 2 (x, y) = x * e x p (x^{2} - y^{2}) / {x, y} \in [- 2, 2]

Fig. 8:

f 3 (x, y) = s i n (x) + s i n (y) / {x, y} \in [- 6, 6]

Fig. 9: Depth matrix of silt

Fig. 10: Depth matrix of mozart

Fig. 11: Depth matrix of penny

We study six different cases of database, for each case we apply different function for the off-line and the on-line process:

Iv-1 off-line: the function f1, on-line the image generated by f2

First we will generate examples of function f1, knowing that the examples are disjoint, function f1 can be generate 4625 examples. The test result on the image generated by the function f2 is shown in Figure 12. The average of the distance of all pixels equals 0.07

Fig. 12: Result of image generating by f2

Iv-2 off-line: the functions:f1 and f2, on-line the image generated by f3

Now we use the functions f1 and f2 to generate the examples database and f3 for the test. f1 and f2 generate 9249 example. The result is show in figure 13 the average of distance is 0.064. So there are pixels that do not have examples nearest.

Fig. 13: Result of image generated by f3

Iv-3 off-line: the functions f1,f2 and f3, on-line: the image of the silt

In this case we will use f1, f2 and f3 in the database and the silt image 9 for the test, the result is shown in Figure 14. The value of the average of distance is 0.1. Note that whenever the image is becoming more complicated the average distance increases.

Fig. 14: Result of silt image

Iv-4 off-line: the functions f1,f2 f3 and silt, on-line: the images mozart and penny:

The Mozart and penny images contain more detail than the others, so there are several pixels that do not have a nearest example. Figure 15 shows the test result on the two images. The average distance is higher 0.19 for the image of Mozart and 0.21 for the image of penny. The distance interval is between 0 and 1, it represents the difference between the gray level and the azimuth of a pixel in the test image and the nearest example in database. The distance is equal is to 1 if the difference equals $π o r - π$ . the distance of penny’s image equal 0.21 and equal also 0.66 rad $(36^{\circ})$ .

Fig. 15: Result of Mozart and Penny images

Iv-5 off-line: all functions, on-line Silt,Mozart and penny

The advantage of our approach is that we can add the examples to the database according to the use case. The result is depend on the the examples in the database. Now we take the best case, a database contain all possible examples, we will put all functions (f1,f2,f3,vase,mozart and penny) in the database, They generate 23004 examples. We test to the three images vase, Mozart and Penny, the average of the distance equal (0.0000554, 0.0004, 0.027), the results are shown in Figures 16 17 18.

Fig. 16: Result of image generated by the silt

Fig. 17: Result of image generated by Mozart object

Fig. 18: Result of image generated by penny object

Iv-6 use simple real images with database of all functions:

In the above results we tested our approach on synthetic images, in which the boundary conditions are known. To test our approach on images without boundary condition, we will use the edge of the object (there are several methods for detecting the edge). We assume that the projection is perpendicular to the tangent toward the outside. The results shown in Figures 19 and 20 are obtained with any additional information.

Fig. 19: Result of image without Boundary Condition

Fig. 20: Result of image without Boundary Condition

V Conclusion

Most of the local resolution methods does not give a good results, because it is difficult to determine the depth variation from the gray level variation. The solutions of the local resolution are generally complex. In this work we proposed a simple local resolution method using Machine learning. It gives very acceptable results compared with other local resolution methods. The advantage of our approach is in the ”learning phase”. the examples database can be specialized (using same object types), i.e. we can create a database according to the use case. In the future work we will improve this approach, so that we can test it on more complex images, and minimize the number of constraints.

References

[1]
J. Ma P. Zhao ; B. Gong , A shape-from-shading method based on surface reflectance component estimation. Fuzzy Systems and Knowledge Discovery (FSKD), 2012 9th International Conference Page(s): 1690 - 1693 , May 2012
[2]
B. Kunsberg and S.W. Zucker The differential geometry of shape from shading: Biology reveals curvature structure. Computer Vision and Pattern Recognition Workshops (CVPRW), 2012 IEEE Computer Society Conference, Pages: 39-46, June 2012
[3] W. Fan and K. Wang and F. Cayre and X. Zhang 3D lighting-based image forgery detection using shape-from-shading. Signal Processing Conference (EUSIPCO), 2012 Proceedings of the 20th European, Page(s): 1777 - 1781, August 2012
[4] E. Prados and O. Faugeras, Shape From Shading: a well-posed problem?. Computer Vision and Pattern Recognition, 2005. CVPR, June 2005.
[5] A.P. Pentland, LOCAL SHADING ANALYSIS. Pattern Analysis and Machine Intelligence, IEEE Transactions Volume: PAMI-6, Issue:2 Pages:170-187,1982.
[6] J. Denis Durou and M. Falcone and M. Sagona, A Survey of Numerical Methods for Shape from Shading. Rapport de Recherche 2004-2-R, Institut de Recherche en Informatique. IRIT N° 2004-2-R January 2004.
[7] E. Prados and O. Faugeras, Key aspect of the modeling in Shape From Shading. 15ème Congrès de Reconnaissance des Formes et Intelligence Artificielle ,INRIA 2005.
[8] Malladi,R Sethian,J.A level set fast marching methods in computer vision. Image Processing, 1996. Proceedings, International Conference,Pages 489-492 vol.1 , 1996
[9] J.D, DUROU and F, COURTEILLE Integration of a Normal Field without Boundary Condition. Proceedings of the First International Workshop on Photometric Analysis For Computer Vision,2007
[10] K.M, Lee and C, Kuo, Shape From Shading with a Linear Triangular Element Surface Model. Pattern Analysis and Machine Intelligence, IEEE Transactions Volume: 15 Pages: 815 - 822 , August 1993.
[11] Li,J Ren, A and J.Zhang, J 3D Reconstruction by Perspective Shape from Shading Using Linearized Triangular Element Surface Model. Mechatronics and Automation, Proceedings of the IEEE , 2006
[12] J. Denis Durou, Shape from shading Eclairages, réflexion et perspectives. Habilitation Searches, university of paul Sabatier toulouse 3, December 2007.
[13] J.D, DUROU and J.F, AUJOL Integration of a Normal Field in the Case of Discontinuities.EMMCVPR ’09 Proceedings of the 7th International Conference on Energy Minimization Methods in Computer Vision and Pattern Recognition , 2010

A Machine learning approach for Shape From Shading

Authors

Ordinary Facet Angles of a Stroked Path Tessellated by Uniform Tangent Angle Steps Are Bounded by Twice the Step Angle

3D Human Shape Reconstruction from a Polarization Image

Surface HOF: Surface Reconstruction from a Single Image Using Higher Order Function Networks

Directional Modulation: A Secure Solution to 5G and Beyond Mobile Networks

Zero-phase angle asteroid taxonomy classification using unsupervised machine learning algorithms

Deep Learning Approach for Very Similar Objects Recognition Application on Chihuahua and Muffin Problem

Segmentation et Interprétation de Nuages de Points pour la Modélisation d'Environnements Urbains

I Introduction