1 Introduction
The 3D reconstruction of an object starting from one or more images is a very interesting inverse problem with many applications. In fact, this problem appears in various fields which range from the digitization of curved documents [15] to the reconstruction of archaeological artifacts [24]. More recently, other applications have been considered in astronomy to obtain a characterization of properties of planets and other astronomical entities [66, 27, 41]
and in security where the same problem has been applied to the facial recognition of individuals.
In real applications, several light sources can appear in the environment and the object surfaces represented in the scene can have different reflection properties because they are made by different materials, so it would be hard to imagine a scene which can satisfy the classical assumptions of the 3D reconstruction models. In particular, the typical Lambertian assumption often used in the literature has to be weakened. Moreover, despite the fact that the formulation of the ShapefromShading problem is rather simple for a single light source and under Lambertian assumptions, its solution is hard and requires rather technical mathematical tools as the use of weak solutions to nonlinear partial differential equations (PDEs). From the numerical point of view the accurate approximation of non regular solutions to these nonlinear PDEs is still a challenging problem. In this paper we want to make a step forward in the direction of a mathematical formulation of nonLambertian models in the case of orthographic projection with a single light source located far from the surface. In this simplified framework, we present a unified approach to two popular models for nonLambertian surfaces proposed by OrenNayar [48, 49, 47, 50] and by Phong [51]. We will consider light sources placed in oblique directions with respect to the surface and we will use that unified formulation to develop a general numerical approximation scheme which is able to solve the corresponding nonlinear partial differential equations arising in the mathematical description of these models.
To better understand the contribution of this paper, let us start from the classical SfS problem where the goal is to reconstruct the surface from a single image. In mathematical terms, given the shading informations contained in a single twodimensional gray level digital image , where , we look for a surface that corresponds to its shape (hence the name Shape from Shading). This problem is described in general by the image irradiance equation introduced by Bruss [8]
(1) 
where the normalized brightness of the given greyvalue image is put in relation with the function that represents the reflectance map giving the value of the light reflection on the surface as a function of its orientation (i.e., of the normal ) at each point . Depending on how we describe the function , different reflection models are determined. In the literature, the most common representation of is based on the Lambertian model (the L–model in the sequel) which takes into account only the angle between the outgoing normal to the surface and the light source , that is
(2) 
where
denotes the standard scalar product between vectors and
indicates the diffuse albedo, i.e. the diffuse reflectivity or reflecting power of a surface. It is the ratio of reflected radiation from the surface to incident radiance upon it. Its dimensionless nature is expressed as a percentage and it is measured on a scale from zero for no reflection of a perfectly black surface to 1 for perfect reflection of a white surface. The data are the greyvalue image , the direction of the light source represented by the unit vector and the albedo . The light source is a unit vector, hence . In the simple case of a vertical light source, that is when the light source is in the direction of the vertical axis, this gives rise to an eikonal equation. Several questions arise, even in the simple case: is a single image sufficient to determine the surface? If not, which set of additional informations is necessary to have uniqueness? How can we compute an approximate solution? Is the approximation accurate? It is well known that for Lambertian surfaces there is no uniqueness and other informations are necessary to select a unique surface (e.g. the height at each point of local maximum for ). However, rather accurate schemes for the classical eikonal equation are now available for the approximation. Despite its simplicity, the Lambertian assumption is very strong and does not match with many real situations that is why we consider in this paper some nonLambertian models trying to give a unified mathematical formulation for these models.In order to set this paper into a mathematical perspective, we should mention that the pioneering work of Horn [28, 29] and his activity with his collaborators at MIT [30, 31] produced the first formulation of the Shape from Shading (SfS) problem via a partial differential equation (PDE) and a variational problem. These works have inspired many other contributions in this research area as one can see looking at the extensive list of references in the two surveys [85, 19]. Several approaches have been proposed, we can group them in two big classes (see the surveys [85, 19]): methods based on the resolution of PDEs and optimization methods based on a variational approximation. In the first group the unknown is directly the height of the surface , one can find here rather old papers based on the method of characteristics [17, 60, 29, 46, 45, 6, 38] where one typically looks for classical solutions. More recently, other contributions were developed in the framework of weak solutions in the viscosity sense starting from the seminal paper by Rouy and Tourin [61] and, one year later, by LionsRouy and Tourin [40] (see e.g. [33, 9, 11, 22, 10, 56, 4, 62, 36, 23, 53]). The second group contains the contribution based on minimization methods for the variational problem where the unknown are the partial derivatives of the surface, and (the socalled normal vector field. See e.g. [30, 16, 67, 26, 68, 32, 7, 82]). It is important to note that in this approach one has to couple the minimization step to compute the normal field with a local reconstruction for which is based usually on a path integration. This necessary step has also been addressed by several authors (see [18] and references therein). We should also mention that a continuous effort has been made by the scientific community to take into account more realistic reflectance models [3, 1, 59, 76, 77], different scenarios including perspective camera projection [44, 70, 54, 2, 75, 14] and/or multiple images of the same object [83, 84]. The images can be taken from the same point of view but with different light sources as in the photometric stereo method [81, 37, 42, 69, 43] or from different points of view but with the same light source as in stereo vision [13]. Recent works have considered more complicated scenarios, e.g. the case when light source is not at the optical center under perspective camera projection [35]
. It is possible to consider in addition other supplementary issues, as the estimation of the albedo
[86, 5, 65, 64] or of the direction of the light source that are usually considered known quantities for the model but in practice are hardly available for real images. The role of boundary conditions which have to be coupled with the PDE is also a hard task. Depending on what we know, the model has to be adapted leading to a calibrated or uncalibrated problem (see [84, 83, 25, 57] for more details). In this work we will assume that the albedo and the light source direction are given.Regarding the modeling of nonLambertian surfaces we also want to mention the important contribution of Ahmed and Farag [2]. These authors have adopted the modeling for SfS proposed by Prados and Faugeras [55, 52] using a perspective projection where the light source is assumed to be located at the optical center of the camera instead at infinity and the light illumination is attenuated by a term ( represents here the distance between the light source and the surface). They have derived the HamiltonJacobi (HJ) equation corresponding to the OrenNayar model, developed an approximation via the LaxFriedrichs sweeping method. They gave there an experimental evidence that the nonLambertian model seems to resolve the classical concave/convex ambiguity in the perspective case if one includes the attenuation term . In [1] they extended their approach for various image conditions under orthographic and perspective projection, comparing their results for the orthographic L–model shown in [63] and in [85]. Finally, we also want to mention the paper by Ragheb and Hancock [58] where they treat a nonLambertian model via a variational approach, investigating the reflectance models described by Wolff and by Oren and Nayar [80, 79, 50].
Our Contribution.
In this paper we will adopt the PDE approach in the framework of weak solutions for HamiltonJacobi equations. As we said, we will focus our attention on a couple of nonLambertian reflectance models: the OrenNayar and the Phong models [48, 49, 47, 50, 51]. Both models are considered for an orthographic projection and a single light source at infinity, so no attenuation term is considered here.
We are able to write the HamiltonJacobi equations in the same fixed point form useful for their analysis and approximation and, using the exponential change of variable introduced by Kružkov in [39], we obtain natural upper bound for the solution. Moreover, we propose a semiLagrangian approximation scheme which can be applied to both the models, prove a convergence result for our scheme that can be applied to this class of HamiltonJacobi equations, hence to both nonLambertian models. Numerical comparisons will show that our approach is more accurate also for the 3D reconstructions of nonsmooth surfaces.
A similar formulation for the Lambertian SfS problem with oblique light direction has been studied in [23] and here is extended to nonLambertian models. We have reported some preliminary results just for the OrenNayar model in [73].
Organization of the paper.
The paper is organized as follows.
After a formulation of the general model presented in Section 2, we present the SfS models starting from the classical Lambertian model (Section 3). In Sections 4 and 5 we will give details on the construction of the nonlinear partial differential equation which corresponds respectively to the OrenNayar and the Phong models. Despite the differences appearing in these nonLambertian models, we will be able to present them in a unified framework showing that the HamiltonJacobi equations for all the above models share a common structure. Moreover, the Hamiltonian appearing in these equations will always be convex in the gradient . Then, in Section 6, we will introduce our general approximation scheme which can be applied to solve this class of problems.
In Section 7 we will apply our approximation to a series of benchmarks based on synthetic and real images. We will discuss some issue like accuracy, efficiency and the capability to obtain the maximal solution showing that the semiLagrangian approximation is rather effective even for real images where several parameters are unknown.
Finally, in the last section we will give a summary of the contributions of this work with some final comments and future research directions.
2 Formulation of the general model
We fix a camera in a threedimensional coordinate system (Oxyz) in such a way that Oxy coincides with the image plane and Oz with the optical axis.
Let (with ) be the unit vector that represents the direction of the light source (the vector points from the object surface to the light source);
let be the function that measures the graylevel of the input image at the point . is the datum in the model since it is measured at each pixel of the image, for example in terms of a greylevel (from to ). In order to construct a continuous model, we will assume that takes real values in the interval , defined in a compact domain called “reconstruction domain” (with open set),
, where the points with a value of are the dark point (blacks), while those with a value of correspond to a completely reflection of the light (white dots, with a maximum reflection).
We consider the following assumptions:
 A1.

there is a single light source placed at infinity in the direction (the light rays are, therefore, parallel to each other);
 A2.

the observer’s eye is placed at an infinite distance from the object you are looking at (i.e. there is no perspective deformation);
 A3.

there are no autoreflections on the surface.
In addition to these assumptions, there are other hypothesis that depend on the different reflectance models (we will see them in the description of the individual models).
Being valid the assumption of orthographic projection, the visible part of the scene is a graph and the unit normal to the regular surface at the point corresponding to is given by:
(3) 
where is the outgoing normal vector.
We assume that the surface is standing on a flat background so the height function, which is the unknown of the problem, will be non negative, . We will denote by the region inside the silhouette and we will assume (just for technical reasons) that is an open and bounded subset of (see Fig. 1).
It is well known that the SfS problem is described by the image irradiance equation (1) and depending on how we describe the function different reflection models are determined. We describe below some of them. To this end, it would be useful to introduce a representation of the brightness function where we can distinguish different terms representing the contribution of ambient, diffuse reflected and specular reflected light. We will write then
(4) 
where , and are respectively the above mentioned components and , and indicate the percentages of these components such that their sum is equal to 1 (we do not consider absorption phenomena). Note that the diffuse or specular albedo is inside the definition of or , respectively. In the sequel, we will always consider normalized in . This will allow to switch on and off the different contributions depending on the model. Let us note that the ambient light term represents light everywhere in a given scene. In the whole paper we will consider it as a constant and we will neglect its contribution fixing . Moreover, for all the models presented below we will suppose uniform diffuse and/or specular albedo and we will put them equal to , that is all the points of the surface reflect completely the light that hits them. We will omit them in what follows. As we will see in the following sections, the intensity of diffusely reflected light in each direction is proportional to the cosine of the angle between surface normal and light source direction, without taking into account the point of view of the observer, but another diffuse model (the Oren–Nayar model) will consider it in addition. The amount of specular reflected light towards the viewer is proportional to , where is the angle between the ideal (mirror) reflection direction of the incoming light and the viewer direction, being a constant modelling the specularity of the material. In this way we have a more general model and, dropping the ambient and specular component, we retrieve the Lambertian reflection as a special case.
3 The Lambertian model (L–model)
For a Lambertian surface, which generates a purely diffuse model, the specular component does not exist, then in (4) we have just the diffuse component on the right side. Lambertian shading is view independent, hence the irradiance equation (1) becomes
(5) 
Under these assumptions, the orthographic SfS problem consists in determining the function that satisfies the equation (5). The unit vector and the function are the only quantities known.
For Lambertian surfaces [30, 31], just considering an orthographic projection of the scene, it is possible to model the SfS problem via a nonlinear PDE of the first order which describes the relation between the surface (our unknown) and the brightness function .
In fact, recalling the definition of the unit normal to a graph given in (3), we can write (5) as
(6) 
where . This is an HamiltonJacobi type equation which does not admit in general a regular solution. It is known that the mathematical framework to describe its weak solutions is the theory of viscosity solutions as in [40].
The vertical light case.
If we choose , the equation (6) becomes the socalled “eikonal equation”:
(7) 
where
(8) 
The points where assumes maximum value correspond to the case in which and have the same direction: these points are usually called “singular points”.
In order to make the problem wellposed, we need to add boundary conditions to the equations (6) or (7): they can require the value of the solution (Dirichlet boundary conditions type), or the value of its normal derivative (Neumann boundary conditions), or an equation that must be satisfied on the boundary (the socalled boundary conditions “state constraint”).
In this paper, we consider Dirichlet boundary conditions equal to zero assuming a surface on a flat background
(9) 
but a second possibility of the same type occurs when it is known the value of on the boundary, which leads to the more general condition
(10) 
Unfortunately, adding a boundary condition to the PDE that describes the SfS model is not enough to obtain a unique solution because of the concave/convex ambiguity. In fact, the Dirichlet problem (6)(10) can have several weak solutions in the viscosity sense and also several classical solutions due to this ambiguity (see [29]). As an example, all the surfaces represented in Fig. 2 are viscosity solutions of the same problem (7)(9) which is a particular case of (6)(10) (in fact the equation is with homogenous Dirichlet boundary condition). The solution represented in Fig. 2a is the maximal solution and is smooth. All the nonsmooth a.e. solutions, which can be obtained by a reflection with respect to a horizontal axis, are still admissible weak solutions (see Fig. 2b). In this example, the lack of uniqueness of the viscosity solution is due to the existence of a singular point where the right hand side of (7) vanishes. An additional effort is then needed to define which is the preferable solution since the lack of uniqueness is also a big drawback when trying to compute a numerical solution. In order to circumvent these difficulties, the problem is usually solved by adding some information such as height at each singular point [40].
For analytical and numerical reasons it is useful to introduce the exponential Kružkov change of variable [39] . In fact, setting the problem in the new variable we will have values in instead of as the original variable so an upper bound will be easy to find. Note that is a free positive parameter which does not have a specific physical meaning in the SfS problem. However, it can play an important role also in our convergence proof as we will see later (see the remark following the end of Theorem 6.1). Assuming that the surface is standing on a flat background and following [23], we can write (6) and (9) in a fixed point form in the new variable . To this end let us define and as
(11) 
(12) 
and let denote the unit ball in . We obtain
Lambertian Model
(13)
where
.

It is important to note for the sequel that the structure of the above first order HamiltonJacobi equation is similar to that related to the dynamic programming approach in control theory where is a vector field describing the dynamics of the system and is a running cost. In that framework the meaning of is that of a value function which allows to characterize the optimal trajectories (here they play the role of characteristic curves). The interested reader can find more details on this interpretation in [21].
4 The OrenNayar model (ON–model)
The diffuse reflectance ON–model [48, 49, 47, 50] is an extension of the previous L–model which
explicitly allows to handle rough surfaces.
The idea of this model is to represent a rough surface as an aggregation of Vshaped cavities, each with Lambertian reflectance properties (see Fig. 3).
In [48] and, with more details, in [50], Oren and Nayar derive a reflectance model for several type of surfaces with different slopearea distributions.
In this paper we will refer to the model called by the authors the “Qualitative Model”, a simpler version obtained by ignoring interreflections (see Section of [48] for more details).
Assuming that there is a linear relation between the irradiance of the image and the image intensity, the brightness equation for the ON–model is given by
(14) 
(15)  
(16)  
(17) 
Note that and are two non negative constants depending on the statistics of the cavities via the roughness parameter . We set , interpreting as the slope of the cavities. In this model (see Fig. 4), represents the angle between the unit normal to the surface and the light source direction , stands for the angle between and the observer direction , is the angle between the projection of the light source direction and the axis onto the plane, denotes the angle between the projection of the observer direction and the axis onto the plane and the two variables and are given by
(18) 
Since the vectors and are fixed and given, their projection on the incident plane is obtained considering their first two components over three (see Eq. (21)). In this way, the quantity is computed only once for a whole image.
For smooth surfaces, we have and in this case the ON–model reduces to the L–model. In the particular case , or, more precisely, when (e.g. the case when the unit vectors and are perpendicular we get ) the equation (14) simplifies and reduces to a L–model scaled by the coefficient . This means that the model is more general and flexible than the L–model. This happens when only one of the two unit vectors is zero or, more in general, when the dot product between the normalized projections onto the plane of and is equal to zero.
Also for this diffuse model we neglect the ambient component, setting . As a consequence, in the general equation (4) the total light intensity is equal to the diffuse component (described by the equation (14)). This is why we write instead of in what follows.
To deal with this equation one has to compute the and operators which appear in (14) and (18).
Hence, we must consider several cases described in detail in what follows. For each case we will derive a partial differential equation that is always a first order nonlinear HJ equation:
Case 1: and
The brightness equation (14) becomes
(24) 
and by using the formulas (19)(23) we arrive to the following HJ equation
(25) 
where and .
Case 2: and
In this case the brightness equation (14) becomes
(26) 
and by using again the formulas (19)(23) we get
(27) 
Case 3: and
In this case we have the implication .
The brightness equation (14) simplifies in
(28) 
and the HJ equation associated to it becomes consequentially
(29) 
that is equal to the L–model scaled by the coefficient .
Case 4: and
This is a particular case when the position of the light source coincides with the observer direction but there are not on the vertical axis.
This choice implies , then defining , the equation (14) simplifies to
(30) 
and we arrive to the following HJ equation
(31) 
Note that this four cases are exactly the same cases reported and analyzed in [35]. This is not surprising since the reflectance model used there is always the same one proposed by Oren and Nayar. However, here we get different HJ equations since we consider an orthographic camera projection and cartesian coordinates whereas in [35] the HJ equations are derived in spherical coordinates under a generalized perspective camera projection. Another major difference is that in that paper the light source is close to the camera but is not located at the optical center of the camera.
The vertical light case.
If , independently of the position of , the analysis is more simple. In fact, the first three cases considered above reduce to a single case corresponding to the following simplified PDE for the brightness equation (14)
(32) 
In this way we can put it in the following eikonal type equation, analogous to the Lambertian eikonal equation (7):
(33) 
where
(34) 
Following [73], we write the surface as , for , , and , so (31) becomes
(35) 
Defining
(36) 
and
(37) 
using the equivalence
(38) 
we get
(39) 
Let us define the vector field for the ONmodel
(40) 
and
(41) 
Then, introducing the exponential Kružkov change of variable as already done for the L–model, we can finally write the fixed point problem in the new variable obtaining the
OrenNayar Model
(42)
where

Note that the simple homogeneous Dirichlet boundary condition is due to the flat background behind the object but a condition like can also be considered if necessary. Moreover, the structure is similar to the previous Lambertian model although the definition of the vector field and of the cost are different.
In the particular case when ,
the equation (14) simply reduces to
(43) 
and, as a consequence, the Dirichlet problem in the variable is equal to (42) with .
5 The Phong model (PH–model)
The PH–model is an empirical model that was developed by Phong [51] in 1975. This model introduces a specular component to the brightness function , representing the diffuse component in (4) as the Lambertian reflectance model.
A simple specular model is obtained putting the incidence angle equal to the reflection one and , and belong to the same plane.
This model describes the specular light component as a power of the cosine of the angle between the unit vectors and (it is the vector representing the reflection of the light on the surface).
Hence, the brightness equation for the PH–model is
(44) 
where the parameter is used to express the specular reflection properties of a material and and indicate the percentages of diffuse and specular components, respectively. Note that the contribution of the specular part decreases as the value of increases.
Starting to see in details the PH–model in the case of oblique light source and oblique observer .
Assuming that is the bisector of the angle between and (see Fig. 5), we obtain
(45) 
which implies
(46) 
From the parallelogram law, taking into account that and are unit vectors, we can write , then we can derive the unit vector as follow:
(47)  
With this definition of the unit vector we have
(48)  
Then, setting , that represents the worst case, equation (44) becomes
(49)  
to which we add a Dirichlet boundary condition equal to zero, assuming that the surface is standind on a flat blackground.
Note that the cosine in the specular term is usually replaced by zero if (and in that case we get back to the L–model).
As we have done for the previous models, we write the surface as , for , , and , so (5) will be written as
(50)  
Dividing by , defining as in (36) and , we get
(51)  
By the equivalence we obtain
(52)  
Let us define the vector field
(53) 
where
(54) 
and
(55) 
Let us also define
(56) 
Again, using the exponential Kružkov change of variable as done for the previous models, we can finally write the nonlinear fixed point problem
Phong Model
(57)
where

A. Oblique light source and vertical position of the observer.
In the case of oblique light source and vertical observer , the dot product becomes
(58) 
The fixed point problem in will be equal to (57) with the following choices
(59) 
B. Vertical light source and oblique position of the observer.
When the definition of the vector reported in (5) becomes
(60) 
and, as a consequence, the dot product with general is
(61) 
Hence, the fixed point problem in is equal to (57) with
(62) 
C. Vertical light source and vertical position of the observer.
If we choose the equation (5) simplifies in
(63)  
Working on this equation one can put it in the following eikonal type form, which is analogous to the Lambertian eikonal equation (7):
(64) 
where now
(65) 
with
(66)  
(67)  
(68) 
Remark on the control interpretation. The above analysis has shown that all the cases corresponding to the models proposed by OrenNayar and by Phong lead to a stationary HamiltonJacobi equation of the same form, namely
where the vector field and the cost can vary according to the model and to the case. This gives to these models a control theoretical interpretation which can be seen as a generalization of the control interpretation for the original Lambertian model (which was related to the minimum time problem). In this framework, is the value function of a rescaled (by the Kružkov change of variable) control problem in which one wants to drive the controlled system governed by
( here is the control function taking values in ) from the initial position to the target (the silhouette of the object) minimizing the cost associated to the trajectory. The running cost associated to the position and the choice of the control will be given by . More informations on this interpretation, which is not crucial to understand the application to the SfS problem presented in this paper, can be found in [21].
6 SemiLagrangian Approximation
Now, let us state a general convergence theorem suitable for the class of differential operators appearing in the models described in the previous sections. As we noticed, the unified approach presented in this paper has the big advantage to give a unique formulation for the three models in the form of a fixed point problem
(69) 
where indicates the model, i.e. .
We will see that the discrete operators of the ON–model and the PH–model described in the previous sections satisfy the properties listed here. In order to obtain the fully discrete approximation we will adopt the semiLagrangian approach described in the book by Falcone and Ferretti [21]. The reader can also refer to [12] for a similar approach to various image processing problems (including nonlinear diffusions models and segmentation).
Let so that will be the vector solution giving the approximation of the height at every node of the grid. Note that in one dimension the index is an integer number, in two dimensions denotes a multiindex, . We consider a semiLagrangian scheme written in a fixed point form, so we will write the fully discrete scheme as
(70) 
Denoting by the global number of nodes in the grid, the operator corresponding to the oblique light source is that is defined componentwise by
(71) 
where
represents an interpolation operator based on the values at the grid nodes and
(72)  
(73)  
(74)  
(75) 
Since is approximated via by interpolation on (which is defined on the grid ), it is important to use a monotone interpolation in order to preserve the properties of the continuous operator in the discretization. To this end, the typical choice is to apply a piecewise linear (or bilinear) interpolation operator which allows to define a function defined for every (and not only on the nodes)
(76) 
where
(77) 
A simple explanation for (76)(77) is that the coefficients represent the local coordinates of the point with respect to the grid nodes (see [21] for more details and other choices of interpolation operators). Clearly, in (71) we will apply the interpolation operator to the point and we will denote by the function defined by .
Comparing (71) with its analogue for the vertical light case we can immediately note that the former has the additional term which requires analysis.
Theorem 6.1
Let the th component of the operator defined as in (71). Then, the following properties hold true:

Let
and assume(78) Then implies .

is a monotone operator, i.e., implies .

is a contraction mapping in if
.
Proof.

To prove that implies we just note that
(79) Let ; then
(80) This implies that if
Comments
There are no comments yet.