This paper presents a novel boundary element/finite element coupling and regularization procedure for the solution of a nonlinear scalar interface problem on an unbounded domain with nonmonotone set-valued transmission conditions that models nonlinear contact problems with nonmonotone friction in infinite elastic media. Such contact problems arise in various fields of science and technology; let us mention geophysics, see e.g. , soil mechanics and civil engineering of underground structures, see e.g. .
In this paper we employ various mathematical techniques for the solution of the interface problem that consists of a nonlinear monotone partial differential equation in a bounded domain and the Laplace operator in the exterior domain coupled by set-valued nonmonotone transmission conditions. Using Clarke generalized differentiation  we describe this coupled boundary value problem as a nonlinear hemivariational inequality (HVI). Further by singular boundary integral methods  we reduce this problem to a HVI that lives on the bounded domain and the coupling boundary, only. Thus for discretization we can develop a BEM/FEM coupling method. In addition we adapt from  smoothing techniques of nondifferentiable optimization and regularize the nonsmooth part in the HVI to arrive at a finite dimensional problem that can be solved by standard optimization tools. We do not only provide convergence results for the total approximation procedure, but also show an error estimate for the regularized HVI that gives the same convergence order as the error estimate of BEM/FEM coupling for the monotone Signorini contact problem in .
The coupling of finite element and boundary element methods combining the best of both ”worlds”  provides nowadays a very effective tool for the numerical solution of boundary value problems in physics and engineering. Indeed, the boundary element method is better suited to problems in which the domain extends to infinity but is usually confined to regions in which the governing equations are linear and homogeneous. On the other hand, the finite element method is restricted to problems in bounded domains but is applicable to problems in which the material properties are not necessarily homogeneous and nonlinearity may occur. This method was originally proposed by engineers (see e. g. Zienkiewicz et al. ). The mathematical analysis goes back to Brezzi, Johnson, and Nédélec [5, 4, 32] and was extended by Wendland [48, 49]. The symmetric coupling of finite element and boundary element methods is due to Costabel [10, 12]. For the coupling of finite element and boundary element methods for various nonlinear interface problems we point out the papers [13, 46, 14, 15, 35, 8, 17, 1, 18] and for a comprehensive exposition of finite element and boundary coupling refer to [26, Chapter 12].
The theory of hemivariational inequalities has been introduced and studied since 1980s by Panagiotopoulos , as a generalization of variational inequalities with an aim to model many problems coming from mechanics when the energy functionals are nonconvex, but locally Lipschitz, so the Clarke generalized differentiation calculus  can used, see [37, 21, 20]. For more recent monographs on hemivariational inequalities with application to contact problems we refer to [36, 45]. In parallel with the mathematical analysis of hemivariational inequalities the interest in efficient and reliable numerical methods for their solution constantly increases. The classical book on the finite element method for hemivariational inequalities is the monograph of Haslinger et al. . More recent work on numerical solution of HVIs modeling contact problems with nonmonotone friction, adhesion, cohesive cracks, and delamination for elastic bodies on bounded domains is contained in the papers [2, 33, 30, 34, 38, 28, 27]; see also  for variationally consistent discretization schemes and numerical algorithms for contact problems.
The plan of the paper is as follows. The next section 2 collects some basic notions of Clarke’s generalized differential calculus that are needed for the analysis of the non-monotone boundary conditions. Then we descibe the interface problem under study. We settle the issues of existence and uniqueness in a general functional analytic framework. Section 3 provides a first equivalent weak variational formulation of the interface problem in terms of a hemivariational inequality (HVI). Since this HVI lives on the unbounded domain (as the orignal problem), this hemivariational formulation cannot numerically treated directly and therefore provides only an intermediate step in the solution procedure. Section 4 employs boundary integral analysis to transform the interface problem to a HVI that lives on the interior bounded domain and the interface boundary, only, and so is amenable to numerical treatment. Section 5 uses regularization techniques of nonsmooth optimization and derives a regularized version of the HVI that becomes a variational equality. Section 6 turns to numerical analysis of the regularized HVI, employs the Galerkin boundary element/finite element method, and presents an asymptotic error estimate. The final section 7 summarizes our findings, gives some concluding remarks, and sketches some directions of further research.
2 Some preliminaries and the interface problem
Let us first recall the central notions of Clarke’s generalized differential calculus , before we pose our interface problem. Let be a (real) Banach space, let be a locally Lipschitz function. Then
is called the generalized directional derivative of in the direction . Note that the function is finite, positively homogeneous, and sublinear, hence convex and continuous; further, the function is upper semicontinuous. The generalized gradient of the function at , denoted by (simply) , is the unique nonempty compact convex subset of the dual space , whose support function is . Thus
When is finite dimensional, then, according to Rademacher’s theorem, is differentiable almost everywhere, and the generalized gradient of at a point can be characterized by
where ”co” denotes the convex hull.
In this paper we treat the following interface problem. Let be a bounded domain with Lipschitz boundary . To describe mixed transmission conditions, we assume that the boundary splits into two non-empty, open disjoint parts and such that . Let denote the unit normal on defined almost everywhere pointing from into .
In the interior part , we consider the nonlinear partial differential equation
where is a continuous function with being monotonously increasing with .
In the exterior part , we consider the Laplace equation
with the radiation condition at infinity
where are real constants (constant for any but varying with ). Let us write and , then the tractions on the coupling boundary are given by the traces of and , respectively.
We prescribe classical transmission conditions on ,
and on , analogously for the tractions,
and the generally nonmonotone, set-valued transmission condition,
Here we consider a function such that is measurable on for all and is locally Lipschitz on for almost all (a.a.) . We write for the generalized gradient of at the point in the sense of Clarke. Moreover, stands for the generalized directional derivative of .
Further, we require the following growth condition on the so-called superpotential : There exist positive constants and such that for a.a. , all and for all the following inequalities hold
To conclude this section, we now discuss the existence and uniqueness of a weak solution to this interface problem in functional analytic terms. To this end, let and introduce the real-valued locally Lipschitz functional
Then by Lebesgue’s theorem of majorized convergence,
As we shall see in the subsequent sections, the weak formulation of the problem (1)–(7) leads, in an abstract setting, to a hemivariational inequality (HVI) with a nonlinear operator and the nonsmooth functional , namely, we are looking for some such that
Here is a closed convex subset of a Banach space , the nonlinear operator is a monotone operator, and denotes the linear continuous trace operator, and the linear form belongs to the dual . Similar to , the operator consists of a nonlinear monotone differential operator (as made precise below) that results from the PDE (1) in the bounded domain and the positive definite Poincaré–Steklov operator on the boundary of that stems from the exterior problem (2)-(4) and can be represented by the boundary integral operators of potential theory. Thus it results that the operator is strongly monotone with some monotonicity constant and Lipschitz continuous on bounded sets. On the other hand, by the compactness of the trace map , the real-valued upper semicontinuous bifunction
can be seen to be pseudo-monotone, see [41, Lemma 1], [24, Lemma 4.1]. The latter result also shows a linear growth of . This and the uniform monotonicity of imply coercivity. Therefore by the theory of pseudo-monotone VIs [22, Theorem 3], , see  for the application to HVIs, we have solvability of (9).
Further suppose that the generalized directional derivative satisfies the one-sided Lipschitz condition: There exists such that
Then the smallness condition
It is noteworthy that under the smallness condition (11) together with (10), fixed point arguments  or the theory of set-valued pseudo-monotone operators  are not needed, but simpler monotonicity arguments are sufficient to conclude unique solvability. Thus the compactness of the trace map is not needed either. In fact, (9) can be framed as a monotone equilibrium problem in the sense of Blum-Oettli . and so the fundamental existence result [3, Theorem 1] directly applies in view of the following
Proof. Obviously vanishes on the diagonal and is convex and lower semicontinuous with respect to the second variable. To show (strict) monotonicity, estimate
To show hemicontinuity, it is enough to consider the bifunction . Then for fixed, one has
and thus hemicontinuity follows from upper semicontinuity of ,
3 An intermediate HVI formulation of the interface problem
In this section we provide a first equivalent weak variational formulation of the interface problem (1) - (7) in terms of a hemivariational inequality (HVI). Since this HVI lives on the unbounded domain (as the orignal problem), this hemivariational formulation cannot numerically treated directly and therefore provides only an intermediate step in the solution procedure.
For the bounded Lipschitz domain we use the standard Sobolev space and the Sobolev spaces on the bounded boundary (see e.g. [26, Chapter 3,4]),
Further we need for the unbounded domain the Frechet space (see e.g. [31, Section 4.1, (4.1.43)])
Due to the trace theorem whenever . Then, we define by
Here is the linear functional
where we assume that is , , and is monotone increasing with . Then, and hence,
is finite for any and strictly convex. The Frechet derivative of ,
is strongly monotone in with respect to the semi-norm , i.e., there exists a constant such that
and then the affine, hence convex set of admissible functions
According to [7, Remark 4], is closed in . Further, we have
Now we are in the position to pose the hemivariational inequality problem : Find such that for all we have
Proof. First, taking into account the definition of the generalized gradient, we note that
is the integral formulation of the nonmonotone boundary inclusion (7).
Let solve (15). To show that solves (1) - (7) in the sense of distributions, first choose such that . Setting in (15) and integration by parts, implies
Since is arbitrary on , we obtain
Next, let and consider in an analogous way to obtain
which implies by (17)
Finally, we define . Taking on , we have on , but is arbitrary on , what gives (16).
Vice versa we show that (15) follows from (1) - (7). Let solve (1) - (7). Due to (2) and (4), . Multiplying (1) and (2) with differences , , respectively, where one chooses arbitrarily, and integrating by parts yields
what shows that solves (15).
4 The boundary/domain HVI formulation of the interface problem
In this section we first briefly recall from [11, 31, 26] some boundary integral operator theory associated to the Laplace equation and then employ it to rewrite the exterior problem (2) - (4) as a boundary variational inequality on . As a result we arrive at an equivalent hemivariational formulation of the original interface problem (1) - (7) that lives on and consists of a weak formulation of the nonlinear differential operator in the bounded domain , the Poincare-Stéklov operator on the bounded boundary , and a nonsmooth functional on the boundary part .
Given , its Cauchy data on are
With the fundamental solution for the Laplacian,
define for the single layer potential and the double layer potential by
Note that (20) determines in as far as one knows its Cauchy data on .
Next define the single layer boundary integral operator , the double layer boundary integral operator , its formal adjoint , and the hypersingular integral operator for as follows:
Due to , the linear operators
are continuous. Moreover, the operator is symmetric and positive definite and is symmetric and positive semi-definite provided that the capacity of is smaller than 1 which can be always arranged by appropriate scaling, see e.g.  for details. Then, (i.e. consists of the constant functions).
maps continuously into its dual space and the bilinear form is positive definite (provided that the capacity of is smaller than 1), i.e., there exists a constant such that
where extends the duality on .
Let with . Define for all by
and the linear functional for all by
Moreover, in case , similarly to , we introduce an additional linear constraint and consider the affine closed subspace
Then, the hemivariational inequality problem reads: Find such that for all
Now we show the equivalence of problems and .
(i) Let solve .
Then, solves , where
for if , if
(ii) Let solve . Take arbitrarily if , whereas if . Define and by the representation formula (20) with replacing , i.e.
Then, solves .
Analogously to , but now with and defined in (i), (ii)), respectively, it can been shown that the sets and correspond to each other.
Now let solve . To show that defined in (i) solves we simply compare the different terms in and .