1 Introduction
Mathematical morphology (MM) is a nonlinear image processing methodology based on computing max/min filters in local neighborhoods defined by structuring elements [16, 17]. By concatenation of two basic operators, i.e., the dilation and the erosion , on the image , we obtain the closing and the opening
, which are filters with scalespace properties and selective feature extraction skills according to the underlaying structuring element
. Other more sophisticated filters are obtained by combinations of openings and closings, to address problems such as nonGaussian denoising, image regularization, etc.Finding the proper pipeline of morphological operators and structuring elements in real applications is a cumbersome and time consuming task. In the machine learning community there has always been lot of interest in learning such operators, but due to the nondifferentiable nature of the max/min filtering only few approaches have been found to succeed, notably one based on LMS (gradient steepest descent algorithm) for rank filters formulated with the sign function [13, 14]. This idea was later revisited [12]
in a neural network framework combining morphological/rank filters and linear FIR filters. Other attempts from the evolutionary community (e.g., genetic algorithms
[7] or simulated annealing [21]) use blackbox optimizers to circumvent the differentiability issue. However, most of the proposed approaches do not cover all operators. More importantly, they cannot learn both the structuring element and the operator, e.g., [11]. This is obviously a quite important limitation as it makes very hard or even impossible the composition of complex filtering pipelines. Furthermore, such systems are usually limited to a very specific application and hardly generalize to complex scenarios.Inspired by recent work [1]
on counterharmonic mean asymptotic morphology, we propose a novel framework to learn morphological pipelines of operators. We combine convolutional neural networks (CNN) with a new type of layer that permits complex pipelines through multiple layers and therefore extends this models to a Morphological Convolutional Neural Network (MPCNN). It extends previous work on deeplearning while making directly applicable all optimization tricks and findings of this field.
Here we focus on methodological foundations and show how the model learns several operator pipelines, from dilation/erosion to tophat transform (i.e., residue of opening/closing). We report an important realworld application from the steel industry, and present a sample application to denoising where operator learning outperforms handcrafted structuring elements.
Our main contributions are:

a novel framework where learning of morphological operators and filtering pipelines can be performed using gradientbased techniques, exploiting recent insights of deeplearning approaches;

the introduction of a novel PConv layer for CNN, to let CNN benefit from highly nonlinear, morphologybased filters;

the stacking of many PConv layers, to learn complex pipelines of operators such as opening, closing and tophats.
2 Background
Here we illustrate the foundations of our approach, introducing the CounterHarmonic Mean formulation for asymptotic morphology and CNN.
2.1 Asymptotic morphology using CounterHarmonic Mean
We start from the notion of counterharmonic mean [2], initially used [19] for constructing robust morphologicallike operators. More recently, its morphological asymptotic behavior was characterized [1]. Let be a 2D realvalued image, i.e., , where denotes the coordinates of the pixel in the image domain . Given a (positive) weighting kernel , being the support window of the filter, the counterharmonic mean (CHM) filter of order , is defined by,
(1) 
where is the image, where each pixel value of is raised to power , indicates pixelwise division, and is the support window of the filter centered on point . We note that the CHM filter can be interpreted as deformed convolution, i.e., . For () the pixels with largest (smallest) values in the local neighborhood will dominate the result of the weighted sum (convolution), therefore morphological dilation and erosion are the limit cases of the CHM filter, i.e., , and , where plays the role of the structuring element. As proven earlier [1], apart from the limit cases (e.g., a typical order of magnitude of ), we have the following behavior:
(2)  
(3) 
which can be interpreted, respectively, as the nonflat dilation (supremal convolution) and nonflat erosion (infimal convolution) using the structuring function . By using constant weight kernels, i.e., if and if , and , we just recover the corresponding flat structuring element , associated to the structuring function if and if .
From a precise morphological viewpoint, we notice that for finite one cannot guarantee that yields exactly a pair of dilation/erosion, in the sense of commutation with max/min [16, 17]. Consequently, stricto sensu, we can only name them as pseudodilation () and pseudoerosion (). The asymptotic cases of the CHM filter can be also combined to approximate opening and closing operators, i.e.,
(4) 
2.2 Convolutional Neural Networks
CNN are hierarchical models alternating two basic operations, convolution and subsampling, reminiscent of simple and complex cells in the primary visual cortex [8]. Their main characteristic is that they exploit the 2D structure of images via weight sharing, learning a set of convolutional filters. Certain CNN scale well to realsized images and excel in many object recognition [3, 4, 6, 10] and segmentation [5, 18] benchmarks. We refer to a stateoftheart CNN as depicted in Figure 1. It consists of several basic building blocks briefly explained here:

Convolutional Layer: performs a 2D filtering between input images and a bank of filters , producing another set of images denoted as maps. Inputoutput correspondences are specified through a connection table (inputImage , filterId , outputImage ). Filter responses from inputs connected to the same output image are linearly combined. This layer performs the following mapping: where
indicates the 2D valid convolution. Then, a nonlinear activation function (e.g.,
, , etc.) is applied to . Recently activations have been found to excel. They are the units we use for all our models. A unit operates as . It is common choice to use as lower bound () and as upper bound (). 
Pooling Layer: downsamples the input images by a constant factor keeping a value (e.g. maximum or average) for every non overlapping subregion of size
in the images. MaxPooling is generally favorable, as it introduces small invariance to translation and distortion, leads to faster convergence and better generalization
[15]. 
Fully Connected Layer
: this is the standard layer of a multilayer network. It performs a linear multiplication of the input vector by a weight matrix.
Note the striking similarity between the maxpooling layer and a dilation transform. The former is in fact a special case of dilation, with a square structuring element of size followed by downsampling (sampling one out of every pixels). Our novel layer, however, does not any longer limit the pooling operation to simple squares, but allows for a much richer repertoire of structuring elements finetuned for given tasks. This is what makes MCNN so powerful.
3 Method
Now we are ready to introduce the novel morphological layer based on CHM filter formulation, referred to as PConv layer. For a single channel image and a single filter the PConv layer performs the following operation
(5) 
It is parametrized by , a scalar which controls the type of operation ( pseudoerosion, pseudodilation and standard linear convolution), and by the weighting kernel , where the corresponding asymptotic structuring function is given by . Since this formulation is differentiable we can use gradient descent on these parameters.
The gradient of such a layer is computed by backpropagation [20, 9]. In minimizing a given objective function , where
represents the set of parameters in the model, backpropagation applies the chain rule of derivatives to propagate the gradients down to the input layer, multiplying them by the Jacobian matrices of the traversed layers. Let us introduce first two partial results of backpropagation
(6) 
The gradient of a PConv layer is computed as follows
(7)  
(8) 
where indicate flipping along the two dimensions and indicates elementwise multiplication. The partial derivative of the PConv layer with respect to its input (to backpropagate the gradient) is instead
(9) 
Learning the tophat operator requires a shortcircuit in the network to allow for subtracting the input image (or the result of an intermediate layer) from the output of a filtering chain. For this purpose we introduce the AbsDiffLayer which takes two layers as input and emits the absolute difference between them. Partial derivatives can still be backpropagated.
3.1 Learning Algorithm
Minimizing a PConv layer is a nonconvex, highly nonlinear operation prone to local convergence. Deeplearning findings tell us that stochastic gradient descent is the most effective algorithm to train such complex models. In our experiments we use its full online version where weights are updated sample by sample. The learning rate decays during training. To further avoid bad local minima we use a momentum term. For the opening/closing tasks we also alternate between learning , keeping fixed, and viceversa. This is common in online dictionary learning and sparse coding. We also constrain .
4 Experiments
We thoroughly evaluate our MCNN on several tasks. First we assess the quality of dilation/erosion operators, which require a single PConv layer. This gives a good measure of how well training can be performed using the CHM derivation. Then we investigate a twolayer network learning openings/closings. This is already a challenging task hardly covered in previous approaches.
We then learn the tophat transform for a very challenging steel industry application. Using filters per layer we learn to simultaneously detect two families of defects without resorting to multiple training. Our implementation allows for learning multiple filters for every layer, thus producing a very rich set of filtered maps. Subsequent convolutional layers can learn highly nonlinear embeddings. (We believe that this will also dramatically improve segmentation capabilities of such models to be investigated in future work.) We also show that a simple CNN does not learn well pipelines of morphological operators. This is actually expected a priori due to the nature of conventional convolutional layers, and shows the value of our novel PConv layer.
As final benchmark we consider denoising. Our MCNN shows the superiority of operator learning over handcrafting structuring elements for nonGaussian (binomial and saltandpepper) noise. We also show that our approach performs well on total variation (TV) approximation for additive Gaussian noise.
In all our experiments we use stochastic gradient descent and a filter size of unless otherwise stated. The perpixel meansquared error loss (MSE) is used.
4.1 Learning dilation and erosion
In this first set of experiments we create a dataset as follows: for every input image we produce a target image using a predetermined flat structuring element and a predetermined operator: or . We train until convergence. Overfitting is not an issue in such a scenario. The net actually learns the true function underlying the data. In fact, for an image of and a structuring element with support of there are patches, way more than the elements in the structuring element. A CNN with equal topology fails, producing mainly Gaussian blurred images, illustrating the need for a PConv layer to handle this kind of nonlinearities. Figure 2 shows the results of a dilation with three structuring elements: a line of pixels and , a square of size pixels and a diamond of side pixels. Figure 3 shows similar results for the erosion transform.
Note that the learned weighted kernels are not exactly uniformly equal to . The corresponding morphological structuring function , obtained after applying the logarithm on the weights, produces a rather flat shape. In practice, we observed that learning an erosion is slightly more difficult than learning the dual dilation. This is related to the asymmetric numerical behavior of CHM for and . Nevertheless, in all cases the learned operator has excellent performance.
4.2 Learning opening and closing
In this set of experiments we train our system to learn openings and closings . Learning such functions is extremely difficult. To the best of our knowledge, we are the first to do this in a flexible and gradientbased framework without any prior. For instance, in classical approaches [13] or more recent ones [11], the operator needs to be fixed a priori.
Figure 4top shows an example of a closing with a line of length and an orientation of , whereas Figure 4bottom shows an example of an opening with a square of size . In both cases, the obtained kernel for the first and second PConv layers are depicted. We see that the associated structuring element is learned with a good approximation. On the other hand, however, we also start to see that learning a flat opening/closing is remarkably hard and that the network output starts to be slightly “blurry”. Why? On the one hand, the obtained values of , e.g., in the closing and , in the opening and are in the interval of asymptotically unflat behavior. On the other hand, they are not totally symmetric. We intend to further study these issues in ongoing work.
4.3 Learning tophat transform
Delegating the learning to a neural network allows for easily constructing complex topologies by linking several simple modules. We recall that the white tophat is the residue of the opening, i.e., , and the black topthat the residue of the closing, i.e., . Thus, to learn a tophat transform we introduce the AbsDiff
layer. It takes two layers as input and emits their absolute difference in output. Backpropagation is performed as usual.
Tophat is particularly relevant in real applications such as steel surface quality control. It is a powerful tool for defect detection. Here we first show that our framework can learn such a transform. Increasing the number of filters per layer from to , we show that our MCNN is also much more powerful when jointly learning two transforms.
Figure 5 shows the results for a single tophat. We create our training set by applying a white tophat , where is a disk of size pixels. This operator extracts only the structures of size smaller than and brighter than the background. We clearly see that the network performs almost perfectly. To further assess the advantages of a PConv layer over a conventional convolutional layer, we also train a CNN with identical topology. The discrepancy between the two models in terms of losses (MSE) is large: we have 1.28E3 for our MCNN and 1.90E3 for the CNN. More parameters are required for a CNN to reach better performance. This clearly establishes the added value of our MCNN.
The steel industry requires many tophat operations. Tuning them one by one is a cumbersome process. Furthermore, several models’ outputs need to be considered to obtain the final detection result. Figure 6 shows that by simply increasing the number of filters per layer we can simultaneously learn two tophat transforms and address this problem. We learn a white tophat with a disk of size and a black tophat with a line of size and orientation of . A convolutional layer is used to combine the output of the two operators. The architecture is as follows: 2 PConv layers, Conv layer, AbsDiff layer with the input. The network is almost perfect from our viewpoint. This opens the possibility of using such a setup in more complex scenarios where several morphological operators should be combined. This is of great interest in multiple class steel defect detection.
4.4 Learning denoising and image regularization
Finally we compare our MCNN to conventional morphological pipelines in the denoising task. Morphological filters are recommended for nonGaussian denoising. The purpose of this evaluation, however, is not to propose a novel noise removal approach, but to show the advantages of a learnable pipeline over a handcrafted one.
We start with binomial noise where % of the image pixels are switchedoff. The topology in this case is: PConv layers and filter size of . We compare to a closing with a square of size , empirically found to deliver best results. We make the task even harder by using largerthanoptimal support. Training is performed in fully online fashion. While the target images are kept fixed, the input is generated by adding random noise sample by sample. So the network never sees the same pattern twice. Figure 7–top compares the two approaches, and shows the noisy image. We see that learning substantially improves the PSNR measure.
We continue with an even more challenging task, a salt’n’pepper denoising. The network is made of PConv layers, a very long pipeline. We compare to an opening with a square of size on a closing with the same structuring element. Training follows the same protocol as the one for the binomial noise. Images are generated online. This creates a possibly infinite dataset with very small memory footprint. Figure 7–bottom shows results. Although we can observe some limitations of our approach, it still exhibits the best PSNR also in this application.
Finally we consider the case of total variation (TV) restoration from an image corrupted by % additive Gaussian noise. Morphological filtering does not excel at this task. A MCNN is trained to learn the mapping from noisy image to TV restored image. How well can it approximate any target transformation with a pseudomorphological pipeline? The architecture is composed of PConv layers with filters each plus an averaging layer. Results are shown in Figure 8.
5 Conclusion and Perspectives
Our MCNN for learning morphological operators is based on a novel PConv convolutional layer and inherits all the benefits of gradientbased deeplearning algorithms. It can easily learn complex topologies and operator chains such as white/black tophats and we showed its application to steel defect detection. In future work we intend to let MCNN simultaneously learn banks of morphological filters and longer filtering pipelines.
References
 [1] Angulo, J.: Pseudomorphological image diffusion using the counterharmonic paradigm. In: Proc. of ACIVS’2010, LNCS Vol. 6474, Part I. pp. 426–437. Springer (2010)
 [2] Bullen, P.: Handbook of Means and Their Inequalities, 2nd Edition. Springer (1987)
 [3] Cireşan, D.C., Meier, U., Masci, J., Schmidhuber, J.: A committee of neural networks for traffic sign classification. In: International Joint Conference on Neural Networks (IJCNN2011) (2011)

[4]
Cireşan, D.C., Meier, U., Masci, J., Schmidhuber, J.: Flexible, high performance convolutional neural networks for image classification. In: International Joint Conference on Artificial Intelligence (IJCAI2011) (2011)

[5]
Ciresan, D.C., Giusti, A., Gambardella, L.M., Schmidhuber, J.: Deep neural networks segment neuronal membranes in electron microscopy images. In: NIPS (2012)
 [6] Ciresan, D.C., Meier, U., Gambardella, L.M., Schmidhuber, J.: Convolutional neural network committees for handwritten character classification. In: ICDAR. pp. 1250–1254 (2011)
 [7] Harvey, N.R., Marshall, S.: The use of genetic algorithms in morphological filter design. Signal Processing: Image Communication 8(1), 55–71 (1996)
 [8] Hubel, D.H., Wiesel, T.N.: Receptive fields and functional architecture of monkey striate cortex. The Journal of physiology 195(1), 215–243 (March 1968)
 [9] LeCun, Y.: Une procédure d’apprentissage pour réseau à seuil asymétrique. Proceedings of Cognitiva 85, Paris pp. 599–604 (1985)
 [10] Masci, J., Meier, U., Cireşan, D.C., G., F., Schmidhuber, J.: Steel defect classification with maxpooling convolutional neural networks. In: International Joint Conference on Neural Networks (2012)
 [11] Nakashizuka, M., Takenaka, S., Iiguni, Y.: Learning of structuring elements for morphological image model with a sparsity prior. In: IEEE International Conference on Image Processing, ICIP 2010. pp. 85–88 (2010)
 [12] Pessoa, L.F.C., Maragos, P.: Mrlfilters: a general class of nonlinear systems and their optimal design for image processing. IEEE Transactions on Image Processing 7(7), 966–978 (1998)
 [13] Salembier, P.: Adaptive rank order based filters. Signal Processing 27, 1–25 (1992)
 [14] Salembier, P.: Structuring element adaptation for morphological filters. J. of Visual Communication and Image Representation 3(2), 115–136 (1992)
 [15] Scherer, D., Müller, A., Behnke, S.: Evaluation of pooling operations in convolutional architectures for object recognition. In: International Conference on Artificial Neural Networks (2010)
 [16] Serra, J.: Image analysis and mathematical morphology. Academic Press, London (1982)
 [17] Soille, P.: Morphological Image Analysis: Principles and Applications. SpringerVerlag Berlin (1999)
 [18] Turaga, S.C., Murray, J.F., Jain, V., Roth, F., Helmstaedter, M., Briggman, K., Denk, W., Seung, H.S.: Convolutional networks can learn to generate affinity graphs for image segmentation. Neural Comput. 22(2), 511–538 (Feb 2010)
 [19] Vliet, L.J.v.: Robust local maxmin filters by normalized powerweighted filtering. In: Proc. of IEEE ICPR’04, Vol. 1. pp. 696–699 (2004)
 [20] Werbos, P.J.: Beyond Regression: New Tools for Prediction and Analysis in the Behavioral Sciences. Ph.D. thesis, Harvard University (1974)
 [21] Wilson, S.S.: Training structuring elements in morphological networks. In: Mathematical Morphology in Image Processing, Chapter 1, pp. 1–42. Marcel Dekker (1993)