Marine Snow Removal Benchmarking Dataset
This paper introduces a new benchmarking dataset for marine snow removal of underwater images. Marine snow is one of the main degradation sources of underwater images that are caused by small particles, e.g., organic matter and sand, between the underwater scene and photosensors. We mathematically model two typical types of marine snow from the observations of real underwater images. The modeled artifacts are synthesized with underwater images to construct large-scale pairs of ground-truth and degraded images to calculate objective qualities for marine snow removal and to train a deep neural network. We propose two marine snow removal tasks using the dataset and show the first benchmarking results of marine snow removal. The Marine Snow Removal Benchmarking Dataset is publicly available online.READ FULL TEXT VIEW PDF
Marine Snow Removal Benchmarking Dataset
Image restoration has been one of the main topics of computer vision for decades[36, 10, 27, 33, 11, 34, 35]
. In the current era of deep learning, the quality of image restoration for various tasks have been significantly improved under a sufficient number of pairs of ground-truth and degraded images. Therefore, the current focus on image restoration will be for those taken under extreme situations such as underwater, satellite, and medical images[1, 6, 23, 30, 31, 22, 14, 20, 12, 37, 39, 26, 29, 25, 32, 3]. This problem is called extreme image restoration herein. Extreme images often have an untypical degradation beyond that of a Gaussian model, and the numbers of available images for training and modifying restoration algorithms are also limited. As a result, extreme image restoration is still a challenging problem.
In this paper, we focus on underwater image restoration, the main challenge of which is mostly due to the fact that we generally have no ground-truth images. This makes it difficult to calculate the objective qualities of the restored images and measure the restoration performance. Moreover, the lack of image pairs causes a difficulty in training a neural network (if we consider using a deep neural network for restoration) because the loss functions of the image restoration typically contain some objective image quality metrics such as mean squared error (MSE) and structured similarity index (SSIM)
. Hence, unsupervised restoration methods, including linear and nonlinear filtering, are still popular in underwater image restoration[6, 23, 2].
Several studies for underwater image restoration with deep neural networks have been proposed [30, 31, 22, 14, 20]. Most focus on removing the color shift from underwater images. That is, the blueish underwater images are restored by enhancing the red and green channels. Conventional underwater image enhancement methods using deep neural networks tackle the problem on the lack of large-scale underwater image datasets in various ways. For example, the studies in [31, 4, 30, 14] synthetically generate degraded images from clean images taken on the ground by simulating the degradation processes of absorption and scattering. Moreover, [20, 24, 15]
uses a generative adversarial network. Although a number of challenges to improving the restoration performance still exist, some promising results have been presented. However, such studies omit another main source of degradation for underwater images:Marine snow.
In this paper, we consider a marine snow removal (MSR) problem for underwater images by constructing a large-scale dataset containing underwater image pairs, as shown in Fig. 1. Marine snow is a typical underwater image degradation, examples of which are shown in Fig. 2. Marine snow artifacts are visible in digital images when we take an underwater image using a flashlight. Small particles in the scene, e.g., organic matter and sand, cause marine snow. The particles reflect the light from the flashlight, and the reflected light will be captured by the photosensors. Marine snow is often annoying in underwater photography because it affects the overall image quality. However, the particles are unevenly distributed in the scene, and the degradation process does not have a typical statistical property. Therefore, standard methods for marine snow removal are still limited to simple median-filter-based approaches [5, 13, 21]. Moreover, as with standard underwater image restoration, we do not have a pair of ground-truth and degraded image pairs to calculate the objective image qualities and tune the parameters of the restoration algorithms.
We propose marine snow synthesis methods for the above-mentioned problems of MSR. In other words, we intentionally append the marine snow artifacts to real underwater images and generate a pair of ground-truth images (i.e., clean underwater images) and degraded images (those with marine snow artifacts). The key for synthesizing artifacts is to model real marine snow in digital images as accurately as possible. We discover that many marine snow artifacts can be mathematically modeled through two typical pixel value distributions. By utilizing this fact, we can synthetically generate many pairs of underwater images with/without marine snow artifacts.
We also propose two marine snow removal tasks and develop datasets corresponding to these tasks. This Marine Snow Removal Benchmarking Dataset (MSRB Dataset) is the first attempt in this field and is publicly available online111https://github.com/ychtanaka/marine-snow. We also present the first MSR benchmarking results using the MSRB Dataset: We compare the objective performance of marine snow removal among median filters and a deep neural network. Real MSR results as well as the limitations are also described.
The remainder of this paper is organized as follows: Section 2 reviews several related studies for underwater image restoration. Two representative marine snow models that we discovered are presented in Section 3 along with their synthesizing method. The MSRB Dataset specifications are introduced in Section 4 and include two tasks. The first MSR benchmarking results with objective image quality metrics are shown in Section C. Real MSR results and limitations are described in Section 6. Finally, we provide some concluding remarks in Section 7.
There have been few studies on modeling marine snow in digital images. The seminal study on modeling marine snow is described in [7, 8]. A marine snow artifact is modeled using a Gaussian function in which the artifact is less transparent in its center than its surrounding area. This method is better than the traditional salt and pepper noise; however, it does not simulate the actual shapes of marine snow artifacts. As a result, the center part of marine snow becomes excessively thick and the resulting images are unrealistic.
MSR is also an underrepresented problem in underwater image restoration. Here, we briefly review some existing approaches.
A widely used method for MSR is a median filter (MF) [9, 19]. If marine snow artifacts can be assumed sufficiently small (typically – pixels in diameter), MF is expected to work because small marine snow artifacts can be considered to have a salt-and-pepper effect. However, it significantly blurs the entire image, particularly when we use a large filter kernel size.
Specific to MSR, a modified version of a MF is proposed in [5, 17]. This method applies the MF selectively if the target pixel has a higher intensity than the surrounding pixels. However, it is still difficult to remove large marine snow artifacts.
A few MSR methods for video sequences have also been proposed. They utilize the fact that marine snow artifacts continuously move in consecutive video frames. In , background modeling is used to remove marine snow artifacts from a static scene. Marine snow artifacts are tracked and a customized MF is applied to the detected artifacts in . However, they cannot be used for a single underwater image.
Importantly, all methods above are based on a model-based approach, and there have been no methods to remove marine snow artifacts based on deep neural networks. This is mainly due to a lack of high-quality datasets, as mentioned in Section 1.
In the following, we address the contributions of this study; marine snow models and the dataset specifications.
There are various sources of marine snow and it is impractical to estimate the sources of all particles from a single underwater image. Instead of the estimation of the sources, we model the pixel value distributions of marine snow artifacts from observations of underwater images.
First, we start by showing marine snow examples in real underwater images. Figs. 3(a) and (c) show enlarged portions of representative marine snow artifacts cropped from real underwater images. Although they look similar, their pixel value distributions are slightly different.
Taking a closer look, the 3D plots of Figs. 3(a) and (c) are shown in Figs. 3(b) and (d), respectively. As clearly observed, they do not have a shape like a Gaussian function in contrast to the conventional assumption in [8, 7]. Rather than a Gaussian function, these 3D plots are similar to elliptic conical frusta, i.e., sliced elliptic cones. Furthermore, the top surfaces of the frusta have different characteristics between Figs. 3
(b) and (d). In our preliminary observation, most marine snow artifacts can be classified into these two representative shapes. In the following, we present the mathematical models of marine snow artifacts that reflect the above-mentioned observations.
The first marine snow model is the Highland Type (type H). The type H marine snow corresponds to Fig. 3(a) and can be modeled as an elliptic conical frustum with a rough surface.
For modeling type H marine snow, first suppose that we have two ellipses and , both centered at the coordinates , where and represent the row and column indices of an image, respectively. As shown in Fig. 4, we assume the focal points of and are located on the horizontal axis, i.e., the ellipses are wider, for simplicity. Let and be the semi-major and semi-minor axes of , respectively. Suppose that is larger than , i.e., and and the eccentricities of and are identical, i.e., .
For notational simplicity, is set to hereafter. The th pixel value of the type H marine snow is formulated as follows:
where is a constant that determines the transparency of the marine snow, calculates the Euclidean distance between two points, and is a small perturbation that mimics a rough surface. Furthermore, and () is the intersection of the ellipse and the straight line crossing , as illustrated in Fig. 4.
An example of the synthesized type H marine snow artifacts is shown in Fig. 5(a) along with its 3D plot in Fig. 5(b). See the similarity in Figs. 3(a) and (b). For creating a dataset, the major axis of is randomly rotated and the transparency is also randomly chosen. These specifications are shown later in Section 4.
The second marine snow model is the volcanic crater type (type V). An example of type V marine snow is shown in Figs. 3(c) and (d). This can be modeled by a modified version of the type H with an overshooted top edge of the frustrum.
For the type V marine snow, we consider another ellipse , where and are its semi-major and semi-minor axes, respectively. It is located inside , as illustrated in Fig. 4. As for the type H, focal points of all ellipses are assumed to be located along the horizontal axis and their eccentricities are identical, i.e., .
Based on the type H function , the th pixel value of the type V marine snow is formulated as , where is the rim term defined as
where is the constant for the maximum rim height and indicates the coordinates of the intersection of the ellipse and the straight line crossing .
By using the marine snow models introduced in the previous subsection, we synthesize the marine snow with real underwater images.
Let be an original underwater image. First, we randomly select a target pixel coordinate to append the marine snow. Second, we randomly select the major and minor axes of the ellipses and . In addition, the direction of the major axis is randomly set. Third, for each channel, type H or V marine snow is independently generated, where the parameters and are independently set. We denote an image with one marine snow artifact as , where the elements in are all-zero except the marine snow artifact. Finally, and are combined as follows to yield the synthesized image :
where is the Gaussian filtered version of that reflects blur owing to marine snow.
We iterate the process until a predetermined number of marine snow artifacts are appended.
In this section, we present the specifications of synthesized marine snow artifacts in the MSRB Dataset. It is designed for two tasks:
Removal of small-sized marine snow artifacts.
Removal of various-sized marine snow artifacts.
Clearly, Task 2 is more difficult than Task 1.
First, detailed parameters and setups shared in both tasks are presented. Second, we introduce MSR Tasks with corresponding synthesized images.
|Gaussian filter size in (3)||where|
|is obtained from the eccentricity of|
|is obtained from the eccentricity of|
is a continuous uniform distribution betweenand .
Each sub-dataset corresponding to a MSR Task contains training image pairs and test image pairs, all having a pixel resolution of . An image pair contains one original underwater image and one image containing synthesized marine snow artifacts. All original images are collected from flickr222https://www.flickr.com/ under a Creative Commons Attribution-NonCommercial-ShareAlike 2.0 Generic (CC BY-NC-SA 2.0) License.
Each synthesized image contains marine snow particles where is chosen from , in which is a discrete uniform distribution between and
. In each synthesized image, type H and V marine snow particles are randomly generated with a probability offor type H and for type V, according to our preliminary observations. The other representative parameters introduced in the previous section are listed in Table 3. Most of the parameters are determined according to our preliminary observations of real underwater images with marine snow artifacts.
We introduce the first MSR Task: Removal of small-sized marine snow artifacts. In Task 1, the maximum width/height of the artifacts is restricted to pixels, which correspond to roughly % of the image width/height. Task 1 is designed for underwater scenes where particles are relatively far from the photosensors. Because the task is relatively simple, the conventional MF approach is expected to be successfully applied (at least partially).
Examples of test image pairs in MSR Task 1 are shown in Fig. 12. As clearly visualized, the synthesized images have various sized (but small) marine snow artifacts.
The second MSR task is designed for underwater scenes containing particles in various distances. As a result, the synthesized images for Task 2 have small- and large-sized marine snow artifacts. For large-sized artifacts, we set the largest width/height of marine snow to pixels, which corresponds to % compared to the image width. Furthermore, the probabilities of small- and large-sized artifacts are set to and , respectively.
Training image examples for Task 2 are also shown in Fig. 12. The images have various-sized artifacts and the degraded areas are larger than the images in Task 1.
In the following, we show the benchmarking results of MSR as well as the real MSR.
In this section, we present the first benchmarking performance for MSR Tasks 1 and 2. The methods used for the benchmarking are 1) MF , 2) adaptive MF , and 3) U-Net . The kernel size of MFs is set to or pixels. Thanks to the MSRB Dataset, we can train a deep neural network for removing (synthesized) marine snow. For a benchmarking purpose, we train U-Net  using the dataset. The parameter settings are slightly different from the original and are shown in the appendix along with the network architecture.
The MSR results for Tasks 1 and 2 are shown in Figs. 7 and 8, respectively. As clearly observed, MF oversmoothes the images, which result in blur even for regions without marine snow artifacts. The adaptive MF suppresses blur; however, many artifacts are not removed mainly owing to the threshold-based algorithm. Furthermore, both MFs do not operate successfully for large marine snow artifacts in Task 2. By contrast, U-Net successfully removes marine snow for both tasks. Even for Task 2, it is able to suppress the large marine snow artifacts because of its multiresolution structure and the large number of trainable parameters.
For an objective comparison, we compute the average PSNRs and SSIMs over the test datasets. We summarize the results in Table 2. The objective measure indicates that U-Net is superior to the MFs for both MSR Tasks. Note that all MFs present smaller PSNRs/SSIMs than those of the synthesized images because they filtered areas without marine snow. By contrast, U-Net achieves a better objective quality than the synthesized images, which implies the effect of a deep neural network for MSR.
|Task 1||Task 2|
|MF (3 3)||28.55||0.846||22.81||0.770|
|MF (5 5)||25.98||0.711||21.93||0.645|
|Adaptive MF (33)||29.88||0.910||23.35||0.842|
|Adaptive MF (55)||28.08||0.861||22.83||0.794|
We construct the MSRB Dataset to mimic real marine snow artifacts. In this section, we show the real MSR results and present limitations of the dataset.
Fig. 9 shows MSR results for some real images. We use MF, adaptive MF, and U-Net as in the previous section. U-Net is trained using MSRB Task 2 Dataset.
As clearly observed, MFs cannot suppress marine snow artifacts because of their various sizes and different transparencies. U-Net presents a better restoration image quality than MFs: It (partly) suppresses artifacts having both small and large sizes. This implies the effectiveness of deep-learning-based methods trained using the MSRB Dataset for real MSR.
Note that thick marine snow artifacts remain in the restored images. The reason for this could be two-fold: First, U-Net is not specifically designed for MSR although we slightly customized its structure. A neural network designed specifically for MSR would improve the performance. Second, our proposed model classifies marine snow artifacts into two representative types. Marine snow artifacts not fitted to these types are not well removed. This limitation is described in the next section.
Because our MSR benchmarking dataset is based on synthesizing marine snow artificially into real underwater images, some limitations exist. First, the dataset is designed for removing marine snow artifacts and does not aim at achieving other underwater image enhancements such as color correction. The design of a simultaneous restoration of MSR and color correction is an interesting area of future study.
Although U-Net trained using the MSRB Dataset presents promising results, the dataset itself is imperfect. Specifically, real MSR performances are occasionally limited if MSR artifacts are not fitted to our marine snow models. Examples are shown in Fig. 10, which compares real underwater images and their restoration results using U-Net. Typically, U-Net fails to suppress artifacts when marine snow artifacts are larger and denser than our MSR benchmarking dataset. Moreover, if the lighting conditions differ from the dataset, the current version of U-Net will fail. Through a future study, this can be improved by carefully updating the MSRB Dataset to reflect various underwater scenes.
In this paper, the Marine Snow Removal Benchmarking Dataset was proposed. We mathematically modeled two representative marine snow artifacts and synthesized them in real underwater images. Two MSR tasks, the removal of small-sized and various-sized marine snow artifacts were also proposed. The first benchmarking results were also shown, which revealed the effectiveness of a deep neural network for MSR compared to median filter-based methods. MSR for real underwater images indicates that our MSRB Dataset contributes to real MSR problems. Our future studies will include improving the marine snow models as well as designing deep neural networks specifically designed for MSR.
This work was supported in part by JST PRESTO under grant JPMJPR1935, JSPS KAKENHI under grant 20H02145.
The structure of U-Net used for our marine snow removal experiments are illustrated in Fig. 11. This is basically the same as the original U-Net but we slightly changed the number of convolution layers and that of channels in each layer. Also, we initialize the weights of convolution layers by the method in 
. The hyperparameters used are summarized in Table3.
Fig. 12 shows additional examples from Marine Snow Removal Benchmarking Dataset (MSRB Dataset) both for MSR Tasks 1 and 2.
Number of epochs
|Weight initializing method|||
|Initialized bias values||0|
Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6723–6732, 2018.
Vehicle detection in satellite images by hybrid deep convolutional neural networks.IEEE Geoscience and remote sensing letters, 11(10):1797–1801, 2014.
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification.In 2015 IEEE International Conference on Computer Vision (ICCV), pages 1026–1034, Dec. 2015.
Semantic image inpainting with deep generative models.In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5485–5493, 2017.