A New Padding Scheme: Partial Convolution based Padding
In this paper, we present a simple yet effective padding scheme that can be used as a drop-in module for existing convolutional neural networks. We call it partial convolution based padding, with the intuition that the padded region can be treated as holes and the original input as non-holes. Specifically, during the convolution operation, the convolution results are re-weighted near image borders based on the ratios between the padded area and the convolution sliding window area. Extensive experiments with various deep network models on ImageNet classification and semantic segmentation demonstrate that the proposed padding scheme consistently outperforms standard zero padding with better accuracy.READ FULL TEXT VIEW PDF
A New Padding Scheme: Partial Convolution based Padding
Simple Tensorflow implementation of "Partial Convolution based Padding" (partialconv)
auto adaptive framework for intrinsic hyperparameter selection, adaptive padding, normalized weights
Convolutional operation often requires padding when part of the filter extends beyond the input image or feature map. Standard approaches include zero padding (extend with zeros), reflection padding (reflect the input values across the border axis) and replication padding (extend by replicating the values along borders). Among them, the most commonly used scheme is zero padding, as was adopted by . Besides its simplicity, zero padding is also computationally more efficient than the other two schemes. Yet, there is no consensus on which padding scheme is the best yet. In this work, we conduct extensive experiments on the ImageNet classification task using these three padding schemes, and found that reflection padding and replication padding can get similar or sometimes worse performance compared with zero padding. While these three padding schemes allow the convolution to safely operate along the borders of the input, they incorporate extrapolated information, which may adversely affect the model’s quality.
Each of the three existing standard padding approaches makes assumptions that may have undesirable effects to the model quality in different ways. Zero padding works like adding extra unrelated data to the input. Reflection and replication padding attempt to pad with plausible data values by re-using what is along the borders of the input. These two types of padding lead to unrealistic image patterns since only some parts of the input are replicated. Moreover, for all these three padding schemes, the added or replicated features are treated equally as the original input, which makes it possible for the network to be confused.
To eliminate the potential undesired effects from the extrapolated inputs, we propose a new padding scheme called partial convolution based padding, which conditions the convolution output only on the valid inputs. In contrast to the zero padding where zeros are used for the missing inputs, our partial convolution based padding adaptively re-weights the output to adjust for the fraction of the missing data. To this end, our padding scheme treats the padded region as holes and applies the partial convolution layer  for the re-weighting in the convolution operation. We demonstrate through experiments on ImageNet classification that VGG and ResNet architectures gain better accuracy using partial convolution based padding than using zero padding. Furthermore, we observed that models trained with zero padding are very sensitive to the padding used during inference, and would have a big performance drop if different padding is used. This suggests that when zero padding is used, part of the parameters in the model are wasted dealing with the padded data. On the other hand, models trained with partial convolution based padding are robust, and perform similarly no matter which padding is used during inference.
Our main contributions can be summarized as follows:
We conduct extensive experiments to show that by replacing the zero padding with partial convolution based padding on various models and tasks, we obtain better results and faster convergence with fewer training iterations.
We show that among the three existing padding schemes, reflection padding and replication padding perform similarly with or worse than zero padding on ImageNet classification task. Padding scheme like zero padding is sensitive to the particular padding used during training. On the other hand, our partial convolution based padding are robust to the input padding type.
We demonstrate that partial convolution based padding can improve semantic segmentation on regions near image boundaries, suggesting that existing padding techniques result in lower accuracy as more of the receptive field is conditioned on padded values.
Researchers have tried to improve the performance of CNN models from almost all the aspects including different variants of SGD optimizer (SGD, Adam 
, RMSprop, Adadelta 
), non-linearity layers (ReLU, LeakyReLU , PReLU ), normalization layers (Batch Norm , Instance Norm , Layer Norm , Group Norm ), etc. However, little attention has been paid to improving the padding schemes. Innamorati et al.  augments networks with a separate set of filters to explicitly handle boundaries. Specifically, it learns a total of filters for a convolutional layer, one for the middle part and the rest for boundary cases. Cheng et al.  propose a special image projection to handle undefined boundaries resulting from projecting a view image onto a 2D image. Images are first projected onto a cube and a final image is formed by concatenating cube faces. As such, large undefined borders can be eliminated. Our proposed padding scheme is orthogonal to all of these tricks and can be used in conjunction with them. It also does not increase the parameter count; instead, it uses a single filter with a simple yet effective re-weighted convolution at boundary pixels.
Deep learning for inputs with missing data
. In our setup, the padded regions are interpreted as missing data. Handling inputs with missing data is related to the problems of image inpainting and other data completion tasks. Recent deep learning based methods[18, 14, 9, 30] initialize the holes (regions with missing data) with some constant placeholder values and treat the newly initialized hole region and original non-hole region equally. Then GAN based networks will be used to re-generate a new image conditioned on this initialized image. The results usually suffer from dependence on the initial hole values, with lack of texture in hole regions, obvious color contrasts or artificial edge responses. Many of them require post-processing, like a refinement network [32, 22], Poisson blending  or other post-optimization techniques . Some other methods [27, 31, 25] ignore the hole placeholder values in the image inpainting or data completion tasks; none of them has explicitly handled the missing data at the padded regions yet.
Reweighted convolution. Reweighted convolution as the variant of typical convolution has been explored in several tasks. Harley et al.  uses soft attention masks to reweight the convolution results for semantic segmentation. PixelCNN designs the reweighted convolution such that the next pixel generation only depends on previous generated pixels. Inpainting methods like [25, 19] take the hole mask into consideration for reweighting the convolution results. For all these methods, none of their corresponding reweighting mechanisms has been used to handle the padding yet.
In this section, we start with reviewing the partial convolution  and then illustrate the idea of partial convolution based padding.
Partial Convolution. Partial convolution  is originally proposed to handle incomplete input data, such as images with holes. Let be the feature values (pixel values) for the current convolution (sliding) window at the position and be the corresponding binary mask with the hole region being 0 and non-hole region being 1. The partial convolution (ignoring bias) at every location is defined as:
denotes element-wise multiplication,
is the all-one vector with the same shape asand is the filter weight matrix. We compute to account for an additional bias term (when ). As can be seen, output values depend only on the unmasked inputs. The scaling factor applies appropriate scaling to adjust for the varying amount of valid (unmasked) inputs.
After each partial convolution operation, we then update our mask as follows: if the convolution was able to condition its output on at least one valid input value, then we mark that location to be valid. This is expressed as:
and can easily be implemented in any deep learning framework as part of the forward pass. With sufficient successive applications of the partial convolution layer, any mask will eventually be all ones, if the input contained any valid pixels.
Partial Convolution based Padding. Partial convolution is extended to handle padding by defining the input region to be non-hole and padded region to be holes. Specifically, given the input image/feature at the border, let denotes the 2D matrix having same height and width as , but with a single channel; denotes the zero padded result of ; denotes the zero padded result of ; denotes the one padded result of ; the visualization of their examples can be found in Figure 2; then the convolution result is computed as following:
Based on Equation 4, we can see that the widely used zero padding is a special case of our partial convolution based padding by directly setting to be 1, which can be formulated as following:
Case of Big Padding Size. In some cases, big padding will be needed. For example, input images may have different image sizes; big paddings may thus be needed to make images uniformly sized before feeding them into the network. Re-sizing by padding is typically preferred because normal re-sizing may introduce distortions due to altered aspect ratio. Convolution at such borders may not have valid/original data because the window size could be smaller than the padding size. For such cases, we follow the original partial convolution formulation in  to include the mask updating step. Specifically, for the very first convolutional layer, the input mask will be the padded result ( = ); and it will produce an output mask using the rule in Equation 3. The mask input for the next layer will be the padded result of , namely = . Thus, the input mask for layer will be = .
. We provide a pure-PyTorch implementation of the convolution layer with the proposed padding scheme on top of existing PyTorch modules. We implement the mask of ones (
) as a single-channel feature with the same batch size, height and width as the input tensor. The 1-padded version of () is directly set to be , where and are the height and width of the kernel. is implemented by calling the convolution operation once with all the weights being 1, bias being 0 and original target padding size. The result of only needs to be computed at the first time and the result can be cached for future iterations as long as the input size does not change. Thus, the execution time starting from the second iteration will be lower than the first iteration. In Table 1, We show the comparison of GPU execution time between zero padding powered networks and partial convolution based padding powered networks for both the first iteration and after iterations. It can be seen that starting from the second iteration partial convolution powered vgg16 (batch normalization layer version) and resnet50 only cost about % more time to do the inference on a input image with a single NVIDIA V100 GPU. Note that this current implementation is based on existing PyTorch interfaces to allow users to use without compiling new CUDA kernel. If this padding scheme is implemented using CUDA directly, the extra cost would be negligible as we only need to re-weight the conovolution results at the border.
|vgg16BN_partial (nd - th)||14.351||104.06%|
|resnet50_partial (nd - th)||5.975||103.39%|
The best top-1 accuracies for each run with 1-crop testing. *_zero, *_partial, *_ref and *_rep indicate the corresponding model with zero padding, partial convolution based padding, reflection padding and replication padding respectively. *_best means the best validation score for each run of the training. Column average represents the average accuracy of the 5 runs. Column diff represents the difference with corresponding network using zero padding. Column stdev represents the standard deviation of the accuracies from 5 runs. PT_official represents the corresponding official accuracies published on PyTorch website:https://pytorch.org/docs/stable/torchvision/models.html
The mean of last 5 epochs’ top-1 accuracy () for each run with 1-crop testing. *_zero, *_partial, *_ref and *_rep indicate the corresponding model with zero padding, partial convolution based padding, reflection padding and replication padding respectively. *_last5 means the mean validation score of the last 5 epochs’ model checkpoints for each run of the training. Column average represents the average accuracy of the 5 runs. Column diff represents the difference with corresponding network using zero padding. Column stdev represents the standard deviation of the accuracies from 5 runs.
Experiments. We conduct the experiments and comparisons on the ImageNet classification tasks. We train the VGG16, VGG19 , ResNet50, ResNet101, ResNet152  models on ImageNet training dataset and evaluate on the ImageNet validation dataset. We use the versions of VGG16 and VGG19 with Batch Normalization layers, as the VGG network with batch normalization is known to be less sensitive to learning rates and initializations, thereby reducing the amount of variation in our experiments. We use the raining scripts and model definitions from the corresponding official PyTorch examples. For each model, we replace all the zero padding with partial convolution based padding while keeping all the other settings and training parameters the same. While the default PyTorch training script trains for epochs, we use in our experiments like . The models are all trained with NVIDIA V100 GPUs on DGX-1 workstations. The initial learning rate is set to be and decreased by a factor of at , and epochs. We train each network
times to account for variances due to initializations and stochastic mini batch sampling. To have a full comparison with all the existing padding schemes, we also run the experiments for reflection padding and replication padding with the same setting. This gives us a total ofsets of trained ImageNet model weights ( architectures ). For each training run, we select the checkpoint with the highest validation set accuracy to represent that run (referred to as “best”).
In addition, we also report results using the experimental procedures from Wu et al. . Specifically, Wu et al.  trained the network once and evaluates the performance on the model and accounts for model variance by reporting the average performance of the final epochs. As such, we also report results in which we average across the last epochs for each run (referred to as “last_”).
Analysis. We evaluate the models using the top-1 classification accuracy based on the single center crop evaluation. Table 2 shows the results for the “best” scenario, whereas Table 3 shows the results for the “last_” setting. Note that the top-1 accuracy of all the baseline models from our training runs closely matched those reported in official PyTorch documentation111https://pytorch.org/docs/stable/torchvision/models.html, shown in the last column in Table 2 under PT_official.
It can be seen that partial convolution based padding (*_partial) provides better validation accuracy than the default zero padding (*_zero) for all the models. The network of VGG19 with partial convolution based padding has the largest improvement with 0.68% accuracy boost. For the ResNet family, ResNet50 model has the largest improvement (0.478%) compared with ResNet101 (0.36%) and ResNet152 (0.248%). This may be contrary to our straightforward intuition that deeper networks would benefit more from our proposed padding scheme, which is reflected in the comparison between VGG19 (0.687%) and VGG16 (0.316%) to some extent. On the other hand, this can also be interpreted as shallower networks have more potential and room for improvement. Furthermore, we found that reflection padding or replication padding leads to similar accuracies, shown from the nd row to the th row in Table 2. We will show the reflection padding and replication padding results of other networks in the supplementary file.
From Tables 2 and 3, it can also been seen that models with partial convolution based padding have smaller standard deviation than the ones with zero padding. It means that partial convolution based padding makes model’s performance stable and robust to randomness.
Convergence Speed. Besides obtaining better accuracy, the model with partial convolution based padding can also achieve the similar/same accuracy with fewer iterations/epochs compared to the model with zero padding. In Figure 3, we plot the average testing/validation accuracy for every epoch’s model checkpoint among the 5 runs of both ResNet50 with partial convolution based padding and ResNet50 with zero padding. Blue lines show the accuracy of the models with partial convolution based padding; yellow lines show the accuracy of the models with zero padding. It can be seen that:
The model with partial convolution based padding takes fewer iterations/epochs to achieve the similar/same accuracy; for example, to achieve the accuracy at ; the model with partial convolution based padding takes 63 epochs to achieve the score while the model with zero padding takes 68 epochs.
The training of the model with partial convolution based padding is more stable with fewer drifts compared to the model with zero padding shown as the blue and yellow lines in Figure 3.
Activation Map Visualization. Figure 4 shows the image examples whose predictions fail for all the 5 runs of resnet50 with zero padding, but succeed for all the 5 runs of resnet50 with partial convolution based padding. The same activation maps of the last column’s two images are shown in Figure 5. It can be seen that for the zero padding based network, the features at the border have the strongest activation responses, which easily confused the network prediction. As these border features are largely dependent on the padded zero values from the previous layers, this could to some extent also imply that there is a chance that the padded zero values could mislead the network.
Cross Testing. One feature of partial convolution based padding is that it does not introduce extra parameters. Thus, it allows us to perform such cross testing that we use the network weights trained with one padding scheme to do the inference with the other padding schemes. Table 4 shows such cross testing on resnet50_zero and resnet50_partial (resnet50_zero with partial convolution based padding); it can be seen that if the training is using partial convolution based padding while the inference is using zero padding, there is only a accuracy drop; to some extent, it is even comparable (close) to the accuracy obtained by using zero padding for both training and testing. However, if the training is using zero padding and the inference is using partial convolution based padding, there would be a big accuracy drop with . One possible explanation could be that the model weights trained with zero padding are sensitive to the image/feature value (scale) changes at the image/feature border; on the other hand, the model trained with partial convolution based padding is less sensitive to such changes. It may also be interpreted as models with zero padding have paid more efforts to resolve the influence brought by adding zero values at the border.
|default split||additional split|
Semantic segmentation involves densely classifying each pixel with its corresponding semantic category. Most recent semantic segmentation networks use an encoder-decoder architecture[20, 2] to ensure the output dimensions match those of the input. It is common to employ an ImageNet pretrained classifier such as ResNet50 as the backbone for the encoder part, followed by a series of deconvolutions or upsampling layers for the decoder. Padding is essential in guaranteeing same input-output dimensions as the center of the filter would not be able to reach the edge pixels otherwise.
Some networks also use dilated convolutions to achieve larger receptive fields without downsampling or increasing the number of filter parameters. For dilated/atrous convolution, large padding is usually needed, increasing the dependence on padded values. Our partial convolution based padding formulation is general and is easily adapted to dilated convolutions.
Dilated convolutions are used in DeepLabV3+, one of the state-of-the-art semantic segmentation networks. DeepLabV3+ uses pretrained ImageNet classifier like Xception  or ResNet as the encoder backbone. The backbone features are fed into an dilated-convolution-based Atrous spatial pyramid pooling module (ASPP) to complete the encoding. Next, a decoder with the skip links to the encoder features and final upsampling is used to upscale the output to the original input size. We train and test a DeepLabV3+ model on the CityScapes semantic segmentation dataset. CityScapes contains images with pixel-level annotations. The default splits are for the training set, for the validation set and for the test set. It also contains coarsely annotated images.
|Input||G.T. Segmentation||zero padding||partial conv based padding|
|Padding||Regular||0 Tile overlap||Tile overlap|
To keep our experimentation simple and focused on the differences between regular and partial convolution based padding, we do not employ external datasets for pre-training as is done to achieve state-of-the-art performance in works such as .
The Cityscapes dataset contains training data from different cities. The default train-val split creates an / train/val split by cities. We create an additional second / split to experiment on as well. Our segmentation network architecture is based on DeepLabV3+ 
with output stride offor the encoder. Motivated by Mapillary , we evaluate partial convolution based padding using WideResnet38  and Resnet50 for our encoder. We also use a data sampling strategy similar to  and use the 20k coarsely annotated images along with the finely annotated images. We run the segmentation network for 31K iterations with SGD with an initial learning rate of and for Resnet50 and WideResnet38 respectively and with a polynomial learning rate decay of . Our momentum and weight decay are set to and respectively. Similar to other semantic segmentation papers [35, 34, 4], we use the following augmentations during training: random scaling, horizontal flipping, Gaussian blur, and color jitter. Our crop size was set to 896 and 736 for Resnet50 and WideResnet38 respectively. Lastly to due to the large crop size, we use a batch size of with synchronized BatchNorm (for distributed training), similar to PSPNet .
Analysis. We compare and evaluate the segmentation models using mIOU metric (). The mIOU is the mean IOU across 19 classes for cityscapes. The ASPP module includes a spatial pooling step that outputs a single 1-D feature vector. During training, this pooling procedure operates on square crops (1:1 aspect ratio). However, during full-image inference, we must now account for the fact that the full images are of size 10242048 with an aspect ratio of 1:2. This means that the pooling module, trained to pool over 1:1 aspect ratios, must now pool over 1:2 aspect ratio input at test time. We resolve this by breaking down the full image into overlapping square tiles of 1:1 aspect ratio. We report two types of results:
regular: directly feed the image into the network regardless of the aspect ratio issue.
tile: dividing the images into square tiles of size .
Tile based evaluation is also used in the work  to obtain better evaluation performance. In Table 5, we show that our segmentation models using partial convolution based padding with Resnet50 and WideResnet38 encoder backbones achieve better mIOU on full image (regular). Resnet50 encoder based segmentation model trained using partial convolution based padding achieved and higher mIOU on the default and additional split respectively. We also observe similar performance gains with WideResnet38+partial convolution based padding outperforming its counterpart by and in the default and additional split respectively.
In Table 6, we see that the tile based evaluation gives better mIOU than the regular evaluation even though their mIOUs on regulari evaluation mode are similar.
Advantage of Partial Convolution based Padding for Tile base Evaluation: One major concern for tile-based evaluation is that by subdividing the image, we significantly increase the number of pixels that lie near the boundaries of the input. As previously stated, this can hurt performance because pixels will have incomplete context when the receptive field extends beyond the image boundaries. We ameliorate this by sampling tiles with overlapping context – allowing a boundary pixel in one tile to be re-sampled near the center in the neighboring tile.
While overlapping tiles can serve to de-emphasize the boundary issue, we demonstrate in Table 6 that the partial convolution based padding models demonstrate a much larger improvement from tiling evaluation. This is because the latter type of model is more robust to the boundary issue in the first place, and thus was much less affected by the increase in pixels near borders. For both the evaluation with non overlapping between tiles and the evaluation with overlapping between tiles, the model with partial convolution based padding is around 0.37% better than the model with zero padding, despite both having similar mIOUs in the regular evaluation scheme.
In Figure 6, we show segmentation comparisons between partial convolution based padding and zero padding for WideResnet38 for the tile based evaluation mode with overlapping. In Figure 8, we show the segmentation comparisons for the tile based evaluation mode without overlapping. It can be seen that partial convolution based padding leads to better segmentation results on border regions.
|center leave-out proportion|
Evaluation Excluding the Center Regions: To better understand the advantage of partial convolution based padding in handling the border regions, we perform some additional evaluations which only evaluated the mIOUs on the border regions. Specifically, we set the target labels to “don’t care” for the center region of the image at varying proportions: , , , , , . Samples of different proportions of leaving out center regions can be found in Figure 7. These evaluations use non-overlapping tiling. Similar to the Table 6 in the main paper, we select two WideResNet38-backbone models with different padding schemes but similar mIOU in the regular evaluation setting. Table 7 shows the corresponding evaluation results by leaving out different proportions of the center regions. It can be seen that as we leave out more proportions of the center region, the evaluation difference between zero padding and partial convolution based padding becomes larger. For example, if we only leave out center regions, the partial convolution based padding only outperform the zero padding with mIOU. However, when we leave out center regions, the difference becomes . This further demonstrates that the partial-convolution based padding scheme significantly improves prediction accuracy near the image boundaries.
|Input||G.T. Segmentation||zero padding||partial conv based padding|
In this work, we present a new padding scheme called partial convolution based padding. We extensively evaluate the effectiveness of the partial convolution based padding compared to existing padding methods. We demonstrate that it outperforms the widely adopted zero padding through intensive experiments on image classification tasks and semantic segmentation tasks. We show that partial convolution based padding achieves better accuracy as well as faster convergence than the default zero padding on image classification. We also show that partial convolution based padding gets better segmentation results, especially at the image border, while the typical zero padding may reduce the model’s certainty.
Acknowledgement We would like to thank Jinwei Gu, Matthieu Le, Andrzej Sulecki, Marek Kolodziej and Hongfu Liu for helpful discussions.
Proceedings of the 27th international conference on machine learning (ICML-10), pages 807–814, 2010.