We propose a multigrid extension of convolutional neural networks (CNNs). Rather than manipulating representations living on a single spatial grid, our network layers operate across scale space, on a pyramid of grids. They consume multigrid inputs and produce multigrid outputs; convolutional filters themselves have both within-scale and cross-scale extent. This aspect is distinct from simple multiscale designs, which only process the input at different scales. Viewed in terms of information flow, a multigrid network passes messages across a spatial pyramid. As a consequence, receptive field size grows exponentially with depth, facilitating rapid integration of context. Most critically, multigrid structure enables networks to learn internal attention and dynamic routing mechanisms, and use them to accomplish tasks on which modern CNNs fail. Experiments demonstrate wide-ranging performance advantages of multigrid. On CIFAR and ImageNet classification tasks, flipping from a single grid to multigrid within the standard CNN paradigm improves accuracy, while being compute and parameter efficient. Multigrid is independent of other architectural choices; we show synergy in combination with residual connections. Multigrid yields dramatic improvement on a synthetic semantic segmentation dataset. Most strikingly, relatively shallow multigrid networks can learn to directly perform spatial transformation tasks, where, in contrast, current CNNs fail. Together, our results suggest that continuous evolution of features on a multigrid pyramid is a more powerful alternative to existing CNN designs on a flat grid.READ FULL TEXT VIEW PDF
Recently, convolutional neural networks (CNNs) have been used as a power...
Existing deep convolutional neural networks (CNNs) require a fixed-size
Deep convolutional neural networks (CNNs) have demonstrated impressive
Accurate lesion detection in computer tomography (CT) slices benefits
In convolutional neural networks (CNNs), the filter grouping in convolut...
Recent work has shown that depth estimation from a stereo pair of images...
Given the variety of the visual world there is not one true scale for
Since Fukushima’s neocognitron , the basic architectural design of convolutional neural networks has persisted in form similar to that shown in the top of Figure 1. Processing begins on a high resolution input, of which filters examine small local pieces. Through stacking many layers, in combination with occasional pooling and subsampling, receptive fields slowly grow with depth, eventually encompassing the entire input. Work following this mold includes LeNet , the breakthrough AlexNet , and the many architectural enhancements that followed, such as VGG , GoogLeNet , residual networks [11, 12, 38] and like [22, 15]. Coupled with large datasets and compute power, this pipeline drives state-of-the-art vision systems.
However, sufficiency of this design does not speak to its optimality. A revolution in performance may have blinded the community to investigating whether or not unfolding computation in this standard manner is the best choice. In fact, there are shortcomings to the typical CNN pipeline:
The fine-to-coarse processing within a standard CNN is in opposition to a near universal principle for efficient algorithm design: coarse-to-fine processing. The first layer in a standard CNN consists of many filters independently looking at tiny, almost meaningless regions of the image. Would it not be more reasonable for the system to observe some coarse-scale context before deciding how to probe the details?
Communication is inefficient. A neuron’s receptive field is determined by the units in the input layer that could propagate a signal to it. Standard CNNs implement a slow propagation scheme, diffusing information across a single grid at rate proportional to convolutional filter size. This may be a reason extremely deep networks[34, 11, 22] appear necessary; many layers are needed to counteract inefficient signal propagation.
These points can be summarized as inherent deficiencies in representation, computation, and communication. Our multigrid architecture (Figure 1, bottom) endows CNNs with additional structural capacity in order to dissolve these deficiencies. It is explicitly multiscale, pushing choices about scale-space representation into the training process.
Computation occurs in parallel at all scales; every layer process both coarse and fine representations. Section 3.2 also explores coarse-to-fine variants that transition from processing on a coarse pyramid to processing on a full pyramid as the network deepens. Pyramids provide not only an efficient computational model, but a unified one. Viewing the network as evolving a representation living on the pyramid, we can combine previous task-specific architectures. For classification, attach an output to the coarsest pyramid level; for segmentation, attach an output to the finest.
Multigrid structure facilitates cross-scale information exchange, thereby destroying the long-established notion of receptive field. Most neurons have receptive field equivalent to the entire input; field size grows exponentially with depth, or, in progressive multigrid networks, begins with the full (coarse) input. Quick communication pathways exist throughout the network, and enable new capabilities.
We specifically demonstrate that multigrid CNNs, trained in a pure end-to-end fashion, can learn to attend and route information. Their emergent behavior may dynamically emulate the routing circuits articulated by Olshausen et al. . We construct a synthetic task that standard CNNs completely fail to learn, but multigrid CNNs accomplish with ease. Here, attentional capacity is key.
As Section 2 reviews, recent CNN architectural innovations ignore scale-space routing capacity, focusing instead on aspects like depth. Multigrid, as Section 3 details, complements such work. Section 4 measures performance improvements due to multigrid on classification tasks (CIFAR and ImageNet) and synthetic semantic segmentation tasks. Multigrid boosts both baseline and residual CNNs. On a synthetic spatial transformation task, multigrid is more than a boost; it is required, as residual networks alone do not possess attentional capacity. Section 5 discusses implications.
In wake of AlexNet 
, exploration of CNNs across computer vision has distilled some rules of thumb for their design. Small (e.g. ) spatial filters, in many successive layers, make for efficient parameter allocation [32, 34, 11]. Feature channels should increase with spatial resolution reduction [21, 32] (e.g. doubling as in Figure 1). Deeper networks are better, so long as a means of overcoming vanishing gradients is engineered into the training process [34, 16] or the network itself [33, 11, 22, 15]. Width matters .
The desire to adapt image classification CNNs to more complex output tasks, such as semantic segmentation, has catalyzed development of ad-hoc architectural additions for restoring spatial resolution. These include skip connections and upsampling [24, 3], hypercolumns 
, and, autoencoder-like, hourglass or U-shaped networks that reduce and re-expand spatial grids[1, 13, 27, 30].
This latter group of methods reflects the classic intuition of connecting bottom-up and top-down signals. Our work differs from these and earlier models  by virtual of decoupling pyramid level from feature abstraction. Representations at all scales evolve over the depth of our network. Such dynamics also separates us from past multiscale CNN work [8, 7, 35], which does not consider embedded and continual cross-scale communication.
While prior CNN designs have varied filter size (e.g. [21, 34]), our choice to instead vary grid resolution is crucial. Applying multiple filter sets, of different spatial size, to a single grid could emulate a multigrid computation. However, we use exponential (power of
) grid size scaling. Emulating this via larger filters quickly becomes cost prohibitive in terms of both parameters and computation; it is impractical to implement in the style of Inception modules. Dilated  and atrous convolution , while related, do not fully capture the pooling and interpolation aspects (detailed in Section 3) of our multigrid operator.
Recent efforts to improve the computational efficiency of CNNs, though too numerous to list in full, often attack via parameter [36, 4, 17] or precision [29, 5] reduction. Concurrent work extends such exploration to include cascades [14, 26], in which explicit routing decisions allow for partial evaluation of a network. Unlike this work, our cross-scale connections are bi-directional and we explore routing in a different sense: internal transport of information (e.g. attention ) for solving a particular task. We focus on a coarse-to-fine aspect of efficiency, borrowing from the use of multigrid concepts in image segmentation [31, 25].
Figure 1 conveys our intention to wire cross-scale connections into network structure at the lowest level. We can think of a multigrid CNN as a standard CNN in which every grid is transformed into a pyramid. Every convolutional filter extends spatially within grids , across grids of multiple scales within a pyramid, and over corresponding feature channels . A pyramidal slice, across scale-space, of the preceding layer contributes to the response at a particular corresponding neuron in the next.
This scheme enables any neuron to transmit a signal up the pyramid, from fine to coarse grids, and back down again. Even with signals jumping just one pyramid level per consecutive layer, the network can exploit this structure to quickly route information between two spatial locations on the finest grid. Communication time is logarithmic in spatial separation, rather than linear as in standard CNNs.
This difference in information routing capability is particularly dramatic in light of the recent trend towards stacking many layers of small convolutional filters [32, 11]. In standard CNNs, this virtually guarantees that either very deep networks, or manually-added pooling and unpooling stages [1, 13, 27, 30], will be needed to propagate information across the pixel grid. Multigrid allows for faster propagation with minimal additional design complexity. Moreover, unlike fixed pooling/unpooling stages, multigrid allows the network to learn data-dependent routing strategies.
Instead of directly implementing multigrid convolution as depicted in Figure 1
, we implement a close alternative that can be easily built out of standard components in existing deep learning frameworks. Figure2 illustrates the multigrid convolutional layer (mg-conv) we use as a drop-in replacement for standard convolutional (conv) layers when converting a network from grids to pyramids.
With multigrid convolution, we may choose to learn independent filter sets for each scale within a layer, or alternatively, tie parameters between scales and learn shared filters. We learn independent sets, assuming that doing so affords the network the chance to squeeze maximum performance out of scale-specific representations. Channel counts between grids need not match in any particular manner. Each of in Figure 2 is independent. For comparisons, we either set channel count on the finest grid to match baseline models, or calibrate channel counts so that total parameters are similar to the baselines. Section 4 reveals quite good performance is achievable by halving channel count with each coarser grid (e.g. , ). Thus, multigrid adds minimal computational overhead; the coarse grids are cheap.
. Convolution acts jointly across the multigrid pyramid, while batch normalization (BN)
and ReLU apply separately to each grid. Extensions are also possible with alternative residual units, but we leave such exploration to future work. Our focus is on comparison of baseline, residual, multigrid, and residual-multigrid networks.
Within an mg-conv layer, max-pooling and subsampling acts as a lateral communication mechanism from fine to coarse grids. Similarly, upsampling facilitates lateral communication from coarse to fine grids. Rather than locating these operations at a few fixed stages in the pipeline, we are actually inserting them everywhere. However, their action is now lateral, combining different-scale grids in the same layer, rather than between grids of different scales in two layers consecutive in depth.
While Figure 1 shows additional max-pooling and subsampling stages acting depth-wise on entire pyramids, we also consider pipelines that do not shrink pyramids. Rather, they simply attach an output to a particular grid (and possibly prune a few grids outside of its communication range). Success of this strategy motivates rethinking the role of pooling in CNNs. Instead of explicit summarization employed at select stages, multigrid yields a view of pooling as implicit communication that pervades the network.
The computation pattern underlying a multigrid CNN’s forward pass has analogy with the multiscale multigrid scheme of Maire and Yu . One can view their eigensolver as a linear diffusion process on a multiscale pyramid. We view a multigrid CNN as a nonlinear process, on a similar pyramid, with the same communication structure.
Extending the analogy, we port their progressive multigrid computation scheme to our setting. Rather than starting directly on the full pyramid, a progressive multigrid CNN can spend several layers processing only a coarse grid. Following this are additional layers processing a small pyramid (e.g. coarse and medium grids together), before the remainder of the network commits to work with the full pyramid.
In all multigrid and progressive multigrid experiments, we set the fine-scale input grid to be the original image and simply feed in downsampled (lower resolution) versions to independent initial convolution layers. The outputs of these initial layers then form a multigrid pyramid which is processed coherently by the rest of the network.
as our baseline. In abuse of notation, we reuse their VGG name. Our 16-layer version, VGG-16, consists of 5 sections of 3 convolutional layers each, with pooling and subsampling between. A final softmax layer produces class predictions (on CIFAR-100). Convolutional layers usefilters. We instantiate variants with different depth by changing the number layers per section (2 for VGG-11, 4 for VGG-21, etc.). Residual baselines (RES) follow the same layout, but use residual units within each section. Recall that each residual unit contains two convolutional layers.
Multigrid (MG) and residual multigrid (R-MG) networks, starting respectively from VGG or RES, simply flip from grids to pyramids and convolution to multigrid convolution. Progressive variants (PMG, R-PMG) expand the first section in order to gradually work up to computation on the complete pyramid. Even as their depth increases, a significant portion of layers in progressive networks avoid processing the full pyramid. As diagrammed, multigrid networks match their fine-grid feature count with baselines, and hence add some capacity and parameters. For classification, we also consider smaller (denoted -sm) multigrid variants with fewer channels per grid in order to be closer in parameter count with baseline VGG and RES networks.
For the semantic segmentation and spatial transformation tasks detailed in the next section, we use networks that produce per-pixel output. As a baseline, we employ the U-NET design of Ronneberger et al. . Again, convolutional filters are , except for the layers immediately following upsampling; here we use filters, following . We examine progressive multigrid alternatives (PMG, R-PMG) that continue to operate on the multiscale pyramid. Unlike the classification setting, we do not reduce pyramid resolution. These networks drop some coarse grids towards the end of their pipelines for the sole reason that such grids do not communicate with the output.
|Method||Params ()||FLOPs ()||Error (%)|
Our experiments focus on systematic exploration and evaluation of the architectural design space, with the goal of quantifying the relative benefits of multigrid and its synergistic combination with residual connections.
We evaluate the array of network architectures listed in Table 1 on the task of CIFAR-100 image classification.
Data. We whiten and then enlarge images to . We use random patches from the training set (with random horizontal flipping) and the center patch from test examples. There are 50K training and 10K test images.
We use SGD with batch size 128, weight decay rate 0.0005, and 300 iterations per epoch for 200 epochs. For VGG, MG, and PMG, learning rate exponentially decays from 0.1 to 0.0001; whereas for RES, R-MG, and R-PMG, it decays at rate 0.2 every 60 epochs.
Results. From Table 1, we see that adding multigrid capacity consistently improves over both basic (compare green entries), as well as residual CNNs (blue entries). Progressive multigrid nets of similar depth achieve better or comparable accuracy at reduced expense, as quantified in terms of both parameters and floating point operations (red entries).
We generate synthetic data (10K training, 1K test images) for semantic segmentation by randomly scaling, rotating, translating, and pasting MNIST  digits onto a canvas, limiting digit overlap to . The task is to decompose the result into per-pixel digit class labels.
In addition to networks already mentioned, we consider a single grid (SG) baseline that mirrors PMG but removes all grids but the finest. We also consider U-MG, which changes resolution like U-NET, but works on pyramids.
|Method||Params ()||mean IoU (%)||mean Error (%)|
|Method||Params ()||mean IoU (%)||mean Error (%)|
|Params||FLOPs||val, 10-crop||val, single-crop|
|ResNet-34 C ||21.8||3.66||24.19||7.40||-||-|
|WRN-34 () ||48.6||14.09||-||-||24.50||7.58|
Training. We use batch size 64, weight decay rate 0.0005, and 150 iterations per epoch for 200 epochs. We set learning rate schedules for non-residual and residual networks in the same manner as done for CIFAR-100.
Results. Table 2 and Figure 4 show dramatic performance advantages of the progressive multigrid network. We report scores of mean intersection over union (IoU) with the true output regions, as well as mean per-pixel multiclass labeling accuracy. These results are a strong indicator that continuous operation on a multigrid pyramid should replace networks that pass through a low resolution bottleneck.
Jaderberg et al. 
engineer a system for inverting spatial distortions by combining a neural network for estimating the parameters of the transformation with a dedicated unit for applying the inverse. Such a split of the problem into two components appears necessary when using standard CNNs; they cannot learn the task end-to-end. But, progressive multigrid networks are able to learn such tasks.
We generate a synthetic spatial transformation dataset (60K training, 10K test images) in the same manner as the one for semantic segmentation, but with only one digit per image and an additional affine transformation with a uniformly random sheering angle in . Training is the same as segmentation, except 800 iterations per epoch.
Results. Table 3 and Figure 5 show that all networks except PMG and R-PMG fail to learn the task. Figure 5 (right side) reveals the reason: U-NET (and others not shown) cannot learn translation. This makes sense for single grid methods, as propagating information across the fine grid would require them to be deeper than tested. U-NET’s failure seems to stem from confusion due to pooling/upsampling. It appears to paste subregions of the target digit into output, but not in the correct arrangement. Figure 6 reveals that, unlike U-NET, PMG and R-PMG exhibit attentional behavior.
We select two designs to train from scratch on ImageNet:
R-PMG-30-SEG: Similar to our semantic segmentation network, but adapted to ImageNet, it maps input onto 4 grids: , , ,
, that are maintained throughout the network without pooling stages. We increase feature count with depth and attach a classifier to the final coarsest grid.
Training uses SGD with batch size 256, weight decay 0.0001, 10K iterations per epoch for 100 epochs, starting learning rate 0.1 with decay rate 0.1 every 30 epochs. Table 4 compares our performance to that of ResNets and wide residual networks (WRN) . We observe:
Multigrid substantially improves performance. R-MG-34 even outperforms the deeper ResNet-50.
Multigrid is a more efficient use of parameters than simply increasing feature channel count on a single grid. Compare R-MG-34 with the double-wide residual network WRN-34: we match its performance with fewer parameters and require drastically fewer FLOPs.
R-PMG-30-SEG outperforms ResNet-34. The same design we used for segmentation is great for classification, lending support to the idea of a single unifying architecture which evolves a multigrid representation.
Our proposed multigrid extension to CNNs yields improved accuracy on classification and semantic segmentation tasks. Progressive multigrid variants open a new pathway towards optimizing CNNs for efficiency. Multigrid appears unique in extending the range of tasks CNNs can accomplish, by integrating into the network structure the capacity to learn routing and attentional mechanisms. These new abilities suggest that multigrid could replace some ad-hoc designs in the current zoo of CNN architectures.
On a speculative note, multigrid neural networks might also have broader implications in neuroscience. Feedforward computation on a sequence of multigrid pyramids looks similar to combined bottom-up/top-down processing across a single larger structure if neurons are embedded in some computational substrate according to their spatial grid, rather than their depth in the processing chain.
Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position.Biological Cybernetics, 1980.