Algorithm-hardware Co-design for Deformable Convolution

02/19/2020
by   Qijing Huang, et al.
9

FPGAs provide a flexible and efficient platform to accelerate rapidly-changing algorithms for computer vision. The majority of existing work focuses on accelerating image classification, while other fundamental vision problems, including object detection and instance segmentation, have not been adequately addressed. Compared with image classification, detection problems are more sensitive to the spatial variance of objects, and therefore, require specialized convolutions to aggregate spatial information. To address this, recent work proposes dynamic deformable convolution to augment regular convolutions. Regular convolutions process a fixed grid of pixels across all the spatial locations in an image, while dynamic deformable convolutions may access arbitrary pixels in the image and the access pattern is input-dependent and varies per spatial location. These properties lead to inefficient memory accesses of inputs with existing hardware. In this work, we first investigate the overhead of the deformable convolution on embedded FPGA SoCs, and then show the accuracy-latency tradeoffs for a set of algorithm modifications including full versus depthwise, fixed-shape, and limited-range. These modifications benefit the energy efficiency for embedded devices in general as they reduce the compute complexity. We then build an efficient object detection network with modified deformable convolutions and quantize the network using state-of-the-art quantization methods. We implement a unified hardware engine on FPGA to support all the operations in the network. Preliminary experiments show that little accuracy is compromised and speedup can be achieved with our co-design optimization for the deformable convolution.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/12/2020

CoDeNet: Algorithm-hardware Co-design for Deformable Convolution

Deploying deep learning models on embedded systems for computer vision t...
research
04/22/2022

DFAM-DETR: Deformable feature based attention mechanism DETR on slender object detection

Object detection is one of the most significant aspects of computer visi...
research
09/26/2022

Going Further With Winograd Convolutions: Tap-Wise Quantization for Efficient Inference on 4x4 Tile

Most of today's computer vision pipelines are built around deep neural n...
research
06/29/2017

Tensor-based approach to accelerate deformable part models

This article provides next step towards solving speed bottleneck of any ...
research
06/09/2020

An Efficient Accelerator Design Methodology for Deformable Convolutional Networks

Deformable convolutional networks have demonstrated outstanding performa...
research
11/27/2018

Deformable ConvNets v2: More Deformable, Better Results

The superior performance of Deformable Convolutional Networks arises fro...
research
07/06/2021

Energy-Efficient Accelerator Design for Deformable Convolution Networks

Deformable convolution networks (DCNs) proposed to address the image rec...

Please sign up or login with your details

Forgot password? Click here to reset