EDD: Efficient Differentiable DNN Architecture and Implementation Co-search for Embedded AI Solutions

05/06/2020
by   Yuhong Li, et al.
0

High quality AI solutions require joint optimization of AI algorithms and their hardware implementations. In this work, we are the first to propose a fully simultaneous, efficient differentiable DNN architecture and implementation co-search (EDD) methodology. We formulate the co-search problem by fusing DNN search variables and hardware implementation variables into one solution space, and maximize both algorithm accuracy and hardware implementation quality. The formulation is differentiable with respect to the fused variables, so that gradient descent algorithm can be applied to greatly reduce the search time. The formulation is also applicable for various devices with different objectives. In the experiments, we demonstrate the effectiveness of our EDD methodology by searching for three representative DNNs, targeting low-latency GPU implementation and FPGA implementations with both recursive and pipelined architectures. Each model produced by EDD achieves similar accuracy as the best existing DNN models searched by neural architecture search (NAS) methods on ImageNet, but with superior performance obtained within 12 GPU-hour searches. Our DNN targeting GPU is 1.40x faster than the state-of-the-art solution reported in Proxyless, and our DNN targeting FPGA delivers 1.45x higher throughput than the state-of-the-art solution reported in DNNBuilder.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/13/2023

Flexible Channel Dimensions for Differentiable Architecture Search

Finding optimal channel dimensions (i.e., the number of filters in DNN l...
research
04/26/2021

HAO: Hardware-aware neural Architecture Optimization for Efficient Inference

Automatic algorithm-hardware co-design for DNN has shown great success i...
research
11/18/2019

NAIS: Neural Architecture and Implementation Search and its Applications in Autonomous Driving

The rapidly growing demands for powerful AI algorithms in many applicati...
research
10/14/2020

Effective Algorithm-Accelerator Co-design for AI Solutions on Edge Devices

High quality AI solutions require joint optimization of AI algorithms, s...
research
06/28/2021

HALF: Holistic Auto Machine Learning for FPGAs

Deep Neural Networks (DNNs) are capable of solving complex problems in d...
research
12/04/2021

Logic Shrinkage: Learned FPGA Netlist Sparsity for Efficient Neural Network Inference

FPGA-specific DNN architectures using the native LUTs as independently t...
research
07/17/2020

Standing on the Shoulders of Giants: Hardware and Neural Architecture Co-Search with Hot Start

Hardware and neural architecture co-search that automatically generates ...

Please sign up or login with your details

Forgot password? Click here to reset