Learning Architectures from an Extended Search Space for Language Modeling

05/06/2020
by   Yinqiao Li, et al.
0

Neural architecture search (NAS) has advanced significantly in recent years but most NAS systems restrict search to learning architectures of a recurrent or convolutional cell. In this paper, we extend the search space of NAS. In particular, we present a general approach to learn both intra-cell and inter-cell architectures (call it ESS). For a better search result, we design a joint learning method to perform intra-cell and inter-cell NAS simultaneously. We implement our model in a differentiable architecture search system. For recurrent neural language modeling, it outperforms a strong baseline significantly on the PTB and WikiText data, with a new state-of-the-art on PTB. Moreover, the learned architectures show good transferability to other systems. E.g., they improve state-of-the-art systems on the CoNLL and WNUT named entity recognition (NER) tasks and CoNLL chunking task, indicating a promising line of research on large-scale pre-learned architectures.

READ FULL TEXT

page 1

page 2

page 3

page 4

10/27/2020

Neural Architecture Search of SPD Manifold Networks

In this paper, we propose a new neural architecture search (NAS) problem...
09/20/2019

Understanding Architectures Learnt by Cell-based Neural Architecture Search

Neural architecture search (NAS) generates architectures automatically f...
04/08/2019

WeNet: Weighted Networks for Recurrent Network Architecture Search

In recent years, there has been increasing demand for automatic architec...
09/15/2021

RankNAS: Efficient Neural Architecture Search by Pairwise Ranking

This paper addresses the efficiency challenge of Neural Architecture Sea...
04/07/2021

Darts-Conformer: Towards Efficient Gradient-Based Neural Architecture Search For End-to-End ASR

Neural architecture search (NAS) has been successfully applied to tasks ...
05/25/2019

Dynamic Cell Structure via Recursive-Recurrent Neural Networks

In a recurrent setting, conventional approaches to neural architecture s...
04/20/2022

SpiderNet: Hybrid Differentiable-Evolutionary Architecture Search via Train-Free Metrics

Neural Architecture Search (NAS) algorithms are intended to remove the b...