Learning Architectures from an Extended Search Space for Language Modeling

by   Yinqiao Li, et al.

Neural architecture search (NAS) has advanced significantly in recent years but most NAS systems restrict search to learning architectures of a recurrent or convolutional cell. In this paper, we extend the search space of NAS. In particular, we present a general approach to learn both intra-cell and inter-cell architectures (call it ESS). For a better search result, we design a joint learning method to perform intra-cell and inter-cell NAS simultaneously. We implement our model in a differentiable architecture search system. For recurrent neural language modeling, it outperforms a strong baseline significantly on the PTB and WikiText data, with a new state-of-the-art on PTB. Moreover, the learned architectures show good transferability to other systems. E.g., they improve state-of-the-art systems on the CoNLL and WNUT named entity recognition (NER) tasks and CoNLL chunking task, indicating a promising line of research on large-scale pre-learned architectures.


page 1

page 2

page 3

page 4


Neural Architecture Search of SPD Manifold Networks

In this paper, we propose a new neural architecture search (NAS) problem...

Understanding Architectures Learnt by Cell-based Neural Architecture Search

Neural architecture search (NAS) generates architectures automatically f...

WeNet: Weighted Networks for Recurrent Network Architecture Search

In recent years, there has been increasing demand for automatic architec...

RankNAS: Efficient Neural Architecture Search by Pairwise Ranking

This paper addresses the efficiency challenge of Neural Architecture Sea...

Learned Transferable Architectures Can Surpass Hand-Designed Architectures for Large Scale Speech Recognition

In this paper, we explore the neural architecture search (NAS) for autom...

Incremental Learning with Differentiable Architecture and Forgetting Search

As progress is made on training machine learning models on incrementally...

Auto-Panoptic: Cooperative Multi-Component Architecture Search for Panoptic Segmentation

Panoptic segmentation is posed as a new popular test-bed for the state-o...

Please sign up or login with your details

Forgot password? Click here to reset