Evaluating the Effectiveness of Efficient Neural Architecture Search for Sentence-Pair Tasks

10/08/2020
by   Ansel MacLaughlin, et al.
0

Neural Architecture Search (NAS) methods, which automatically learn entire neural model or individual neural cell architectures, have recently achieved competitive or state-of-the-art (SOTA) performance on variety of natural language processing and computer vision tasks, including language modeling, natural language inference, and image classification. In this work, we explore the applicability of a SOTA NAS algorithm, Efficient Neural Architecture Search (ENAS) (Pham et al., 2018) to two sentence pair tasks, paraphrase detection and semantic textual similarity. We use ENAS to perform a micro-level search and learn a task-optimized RNN cell architecture as a drop-in replacement for an LSTM. We explore the effectiveness of ENAS through experiments on three datasets (MRPC, SICK, STS-B), with two different models (ESIM, BiLSTM-Max), and two sets of embeddings (Glove, BERT). In contrast to prior work applying ENAS to NLP tasks, our results are mixed – we find that ENAS architectures sometimes, but not always, outperform LSTMs and perform similarly to random architecture search.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

06/12/2020

NAS-Bench-NLP: Neural Architecture Search Benchmark for Natural Language Processing

Neural Architecture Search (NAS) is a promising and rapidly evolving res...
02/21/2019

Overcoming Multi-Model Forgetting

We identify a phenomenon, which we refer to as multi-model forgetting, t...
01/04/2021

Tensorizing Subgraph Search in the Supernet

Recently, a special kind of graph, i.e., supernet, which allows two node...
06/12/2019

Continual and Multi-Task Architecture Search

Architecture search is the process of automatically learning the neural ...
08/15/2020

Finding Fast Transformers: One-Shot Neural Architecture Search by Component Composition

Transformer-based models have achieved stateof-the-art results in many t...
02/25/2022

Accelerating Neural Architecture Exploration Across Modalities Using Genetic Algorithms

Neural architecture search (NAS), the study of automating the discovery ...
07/17/2018

Power Networks: A Novel Neural Architecture to Predict Power Relations

Can language analysis reveal the underlying social power relations that ...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.