AI Chat AI Image Generator AI Video Text to Speech

Circling Back to Recurrent Models of Language

11/03/2022

∙

by Gábor Melis, et al.

∙

∙

Just because some purely recurrent models suffer from being hard to optimize and inefficient on today's hardware, they are not necessarily bad models of language. We demonstrate this by the extent to which these models can still be improved by a combination of a slightly better recurrent cell, architecture, objective, as well as optimization. In the process, we establish a new state of the art for language modelling on small datasets and on enwik8 with dynamic evaluation.

page 1

page 2

page 3

page 4

research

∙ 09/04/2019

Mogrifier LSTM

Many advances in Natural Language Processing have been based upon more e...

0 Gábor Melis, et al. ∙

research

∙ 05/25/2019

Dynamic Cell Structure via Recursive-Recurrent Neural Networks

In a recurrent setting, conventional approaches to neural architecture s...

4 Xin Qian, et al. ∙

research

∙ 07/18/2017

On the State of the Art of Evaluation in Neural Language Models

Ongoing innovations in recurrent neural network architectures have provi...

0 Gábor Melis, et al. ∙

research

∙ 11/05/2016

Neural Architecture Search with Reinforcement Learning

Neural networks are powerful and flexible models that work well for many...

0 Barret Zoph, et al. ∙

research

∙ 02/05/2019

The Referential Reader: A Recurrent Entity Network for Anaphora Resolution

We present a new architecture for storing and accessing entity mentions ...

0 Fei Liu, et al. ∙

research

∙ 12/10/2020

Recurrent Point Review Models

Deep neural network models represent the state-of-the-art methodologies ...

0 Kostadin Cvejoski, et al. ∙

research

∙ 11/26/2020

A Recurrent Vision-and-Language BERT for Navigation

Accuracy of many visiolinguistic tasks has benefited significantly from ...

0 Yicong Hong, et al. ∙