Neuromodulation Gated Transformer

05/05/2023

∙

We introduce a novel architecture, the Neuromodulation Gated Transformer (NGT), which is a simple implementation of neuromodulation in transformers via a multiplicative effect. We compare it to baselines and show that it results in the best average performance on the SuperGLUE benchmark validation sets.

READ FULL TEXT

Neuromodulation Gated Transformer

Convolutional Gated MLP: Combining Convolutions gMLP

Supplementary Material: Implementation and Experiments for GAU-based Model

Classification Models for Partially Ordered Sequences

An Introduction to Transformers

Building High-accuracy Multilingual ASR with Gated Language Experts and Curriculum Training

Geometric Transformer for End-to-End Molecule Properties Prediction

The Naughtyformer: A Transformer Understands Offensive Humor

Neuromodulation Gated Transformer

Related Research

Convolutional Gated MLP: Combining Convolutions gMLP

Supplementary Material: Implementation and Experiments for GAU-based Model

Classification Models for Partially Ordered Sequences

An Introduction to Transformers

Building High-accuracy Multilingual ASR with Gated Language Experts and Curriculum Training

Geometric Transformer for End-to-End Molecule Properties Prediction

The Naughtyformer: A Transformer Understands Offensive Humor