Neuromodulation Gated Transformer

05/05/2023

∙

by   Kobe Knowles, et al.

∙

0

∙

share

We introduce a novel architecture, the Neuromodulation Gated Transformer (NGT), which is a simple implementation of neuromodulation in transformers via a multiplicative effect. We compare it to baselines and show that it results in the best average performance on the SuperGLUE benchmark validation sets.