DeepAI AI Chat
Log In Sign Up

An Augmented Transformer Architecture for Natural Language Generation Tasks

by   Hailiang Li, et al.

The Transformer based neural networks have been showing significant advantages on most evaluations of various natural language processing and other sequence-to-sequence tasks due to its inherent architecture based superiorities. Although the main architecture of the Transformer has been continuously being explored, little attention was paid to the positional encoding module. In this paper, we enhance the sinusoidal positional encoding algorithm by maximizing the variances between encoded consecutive positions to obtain additional promotion. Furthermore, we propose an augmented Transformer architecture encoded with additional linguistic knowledge, such as the Part-of-Speech (POS) tagging, to boost the performance on some natural language generation tasks, e.g., the automatic translation and summarization tasks. Experiments show that the proposed architecture attains constantly superior results compared to the vanilla Transformer.


page 1

page 6


Injecting Hierarchy with U-Net Transformers

The Transformer architecture has become increasingly popular over the pa...

Multi-level Head-wise Match and Aggregation in Transformer for Textual Sequence Matching

Transformer has been successfully applied to many natural language proce...

A Comparative Study on Transformer vs RNN in Speech Applications

Sequence-to-sequence models have been widely used in end-to-end speech p...

Transformer based Grapheme-to-Phoneme Conversion

Attention mechanism is one of the most successful techniques in deep lea...

A Generalization of Transformer Networks to Graphs

We propose a generalization of transformer neural network architecture f...

Do Transformer Modifications Transfer Across Implementations and Applications?

The research community has proposed copious modifications to the Transfo...

How Crucial is Transformer in Decision Transformer?

Decision Transformer (DT) is a recently proposed architecture for Reinfo...