Explaining Character-Aware Neural Networks for Word-Level Prediction: Do They Discover Linguistic Rules?

by   Fréderic Godin, et al.

Character-level features are currently used in different neural network-based natural language processing algorithms. However, little is known about the character-level patterns those models learn. Moreover, models are often compared only quantitatively while a qualitative analysis is missing. In this paper, we investigate which character-level patterns neural networks learn and if those patterns coincide with manually-defined word segmentations and annotations. To that end, we extend the contextual decomposition technique (Murdoch et al. 2018) to convolutional neural networks which allows us to compare convolutional neural networks and bidirectional long short-term memory networks. We evaluate and compare these models for the task of morphological tagging on three morphologically different languages and show that these models implicitly discover understandable linguistic rules. Our implementation can be found at https://github.com/FredericGodin/ContextualDecomposition-NLP .



There are no comments yet.


page 1

page 2

page 3

page 4


Dual Long Short-Term Memory Networks for Sub-Character Representation Learning

Characters have commonly been regarded as the minimal processing unit in...

Evaluating neural network explanation methods using hybrid documents and morphological prediction

We propose two novel paradigms for evaluating neural network explanation...

TypeShift: A User Interface for Visualizing the Typing Production Process

TypeShift is a tool for visualizing linguistic patterns in the timing of...

Character Transformations for Non-Autoregressive GEC Tagging

We propose a character-based nonautoregressive GEC approach, with automa...

Character-based Neural Embeddings for Tweet Clustering

In this paper we show how the performance of tweet clustering can be imp...

Relational Weight Priors in Neural Networks for Abstract Pattern Learning and Language Modelling

Deep neural networks have become the dominant approach in natural langua...

Dilated Convolutional Neural Networks for Lightweight Diacritics Restoration

Diacritics restoration has become a ubiquitous task in the Latin-alphabe...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.