Stroke Constrained Attention Network for Online Handwritten Mathematical Expression Recognition

02/20/2020
by   Jiaming Wang, et al.
0

In this paper, we propose a novel stroke constrained attention network (SCAN) which treats stroke as the basic unit for encoder-decoder based online handwritten mathematical expression recognition (HMER). Unlike previous methods which use trace points or image pixels as basic units, SCAN makes full use of stroke-level information for better alignment and representation. The proposed SCAN can be adopted in both single-modal (online or offline) and multi-modal HMER. For single-modal HMER, SCAN first employs a CNN-GRU encoder to extract point-level features from input traces in online mode and employs a CNN encoder to extract pixel-level features from input images in offline mode, then use stroke constrained information to convert them into online and offline stroke-level features. Using stroke-level features can explicitly group points or pixels belonging to the same stroke, therefore reduces the difficulty of symbol segmentation and recognition via the decoder with attention mechanism. For multi-modal HMER, other than fusing multi-modal information in decoder, SCAN can also fuse multi-modal information in encoder by utilizing the stroke based alignments between online and offline modalities. The encoder fusion is a better way for combining multi-modal information as it implements the information interaction one step before the decoder fusion so that the advantages of multiple modalities can be exploited earlier and more adequately when training the encoder-decoder model. Evaluated on a benchmark published by CROHME competition, the proposed SCAN achieves the state-of-the-art performance.

READ FULL TEXT

page 22

page 23

page 24

research
12/04/2017

A GRU-based Encoder-Decoder Approach with Attention for Online Handwritten Mathematical Expression Recognition

In this study, we present a novel end-to-end approach based on the encod...
research
11/21/2021

MaIL: A Unified Mask-Image-Language Trimodal Network for Referring Image Segmentation

Referring image segmentation is a typical multi-modal task, which aims a...
research
08/20/2022

Offline Handwritten Mathematical Recognition using Adversarial Learning and Transformers

Offline Handwritten Mathematical Expression Recognition (HMER) is a majo...
research
03/08/2021

Self-Augmented Multi-Modal Feature Embedding

Oftentimes, patterns can be represented through different modalities. Fo...
research
05/05/2020

Multi-interactive Encoder-decoder Network for RGBT Salient Object Detection

RGBT salient object detection (SOD) aims to segment the common prominent...
research
07/06/2020

EDSL: An Encoder-Decoder Architecture with Symbol-Level Features for Printed Mathematical Expression Recognition

Printed Mathematical expression recognition (PMER) aims to transcribe a ...
research
08/17/2022

ParaColorizer: Realistic Image Colorization using Parallel Generative Networks

Grayscale image colorization is a fascinating application of AI for info...

Please sign up or login with your details

Forgot password? Click here to reset