A Transformer Architecture for Online Gesture Recognition of Mathematical Expressions

11/04/2022
by   Mirco Ramo, et al.
0

The Transformer architecture is shown to provide a powerful framework as an end-to-end model for building expression trees from online handwritten gestures corresponding to glyph strokes. In particular, the attention mechanism was successfully used to encode, learn and enforce the underlying syntax of expressions creating latent representations that are correctly decoded to the exact mathematical expression tree, providing robustness to ablated inputs and unseen glyphs. For the first time, the encoder is fed with spatio-temporal data tokens potentially forming an infinitely large vocabulary, which finds applications beyond that of online gesture recognition. A new supervised dataset of online handwriting gestures is provided for training models on generic handwriting recognition tasks and a new metric is proposed for the evaluation of the syntactic correctness of the output expression trees. A small Transformer model suitable for edge inference was successfully trained to an average normalised Levenshtein accuracy of 94 tree representation for 94

READ FULL TEXT

page 8

page 9

research
05/05/2023

Online Gesture Recognition using Transformer and Natural Language Processing

The Transformer architecture is shown to provide a powerful machine tran...
research
03/13/2023

Spatial Attention and Syntax Rule Enhanced Tree Decoder for Offine Handwritten Mathematical Expression Recognition

Offline Handwritten Mathematical Expression Recognition (HMER) has been ...
research
12/10/2021

Short and Long Range Relation Based Spatio-Temporal Transformer for Micro-Expression Recognition

Being spontaneous, micro-expressions are useful in the inference of a pe...
research
03/28/2022

TraHGR: Few-shot Learning for Hand Gesture Recognition via ElectroMyography

Deep learning-based Hand Gesture Recognition (HGR) via surface Electromy...
research
01/25/2022

ViT-HGR: Vision Transformer-based Hand Gesture Recognition from High Density Surface EMG Signals

Recently, there has been a surge of significant interest on application ...
research
08/11/2021

A Transformer-based Math Language Model for Handwritten Math Expression Recognition

Handwritten mathematical expressions (HMEs) contain ambiguities in their...
research
07/10/2022

CoMER: Modeling Coverage for Transformer-based Handwritten Mathematical Expression Recognition

The Transformer-based encoder-decoder architecture has recently made sig...

Please sign up or login with your details

Forgot password? Click here to reset