research
∙
04/21/2023
Tokenization Tractability for Human and Machine Learning Model: An Annotation Study
Is tractable tokenization for humans also tractable for machine learning...
share
research
∙
04/21/2023
Downstream Task-Oriented Neural Tokenizer Optimization with Vocabulary Restriction as Post Processing
This paper proposes a method to optimize tokenization for the performanc...
research
∙
09/09/2022
MaxMatch-Dropout: Subword Regularization for WordPiece
We present a subword regularization method for WordPiece, which uses a m...
research
∙
03/25/2022
Single Model Ensemble for Subword Regularized Models in Low-Resource Machine Translation
Subword regularizations use multiple subword segmentations during traini...
research
∙
05/26/2021
Joint Optimization of Tokenization and Downstream Model
Since traditional tokenizers are isolated from a downstream task and mod...
research
∙
10/15/2020