A Statistical Model of Word Rank Evolution

07/21/2021
by   Alex John Quijano, et al.
0

The availability of large linguistic data sets enables data-driven approaches to study linguistic change. This work explores the word rank dynamics of eight languages by investigating the Google Books corpus unigram frequency data set. We observed the rank changes of the unigrams from 1900 to 2008 and compared it to a Wright-Fisher inspired model that we developed for our analysis. The model simulates a neutral evolutionary process with the restriction of having no disappearing words. This work explains the mathematical framework of the model - written as a Markov Chain with multinomial transition probabilities - to show how frequencies of words change in time. From our observations in the data and our model, word rank stability shows two types of characteristics: (1) the increase/decrease in ranks are monotonic, or (2) the average rank stays the same. Based on our model, high-ranked words tend to be more stable while low-ranked words tend to be more volatile. Some words change in ranks in two ways: (a) by an accumulation of small increasing/decreasing rank changes in time and (b) by shocks of increase/decrease in ranks. Most of the stopwords and Swadesh words are observed to be stable in ranks across eight languages. These signatures suggest unigram frequencies in all languages have changed in a manner inconsistent with a purely neutral evolutionary process.

READ FULL TEXT

page 31

page 39

research
01/17/2023

Statistical analysis of word flow among five Indo-European languages

A recent increase in data availability has allowed the possibility to pe...
research
08/19/2019

The Natural Selection of Words: Finding the Features of Fitness

We introduce a dataset for studying the evolution of words, constructed ...
research
11/23/2018

Rank-frequency distribution of natural languages: a difference of probabilities approach

The time variation of the rank k of words for six Indo-European language...
research
03/09/2022

Slangvolution: A Causal Analysis of Semantic Change and Frequency Dynamics in Slang

Languages are continuously undergoing changes, and the mechanisms that u...
research
10/12/2021

Quantifying Cognitive Factors in Lexical Decline

We adopt an evolutionary view on language change in which cognitive fact...
research
05/09/2019

A joint text mining-rank size investigation of the rhetoric structures of the US Presidents' speeches

This work presents a text mining context and its use for a deep analysis...
research
03/08/2023

Self-contained Beta-with-Spikes Approximation for Inference Under a Wright-Fisher Model

We construct a reliable estimation of evolutionary parameters within the...

Please sign up or login with your details

Forgot password? Click here to reset