DeepAI AI Chat
Log In Sign Up

A Statistical Model of Word Rank Evolution

by   Alex John Quijano, et al.
University of California, Merced

The availability of large linguistic data sets enables data-driven approaches to study linguistic change. This work explores the word rank dynamics of eight languages by investigating the Google Books corpus unigram frequency data set. We observed the rank changes of the unigrams from 1900 to 2008 and compared it to a Wright-Fisher inspired model that we developed for our analysis. The model simulates a neutral evolutionary process with the restriction of having no disappearing words. This work explains the mathematical framework of the model - written as a Markov Chain with multinomial transition probabilities - to show how frequencies of words change in time. From our observations in the data and our model, word rank stability shows two types of characteristics: (1) the increase/decrease in ranks are monotonic, or (2) the average rank stays the same. Based on our model, high-ranked words tend to be more stable while low-ranked words tend to be more volatile. Some words change in ranks in two ways: (a) by an accumulation of small increasing/decreasing rank changes in time and (b) by shocks of increase/decrease in ranks. Most of the stopwords and Swadesh words are observed to be stable in ranks across eight languages. These signatures suggest unigram frequencies in all languages have changed in a manner inconsistent with a purely neutral evolutionary process.


page 31

page 39


Statistical analysis of word flow among five Indo-European languages

A recent increase in data availability has allowed the possibility to pe...

The Natural Selection of Words: Finding the Features of Fitness

We introduce a dataset for studying the evolution of words, constructed ...

Rank-frequency distribution of natural languages: a difference of probabilities approach

The time variation of the rank k of words for six Indo-European language...

Slangvolution: A Causal Analysis of Semantic Change and Frequency Dynamics in Slang

Languages are continuously undergoing changes, and the mechanisms that u...

Quantifying Cognitive Factors in Lexical Decline

We adopt an evolutionary view on language change in which cognitive fact...

A joint text mining-rank size investigation of the rhetoric structures of the US Presidents' speeches

This work presents a text mining context and its use for a deep analysis...

Self-contained Beta-with-Spikes Approximation for Inference Under a Wright-Fisher Model

We construct a reliable estimation of evolutionary parameters within the...