Integrating large language models and active inference to understand eye movements in reading and dyslexia

by   Francesco Donnarumma, et al.

We present a novel computational model employing hierarchical active inference to simulate reading and eye movements. The model characterizes linguistic processing as inference over a hierarchical generative model, facilitating predictions and inferences at various levels of granularity, from syllables to sentences. Our approach combines the strengths of large language models for realistic textual predictions and active inference for guiding eye movements to informative textual information, enabling the testing of predictions. The model exhibits proficiency in reading both known and unknown words and sentences, adhering to the distinction between lexical and nonlexical routes in dual-route theories of reading. Notably, our model permits the exploration of maladaptive inference effects on eye movements during reading, such as in dyslexia. To simulate this condition, we attenuate the contribution of priors during the reading process, leading to incorrect inferences and a more fragmented reading style, characterized by a greater number of shorter saccades. This alignment with empirical findings regarding eye movements in dyslexic individuals highlights the model's potential to aid in understanding the cognitive processes underlying reading and eye movements, as well as how reading deficits associated with dyslexia may emerge from maladaptive predictive processing. In summary, our model represents a significant advancement in comprehending the intricate cognitive processes involved in reading and eye movements, with potential implications for understanding and addressing dyslexia through the simulation of maladaptive inference. It may offer valuable insights into this condition and contribute to the development of more effective interventions for treatment.


Eyettention: An Attention-based Dual-Sequence Model for Predicting Human Scanpaths during Reading

Eye movements during reading offer insights into both the reader's cogni...

SEAM: An Integrated Activation-Coupled Model of Sentence Processing and Eye Movements in Reading

Models of eye-movement control during reading, developed largely within ...

Assessing Language Proficiency from Eye Movements in Reading

We present a novel approach for determining learners' second language pr...

Understanding Humans' Strategies in Maze Solving

Navigating through a visual maze relies on the strategic use of eye move...

An active efficient coding model of the optokinetic nystagmus

Optokinetic nystagmus (OKN) is an involuntary eye movement responsible f...

DysLexML: Screening Tool for Dyslexia Using Machine Learning

Eye movements during text reading can provide insights about reading dis...

Analyzing Wrap-Up Effects through an Information-Theoretic Lens

Numerous analyses of reading time (RT) data have been implemented – all ...

Please sign up or login with your details

Forgot password? Click here to reset