Towards naturalistic human neuroscience and neuroengineering: behavior mining in long-term video and neural recordings

by   Satpreet H. Singh, et al.

Recent advances in brain recording technology and artificial intelligence are propelling a new paradigm in neuroscience beyond the traditional controlled experiment. Naturalistic neuroscience studies neural computations associated with spontaneous behaviors performed in unconstrained settings. Analyzing such unstructured data lacking a priori experimental design remains a significant challenge, especially when the data is multi-modal and long-term. Here we describe an automated approach for analyzing large (≈250 GB/subject) datasets of simultaneously recorded human electrocorticography (ECoG) and naturalistic behavior video data for 12 subjects. Our pipeline discovers and annotates thousands of instances of human upper-limb movement events in long-term (7–9 day) naturalistic behavior data using a combination of computer vision, discrete latent-variable modeling, and string pattern-matching. Analysis of the simultaneously recorded brain data uncovers neural signatures of movement that corroborate prior findings from traditional controlled experiments. We also prototype a decoder for a movement initiation detection task to demonstrate the efficacy of our pipeline as a source of training data for brain-computer interfacing applications. We plan to publish our curated dataset, which captures naturalistic neural and behavioral variability at a scale not previously available. We believe this data will enable further research on models of neural function and decoding that incorporate such naturalistic variability and perform more robustly in real-world settings.



page 2

page 3

page 5


AJILE Movement Prediction: Multimodal Deep Learning for Natural Human Neural Recordings and Video

Developing useful interfaces between brains and machines is a grand chal...

Target-Specific Action Classification for Automated Assessment of Human Motor Behavior from Video

Objective monitoring and assessment of human motor behavior can improve ...

A Study on the Extraction and Analysis of a Large Set of Eye Movement Features during Reading

This work presents a study on the extraction and analysis of a set of 10...

Neurorobots as a Means Toward Neuroethology and Explainable AI

Understanding why deep neural networks and machine learning algorithms a...

The Algonauts Project 2021 Challenge: How the Human Brain Makes Sense of a World in Motion

The sciences of natural and artificial intelligence are fundamentally co...

Open-Source Tools for Behavioral Video Analysis: Setup, Methods, and Development

Recently developed methods for video analysis, especially models for pos...

Overcoming the Domain Gap in Neural Action Representations

Relating animal behaviors to brain activity is a fundamental goal in neu...

Code Repositories


Code to accompany:

view repo
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.