Towards naturalistic human neuroscience and neuroengineering: behavior mining in long-term video and neural recordings

01/23/2020
by   Satpreet H. Singh, et al.
17

Recent advances in brain recording technology and artificial intelligence are propelling a new paradigm in neuroscience beyond the traditional controlled experiment. Naturalistic neuroscience studies neural computations associated with spontaneous behaviors performed in unconstrained settings. Analyzing such unstructured data lacking a priori experimental design remains a significant challenge, especially when the data is multi-modal and long-term. Here we describe an automated approach for analyzing large (≈250 GB/subject) datasets of simultaneously recorded human electrocorticography (ECoG) and naturalistic behavior video data for 12 subjects. Our pipeline discovers and annotates thousands of instances of human upper-limb movement events in long-term (7–9 day) naturalistic behavior data using a combination of computer vision, discrete latent-variable modeling, and string pattern-matching. Analysis of the simultaneously recorded brain data uncovers neural signatures of movement that corroborate prior findings from traditional controlled experiments. We also prototype a decoder for a movement initiation detection task to demonstrate the efficacy of our pipeline as a source of training data for brain-computer interfacing applications. We plan to publish our curated dataset, which captures naturalistic neural and behavioral variability at a scale not previously available. We believe this data will enable further research on models of neural function and decoding that incorporate such naturalistic variability and perform more robustly in real-world settings.

READ FULL TEXT

Authors

page 2

page 3

page 5

09/13/2017

AJILE Movement Prediction: Multimodal Deep Learning for Natural Human Neural Recordings and Video

Developing useful interfaces between brains and machines is a grand chal...
09/20/2019

Target-Specific Action Classification for Automated Assessment of Human Motor Behavior from Video

Objective monitoring and assessment of human motor behavior can improve ...
03/27/2017

A Study on the Extraction and Analysis of a Large Set of Eye Movement Features during Reading

This work presents a study on the extraction and analysis of a set of 10...
02/11/2021

Neurorobots as a Means Toward Neuroethology and Explainable AI

Understanding why deep neural networks and machine learning algorithms a...
04/28/2021

The Algonauts Project 2021 Challenge: How the Human Brain Makes Sense of a World in Motion

The sciences of natural and artificial intelligence are fundamentally co...
04/06/2022

Open-Source Tools for Behavioral Video Analysis: Setup, Methods, and Development

Recently developed methods for video analysis, especially models for pos...
12/02/2021

Overcoming the Domain Gap in Neural Action Representations

Relating animal behaviors to brain activity is a fundamental goal in neu...

Code Repositories

mining2021

Code to accompany: https://arxiv.org/abs/2001.08349


view repo
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.