Tracing and Manipulating Intermediate Values in Neural Math Problem Solvers

01/17/2023
by   Yuta Matsumoto, et al.
0

How language models process complex input that requires multiple steps of inference is not well understood. Previous research has shown that information about intermediate values of these inputs can be extracted from the activations of the models, but it is unclear where that information is encoded and whether that information is indeed used during inference. We introduce a method for analyzing how a Transformer model processes these inputs by focusing on simple arithmetic problems and their intermediate values. To trace where information about intermediate values is encoded, we measure the correlation between intermediate values and the activations of the model using principal component analysis (PCA). Then, we perform a causal intervention by manipulating model weights. This intervention shows that the weights identified via tracing are not merely correlated with intermediate values, but causally related to model predictions. Our findings show that the model has a locality to certain intermediate values, and this is useful for enhancing the interpretability of the models.

READ FULL TEXT
research
08/26/2021

Rethinking Why Intermediate-Task Fine-Tuning Works

Supplementary Training on Intermediate Labeled-data Tasks (STILTs) is a ...
research
06/06/2023

Inference-Time Intervention: Eliciting Truthful Answers from a Language Model

We introduce Inference-Time Intervention (ITI), a technique designed to ...
research
06/15/2018

Sparse Principal Component based High-Dimensional Mediation Analysis

Causal mediation analysis aims to quantify the intermediate effect of a ...
research
02/03/2023

SPARLING: Learning Latent Representations with Extremely Sparse Activations

Real-world processes often contain intermediate state that can be modele...
research
05/23/2022

Tracing Knowledge in Language Models Back to the Training Data

Neural language models (LMs) have been shown to memorize a great deal of...
research
06/09/2022

Cross-Model Conjunctive Queries over Relation and Tree-structured Data (Extended)

Conjunctive queries are the most basic and central class of database que...
research
10/23/2019

Deja-vu: Double Feature Presentation in Deep Transformer Networks

Deep acoustic models typically receive features in the first layer of th...

Please sign up or login with your details

Forgot password? Click here to reset