DeepAI AI Chat
Log In Sign Up

Debugging using Orthogonal Gradient Descent

by   Narsimha Chilkuri, et al.
University of Waterloo

In this report we consider the following problem: Given a trained model that is partially faulty, can we correct its behaviour without having to train the model from scratch? In other words, can we “debug" neural networks similar to how we address bugs in our mathematical models and standard computer code. We base our approach on the hypothesis that debugging can be treated as a two-task continual learning problem. In particular, we employ a modified version of a continual learning algorithm called Orthogonal Gradient Descent (OGD) to demonstrate, via two simple experiments on the MNIST dataset, that we can in-fact unlearn the undesirable behaviour while retaining the general performance of the model, and we can additionally relearn the appropriate behaviour, both without having to train the model from scratch.


page 1

page 2

page 3

page 4


Continual Backprop: Stochastic Gradient Descent with Persistent Randomness

The Backprop algorithm for learning in neural networks utilizes two mech...

Generalisation Guarantees for Continual Learning with Orthogonal Gradient Descent

In continual learning settings, deep neural networks are prone to catast...

Scaling the Number of Tasks in Continual Learning

Standard gradient descent algorithms applied to sequences of tasks are k...

Continual learning using hash-routed convolutional neural networks

Continual learning could shift the machine learning paradigm from data c...