Stochastic optimization approaches to learning concise representations

07/03/2018
by   William La Cava, et al.
2

We propose and study a method for learning interpretable features via stochastic optimization of feature architectures. Features are represented as multi-type expression trees using a set of activation functions common in neural networks in addition to other elementary functions. Continuous features are trained via gradient descent, and the performance of features in ML models is used to weight the rate of change among subcomponents of representations. The search process maintains an archive of representations with accuracy-complexity trade-offs to assist in generalization and interpretation. We compare several stochastic optimization approaches within this framework. We benchmark these variants on many real-world regression problems in comparison to other machine learning approaches. The best results across methods are obtained by search methods that directly optimize the accuracy-complexity trade-off in order to find simple architectures that generalize well.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
07/03/2018

Learning concise representations for regression by evolving networks of trees

We propose and study a method for learning interpretable representations...
research
05/02/2022

Gradient Descent, Stochastic Optimization, and Other Tales

The goal of this paper is to debunk and dispel the magic behind black-bo...
research
12/19/2011

Scenario trees and policy selection for multistage stochastic programming using machine learning

We propose a hybrid algorithmic strategy for complex stochastic optimiza...
research
07/09/2019

SVGD: A Virtual Gradients Descent Method for Stochastic Optimization

Inspired by dynamic programming, we propose Stochastic Virtual Gradient ...
research
05/08/2022

Decentralized Stochastic Optimization with Inherent Privacy Protection

Decentralized stochastic optimization is the basic building block of mod...
research
07/12/2018

Maximizing Invariant Data Perturbation with Stochastic Optimization

Feature attribution methods, or saliency maps, are one of the most popul...
research
06/01/2010

Brain-Like Stochastic Search: A Research Challenge and Funding Opportunity

Brain-Like Stochastic Search (BLiSS) refers to this task: given a family...

Please sign up or login with your details

Forgot password? Click here to reset