Stochastic optimization approaches to learning concise representations

by   William La Cava, et al.
University of Pennsylvania

We propose and study a method for learning interpretable features via stochastic optimization of feature architectures. Features are represented as multi-type expression trees using a set of activation functions common in neural networks in addition to other elementary functions. Continuous features are trained via gradient descent, and the performance of features in ML models is used to weight the rate of change among subcomponents of representations. The search process maintains an archive of representations with accuracy-complexity trade-offs to assist in generalization and interpretation. We compare several stochastic optimization approaches within this framework. We benchmark these variants on many real-world regression problems in comparison to other machine learning approaches. The best results across methods are obtained by search methods that directly optimize the accuracy-complexity trade-off in order to find simple architectures that generalize well.


page 1

page 2

page 3

page 4


Learning concise representations for regression by evolving networks of trees

We propose and study a method for learning interpretable representations...

Gradient Descent, Stochastic Optimization, and Other Tales

The goal of this paper is to debunk and dispel the magic behind black-bo...

Scenario trees and policy selection for multistage stochastic programming using machine learning

We propose a hybrid algorithmic strategy for complex stochastic optimiza...

SVGD: A Virtual Gradients Descent Method for Stochastic Optimization

Inspired by dynamic programming, we propose Stochastic Virtual Gradient ...

Decentralized Stochastic Optimization with Inherent Privacy Protection

Decentralized stochastic optimization is the basic building block of mod...

Maximizing Invariant Data Perturbation with Stochastic Optimization

Feature attribution methods, or saliency maps, are one of the most popul...

Brain-Like Stochastic Search: A Research Challenge and Funding Opportunity

Brain-Like Stochastic Search (BLiSS) refers to this task: given a family...

Please sign up or login with your details

Forgot password? Click here to reset