Optimizing Kernel Machines using Deep Learning

11/15/2017
by   Huan Song, et al.
0

Building highly non-linear and non-parametric models is central to several state-of-the-art machine learning systems. Kernel methods form an important class of techniques that induce a reproducing kernel Hilbert space (RKHS) for inferring non-linear models through the construction of similarity functions from data. These methods are particularly preferred in cases where the training data sizes are limited and when prior knowledge of the data similarities is available. Despite their usefulness, they are limited by the computational complexity and their inability to support end-to-end learning with a task-specific objective. On the other hand, deep neural networks have become the de facto solution for end-to-end inference in several learning paradigms. In this article, we explore the idea of using deep architectures to perform kernel machine optimization, for both computational efficiency and end-to-end inferencing. To this end, we develop the DKMO (Deep Kernel Machine Optimization) framework, that creates an ensemble of dense embeddings using Nystrom kernel approximations and utilizes deep learning to generate task-specific representations through the fusion of the embeddings. Intuitively, the filters of the network are trained to fuse information from an ensemble of linear subspaces in the RKHS. Furthermore, we introduce the kernel dropout regularization to enable improved training convergence. Finally, we extend this framework to the multiple kernel case, by coupling a global fusion layer with pre-trained deep kernel machines for each of the constituent kernels. Using case studies with limited training data, and lack of explicit feature sources, we demonstrate the effectiveness of our framework over conventional model inferencing techniques.

READ FULL TEXT

page 7

page 9

research
02/11/2018

Learning Multiple Levels of Representations with Kernel Machines

We propose a connectionist-inspired kernel machine model with three key ...
research
12/28/2016

A Deep Learning Approach To Multiple Kernel Fusion

Kernel fusion is a popular and effective approach for combining multiple...
research
01/15/2010

Kernel machines with two layers and multiple kernel learning

In this paper, the framework of kernel machines with two layers is intro...
research
05/20/2016

End-to-End Kernel Learning with Supervised Convolutional Kernel Networks

In this paper, we introduce a new image representation based on a multil...
research
01/01/2020

Fast Estimation of Information Theoretic Learning Descriptors using Explicit Inner Product Spaces

Kernel methods form a theoretically-grounded, powerful and versatile fra...
research
08/19/2019

Deep neural network or dermatologist?

Deep learning techniques have proven high accuracy for identifying melan...
research
09/07/2022

Bayesian learning of feature spaces for multitasks problems

This paper presents a Bayesian framework to construct non-linear, parsim...

Please sign up or login with your details

Forgot password? Click here to reset