An effective algorithm for hyperparameter optimization of neural networks

05/23/2017
by   Gonzalo Diaz, et al.
0

A major challenge in designing neural network (NN) systems is to determine the best structure and parameters for the network given the data for the machine learning problem at hand. Examples of parameters are the number of layers and nodes, the learning rates, and the dropout rates. Typically, these parameters are chosen based on heuristic rules and manually fine-tuned, which may be very time-consuming, because evaluating the performance of a single parametrization of the NN may require several hours. This paper addresses the problem of choosing appropriate parameters for the NN by formulating it as a box-constrained mathematical optimization problem, and applying a derivative-free optimization tool that automatically and effectively searches the parameter space. The optimization tool employs a radial basis function model of the objective function (the prediction accuracy of the NN) to accelerate the discovery of configurations yielding high accuracy. Candidate configurations explored by the algorithm are trained to a small number of epochs, and only the most promising candidates receive full training. The performance of the proposed methodology is assessed on benchmark sets and in the context of predicting drug-drug interactions, showing promising results. The optimization tool used in this paper is open-source.

READ FULL TEXT

page 16

page 17

research
08/08/2022

Neural Optimization Machine: A Neural Network Approach for Optimization

A novel neural network (NN) approach is proposed for constrained optimiz...
research
10/31/2022

Evaluating Point-Prediction Uncertainties in Neural Networks for Drug Discovery

Neural Network (NN) models provide potential to speed up the drug discov...
research
09/13/2019

Classifying Topological Charge in SU(3) Yang-Mills Theory with Machine Learning

We apply a machine learning technique for identifying the topological ch...
research
08/29/2019

Enabling Simulation-Based Optimization Through Machine Learning: A Case Study on Antenna Design

Complex phenomena are generally modeled with sophisticated simulators th...
research
07/06/2017

Simultaneous Optimization of Neural Network Weights and Active Nodes using Metaheuristics

Optimization of neural network (NN) significantly influenced by the tran...
research
11/15/2017

Bridging the Gap Between Neural Networks and Neuromorphic Hardware with A Neural Network Compiler

Different from training common neural networks (NNs) for inference on ge...
research
01/22/2021

Baseline Pruning-Based Approach to Trojan Detection in Neural Networks

This paper addresses the problem of detecting trojans in neural networks...

Please sign up or login with your details

Forgot password? Click here to reset