Sequence Training of DNN Acoustic Models With Natural Gradient

04/06/2018
by   Adnan Haider, et al.
0

Deep Neural Network (DNN) acoustic models often use discriminative sequence training that optimises an objective function that better approximates the word error rate (WER) than frame-based training. Sequence training is normally implemented using Stochastic Gradient Descent (SGD) or Hessian Free (HF) training. This paper proposes an alternative batch style optimisation framework that employs a Natural Gradient (NG) approach to traverse through the parameter space. By correcting the gradient according to the local curvature of the KL-divergence, the NG optimisation process converges more quickly than HF. Furthermore, the proposed NG approach can be applied to any sequence discriminative training criterion. The efficacy of the NG method is shown using experiments on a Multi-Genre Broadcast (MGB) transcription task that demonstrates both the computational efficiency and the accuracy of the resulting DNN models.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
10/03/2018

Combining Natural Gradient with Hessian Free Methods for Sequence Training

This paper presents a new optimisation approach to train Deep Neural Net...
research
10/11/2016

GMM-Free Flat Start Sequence-Discriminative DNN Training

Recently, attempts have been made to remove Gaussian mixture models (GMM...
research
03/12/2021

A Distributed Optimisation Framework Combining Natural Gradient with Hessian-Free for Discriminative Sequence Training

This paper presents a novel natural gradient and Hessian-free (NGHF) opt...
research
11/10/2017

Deep Within-Class Covariance Analysis for Acoustic Scene Classification

Within-Class Covariance Normalization (WCCN) is a powerful post-processi...
research
08/08/2021

Deep Neural Network for DrawiNg Networks, (DNN)^2

By leveraging recent progress of stochastic gradient descent methods, se...
research
03/05/2023

Revisiting the Noise Model of Stochastic Gradient Descent

The stochastic gradient noise (SGN) is a significant factor in the succe...
research
05/16/2019

Learning discriminative features in sequence training without requiring framewise labelled data

In this work, we try to answer two questions: Can deeply learned feature...

Please sign up or login with your details

Forgot password? Click here to reset