
Efficient and Robust Machine Learning for RealWorld Systems
While machine learning is traditionally a resource intensive task, embedded systems, autonomous navigation and the vision of the InternetofThings fuel the interest in resource efficient approaches. These approaches require a carefully chosen tradeoff between performance and resource consumption in terms of computation and energy. On top of this, it is crucial to treat uncertainty in a consistent manner in all but the simplest applications of machine learning systems. In particular, a desideratum for any realworld system is to be robust in the presence of outliers and corrupted data, as well as being `aware' of its limits, i.e. the system should maintain and provide an uncertainty estimate over its own predictions. These complex demands are among the major challenges in current machine learning research and key to ensure a smooth transition of machine learning technology into every day's applications. In this article, we provide an overview of the current state of the art of machine learning techniques facilitating these realworld requirements. First we provide a comprehensive review of resourceefficiency in deep neural networks with focus on techniques for model size reduction, compression and reduced precision. These techniques can be applied during training or as postprocessing and are widely used to reduce both computational complexity and memory footprint. As most (practical) neural networks are limited in their ways to treat uncertainty, we contrast them with probabilistic graphical models, which readily serve these desiderata by means of probabilistic inference. In that way, we provide an extensive overview of the current stateoftheart of robust and efficient machine learning for realworld systems.
12/05/2018 ∙ by Franz Pernkopf, et al. ∙ 16 ∙ shareread it

Complex Signal Denoising and Interference Mitigation for Automotive Radar Using Convolutional Neural Networks
Driver assistance systems as well as autonomous cars have to rely on sensors to perceive their environment. A heterogeneous set of sensors is used to perform this task robustly. Among them, radar sensors are indispensable because of their range resolution and the possibility to directly measure velocity. Since more and more radar sensors are deployed on the streets, mutual interference must be dealt with. In the so far unregulated automotive radar frequency band, a sensor must be capable of detecting, or even mitigating the harmful effects of interference, which include a decreased detection sensitivity. In this paper, we address this issue with Convolutional Neural Networks (CNNs), which are stateoftheart machine learning tools. We show that the ability of CNNs to find structured information in data while preserving local information enables superior denoising performance. To achieve this, CNN parameters are found using training with simulated data and integrated into the automotive radar signal processing chain. The presented method is compared with the state of the art, highlighting its promising performance. Hence, CNNs can be employed for interference mitigation as an alternative to conventional signal processing methods. Code and pretrained models are available at https://github.com/johannarock/imRICnn.
06/24/2019 ∙ by Johanna Rock, et al. ∙ 3 ∙ shareread it

Safe SemiSupervised Learning of SumProduct Networks
In several domains obtaining class annotations is expensive while at the same time unlabelled data are abundant. While most semisupervised approaches enforce restrictive assumptions on the data distribution, recent work has managed to learn semisupervised models in a nonrestrictive regime. However, so far such approaches have only been proposed for linear models. In this work, we introduce semisupervised parameter learning for SumProduct Networks (SPNs). SPNs are deep probabilistic models admitting inference in linear time in number of network edges. Our approach has several advantages, as it (1) allows generative and discriminative semisupervised learning, (2) guarantees that adding unlabelled data can increase, but not degrade, the performance (safe), and (3) is computationally efficient and does not enforce restrictive assumptions on the data distribution. We show on a variety of data sets that safe semisupervised learning with SPNs is competitive compared to stateoftheart and can lead to a better generative and discriminative objective value than a purely supervised approach.
10/10/2017 ∙ by Martin Trapp, et al. ∙ 0 ∙ shareread it

On the Latent Variable Interpretation in SumProduct Networks
One of the central themes in SumProduct networks (SPNs) is the interpretation of sum nodes as marginalized latent variables (LVs). This interpretation yields an increased syntactic or semantic structure, allows the application of the EM algorithm and to efficiently perform MPE inference. In literature, the LV interpretation was justified by explicitly introducing the indicator variables corresponding to the LVs' states. However, as pointed out in this paper, this approach is in conflict with the completeness condition in SPNs and does not fully specify the probabilistic model. We propose a remedy for this problem by modifying the original approach for introducing the LVs, which we call SPN augmentation. We discuss conditional independencies in augmented SPNs, formally establish the probabilistic interpretation of the sumweights and give an interpretation of augmented SPNs as Bayesian networks. Based on these results, we find a sound derivation of the EM algorithm for SPNs. Furthermore, the Viterbistyle algorithm for MPE proposed in literature was never proven to be correct. We show that this is indeed a correct algorithm, when applied to selective SPNs, and in particular when applied to augmented SPNs. Our theoretical results are confirmed in experiments on synthetic data and 103 realworld datasets.
01/22/2016 ∙ by Robert Peharz, et al. ∙ 0 ∙ shareread it

Fixed Points of Belief Propagation  An Analysis via Polynomial Homotopy Continuation
Belief propagation (BP) is an iterative method to perform approximate inference on arbitrary graphical models. Whether BP converges and if the solution is a unique fixed point depends on both the structure and the parametrization of the model. To understand this dependence it is interesting to find all fixed points. In this work, we formulate a set of polynomial equations, the solutions of which correspond to BP fixed points. To solve such a nonlinear system we present the numerical polynomialhomotopycontinuation (NPHC) method. Experiments on binary Ising models and on errorcorrecting codes show how our method is capable of obtaining all BP fixed points. On Ising models with fixed parameters we show how the structure influences both the number of fixed points and the convergence properties. We further asses the accuracy of the marginals and weighted combinations thereof. Weighting marginals with their respective partition function increases the accuracy in all experiments. Contrary to the conjecture that uniqueness of BP fixed points implies convergence, we find graphs for which BP fails to converge, even though a unique fixed point exists. Moreover, we show that this fixed point gives a good approximation, and the NPHC method is able to obtain this fixed point.
05/20/2016 ∙ by Christian Knoll, et al. ∙ 0 ∙ shareread it

Exact Maximum Margin Structure Learning of Bayesian Networks
Recently, there has been much interest in finding globally optimal Bayesian network structures. These techniques were developed for generative scores and can not be directly extended to discriminative scores, as desired for classification. In this paper, we propose an exact method for finding network structures maximizing the probabilistic soft margin, a successfully applied discriminative score. Our method is based on branchandbound techniques within a linear programming framework and maintains an anytime solution, together with worstcase suboptimality bounds. We apply a set of order constraints for enforcing the network structure to be acyclic, which allows a compact problem representation and the use of generalpurpose optimization techniques. In classification experiments, our methods clearly outperform generatively trained network structures and compete with support vector machines.
06/27/2012 ∙ by Robert Peharz, et al. ∙ 0 ∙ shareread it

SumProduct Networks for Sequence Labeling
We consider higherorder linearchain conditional random fields (HOLCCRFs) for sequence modelling, and use sumproduct networks (SPNs) for representing higherorder input and outputdependent factors. SPNs are a recently introduced class of deep models for which exact and efficient inference can be performed. By combining HOLCCRFs with SPNs, expressive models over both the output labels and the hidden variables are instantiated while still enabling efficient exact inference. Furthermore, the use of higherorder factors allows us to capture relations of multiple input segments and multiple output labels as often present in realworld data. These relations can not be modelled by the commonly used firstorder models and higherorder models with local factors including only a single output label. We demonstrate the effectiveness of our proposed models for sequence labeling. In extensive experiments, we outperform other stateoftheart methods in optical character recognition and achieve competitive results in phone classification.
07/06/2018 ∙ by Martin Ratajczak, et al. ∙ 0 ∙ shareread it

Automatic Clustering of a Network Protocol with WeaklySupervised Clustering
Abstraction is a fundamental part when learning behavioral models of systems. Usually the process of abstraction is manually defined by domain experts. This paper presents a method to perform automatic abstraction for network protocols. In particular a weakly supervised clustering algorithm is used to build an abstraction with a small vocabulary size for the widely used TLS protocol. To show the effectiveness of the proposed method we compare the resultant abstract messages to a manually constructed (reference) abstraction. With a small amount of sideinformation in the form of a few labeled examples this method finds an abstraction that matches the reference abstraction perfectly.
06/04/2018 ∙ by Tobias Schrank, et al. ∙ 0 ∙ shareread it

Learning Deep Mixtures of Gaussian Process Experts Using SumProduct Networks
While Gaussian processes (GPs) are the method of choice for regression tasks, they also come with practical difficulties, as inference cost scales cubic in time and quadratic in memory. In this paper, we introduce a natural and expressive way to tackle these problems, by incorporating GPs in sumproduct networks (SPNs), a recently proposed tractable probabilistic model allowing exact and efficient inference. In particular, by using GPs as leaves of an SPN we obtain a novel flexible prior over functions, which implicitly represents an exponentially large mixture of local GPs. Exact and efficient posterior inference in this model can be done in a natural interplay of the inference mechanisms in GPs and SPNs. Thereby, each GP is  similarly as in a mixture of experts approach  responsible only for a subset of data points, which effectively reduces inference cost in a divide and conquer fashion. We show that integrating GPs into the SPN framework leads to a promising probabilistic regression model which is: (1) computational and memory efficient, (2) allows efficient and exact posterior inference, (3) is flexible enough to mix different kernel functions, and (4) naturally accounts for nonstationarities in time series. In a variate of experiments, we show that the SPNGP model can learn input dependent parameters and hyperparameters and is on par with or outperforms the traditional GPs as well as state of the art approximations on realworld data.
09/12/2018 ∙ by Martin Trapp, et al. ∙ 0 ∙ shareread it

SelfGuided Belief Propagation  A Homotopy Continuation Method
We propose selfguided belief propagation (SBP) that modifies belief propagation (BP) by incorporating the pairwise potentials only gradually. This homotopy continuation method converges to a unique solution and increases the accuracy without increasing the computational burden. We apply SBP to grid graphs, complete graphs, and random graphs with random Ising potentials and show that: (i) SBP is superior in terms of accuracy whenever BP converges, and (ii) SBP obtains a unique, stable, and accurate solution whenever BP does not converge. We further provide a formal analysis to demonstrate that SBP obtains the global optimum of the Bethe approximation for attractive models with unidirectional fields.
12/04/2018 ∙ by Christian Knoll, et al. ∙ 0 ∙ shareread it

Sound event detection using weaklylabeled semisupervised data with GCRNNS, VAT and SelfAdaptive Label Refinement
In this paper, we present a gated convolutional recurrent neural network based approach to solve task 4, largescale weakly labelled semisupervised sound event detection in domestic environments, of the DCASE 2018 challenge. Gated linear units and a temporal attention layer are used to predict the onset and offset of sound events in 10s long audio clips. Whereby for training only weaklylabelled data is used. Virtual adversarial training is used for regularization, utilizing both labelled and unlabeled data. Furthermore, we introduce selfadaptive label refinement, a method which allows unsupervised adaption of our trained system to refine the accuracy of framelevel class predictions. The proposed system reaches an overall macro averaged eventbased Fscore of 34.6 baseline system.
10/16/2018 ∙ by Robert Harb, et al. ∙ 0 ∙ shareread it
Franz Pernkopf
is this you? claim profile