Exploiting Test Time Evidence to Improve Predictions of Deep Neural Networks

by   Dinesh Khandelwal, et al.

Many prediction tasks, especially in computer vision, are often inherently ambiguous. For example, the output of semantic segmentation may depend on the scale one is looking at, and image saliency or video summarization is often user or context dependent. Arguably, in such scenarios, exploiting instance specific evidence, such as scale or user context, can help resolve the underlying ambiguity leading to the improved predictions. While existing literature has considered incorporating such evidence in classical models such as probabilistic graphical models (PGMs), there is limited (or no) prior work looking at this problem in the context of deep neural network (DNN) models. In this paper, we present a generic multi task learning (MTL) based framework which handles the evidence as the output of one or more secondary tasks, while modeling the original problem as the primary task of interest. Our training phase is identical to the one used by standard MTL architectures. During prediction, we back-propagate the loss on secondary task(s) such that network weights are re-adjusted to match the evidence. An early stopping or two norm based regularizer ensures weights do not deviate significantly from the ones learned originally. Implementation in two specific scenarios (a) predicting semantic segmentation given the image level tags (b) predicting instance level segmentation given the text description of the image, clearly demonstrates the effectiveness of our proposed approach.


page 1

page 6

page 8

page 11

page 12

page 13


Improving Online Performance Prediction for Semantic Segmentation

In this work we address the task of observing the performance of a seman...

Multi-Task Self-Supervised Learning for Image Segmentation Task

Thanks to breakthroughs in AI and Deep learning methodology, Computer vi...

Neither Quick Nor Proper -- Evaluation of QuickProp for Learning Deep Neural Networks

Neural networks and especially convolutional neural networks are of grea...

Deeply Cascaded U-Net for Multi-Task Image Processing

In current practice, many image processing tasks are done sequentially (...

Deep Automated Multi-task Learning

Multi-task learning (MTL) has recently contributed to learning better re...

Deep Neural Networks for Automatic Grain-matrix Segmentation in Plane and Cross-polarized Sandstone Photomicrographs

Grain segmentation of sandstone that is partitioning the grain from its ...

Please sign up or login with your details

Forgot password? Click here to reset