DeepAI AI Chat
Log In Sign Up

Overton: A Data System for Monitoring and Improving Machine-Learned Products

09/07/2019
by   Christopher Ré, et al.
0

We describe a system called Overton, whose main design goal is to support engineers in building, monitoring, and improving production machine learning systems. Key challenges engineers face are monitoring fine-grained quality, diagnosing errors in sophisticated applications, and handling contradictory or incomplete supervision data. Overton automates the life cycle of model construction, deployment, and monitoring by providing a set of novel high-level, declarative abstractions. Overton's vision is to shift developers to these higher-level tasks instead of lower-level machine learning tasks. In fact, using Overton, engineers can build deep-learning-based applications without writing any code in frameworks like TensorFlow. For over a year, Overton has been used in production to support multiple applications in both near-real-time applications and back-of-house processing. In that time, Overton-based applications have answered billions of queries in multiple languages and processed trillions of records reducing errors 1.7-2.9 times versus production systems.

READ FULL TEXT

page 1

page 2

page 3

page 4

07/13/2020

Monitoring and explainability of models in production

The machine learning lifecycle extends beyond the deployment stage. Moni...
07/09/2020

Green Lighting ML: Confidentiality, Integrity, and Availability of Machine Learning Systems in Deployment

Security and ethics are both core to ensuring that a machine learning sy...
12/22/2021

Log severity level classification: an approach for systems in production

Context: Logs are often the primary source of information for system dev...
07/24/2020

Orpheus: A New Deep Learning Framework for Easy Deployment and Evaluation of Edge Inference

Optimising deep learning inference across edge devices and optimisation ...
07/17/2020

Technologies for Trustworthy Machine Learning: A Survey in a Socio-Technical Context

Concerns about the societal impact of AI-based services and systems has ...
09/17/2019

Ludwig: a type-based declarative deep learning toolbox

In this work we present Ludwig, a flexible, extensible and easy to use t...
04/28/2021

AI Enabled Data Quality Monitoring with Hydra

Data quality monitoring is critical to all experiments impacting the qua...