Achieving High Throughput and Elasticity in a Larger-than-Memory Store

06/05/2020
by   Chinmay Kulkarni, et al.
0

Millions of sensors, mobile applications and machines are now generating billions of events. This has led to specialized many-core key-value stores (KVSs) that can ingest and index these events at high rates (100 Mops/s/machine). These systems are efficient if events are generated on the same machine, but, to be practical and cost-effective they must ingest events over the network and scale across cloud resources elastically. We present Shadowfax, a new distributed key-value store that transparently spans DRAM, SSDs, and cloud blob storage while serving 130 Mops/s/VM over commodity Azure VMs using conventional Linux TCP. Beyond high single-VM performance, Shadowfax uses a unique approach to distributed reconfiguration that avoids any server-side key ownership checks or cross-core coordination both during normal operation and migration. Hence, Shadowfax can shift load in 17 s to improve cluster throughput by 10 the state-of-the-art, it has 8x better throughput (than Seastar+memcached) and scales out 6x faster.

READ FULL TEXT
research
05/02/2023

F2: Designing a Key-Value Store for Large Skewed Workloads

Today's key-value stores are either disk-optimized, focusing on large da...
research
05/29/2018

LaKe: An Energy Efficient, Low Latency, Accelerated Key-Value Store

Key-value store is a popular type of cloud computing applications. The p...
research
10/26/2020

TurboKV: Scaling Up The Performance of Distributed Key-Value Stores With In-Switch Coordination

The power and flexibility of software-defined networks lead to a program...
research
09/19/2022

DINOMO: An Elastic, Scalable, High-Performance Key-Value Store for Disaggregated Persistent Memory (Extended Version)

We present Dinomo, a novel key-value store for disaggregated persistent ...
research
08/05/2022

Scale-friendly In-network Coordination

The programmability of modern network devices has led to innovative rese...
research
09/06/2023

Grove: a Separation-Logic Library for Verifying Distributed Systems (Extended Version)

Grove is a concurrent separation logic library for verifying distributed...
research
03/01/2022

Pond: CXL-Based Memory Pooling Systems for Cloud Platforms

Public cloud providers seek to meet stringent performance requirements a...

Please sign up or login with your details

Forgot password? Click here to reset