S^3: Increasing GPU Utilization during Generative Inference for Higher Throughput

06/09/2023

∙

Generating texts with a large language model (LLM) consumes massive amounts of memory. Apart from the already-large model parameters, the key/value (KV) cache that holds information about previous tokens in a sequence can grow to be even larger than the model itself. This problem is exacerbated in one of the current LLM serving frameworks which reserves the maximum sequence length of memory for the KV cache to guarantee generating a complete sequence as they do not know the output sequence length. This restricts us to use a smaller batch size leading to lower GPU utilization and above all, lower throughput. We argue that designing a system with a priori knowledge of the output sequence can mitigate this problem. To this end, we propose S^3, which predicts the output sequence length, schedules generation queries based on the prediction to increase device resource utilization and throughput, and handle mispredictions. Our proposed method achieves 6.49× throughput over those systems that assume the worst case for the output sequence length.

READ FULL TEXT

S^3: Increasing GPU Utilization during Generative Inference for Higher Throughput

FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU

H_2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models

SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills

A Study on the Intersection of GPU Utilization and CNN Inference

Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time

TurboTransformers: An Efficient GPU Serving System For Transformer Models

Injecting knowledge into language generation: a case study in auto-charting after-visit care instructions from medical dialogue

S^3: Increasing GPU Utilization during Generative Inference for Higher Throughput

Related Research

FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU

H_2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models

SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills

A Study on the Intersection of GPU Utilization and CNN Inference

Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time

TurboTransformers: An Efficient GPU Serving System For Transformer Models

Injecting knowledge into language generation: a case study in auto-charting after-visit care instructions from medical dialogue