A Hybrid Control Scheme for Adaptive Live Streaming

by   Huan Peng, et al.

The live streaming is more challenging than on-demand streaming, because the low latency is also a strong requirement in addition to the trade-off between video quality and jitters in playback. To balance several inherently conflicting performance metrics and improve the overall quality of experience (QoE), many adaptation schemes have been proposed. Bitrate adaptation is one of the major solutions for video streaming under time-varying network conditions, which works even better combining with some latency control methods, such as adaptive playback rate control and frame dropping. However, it still remains a challenging problem to design an algorithm to combine these adaptation schemes together. To tackle this problem, we propose a hybrid control scheme for adaptive live streaming, namely HYSA, based on heuristic playback rate control, latency-constrained bitrate control and QoE-oriented adaptive frame dropping. The proposed scheme utilizes Kaufman's Adaptive Moving Average (KAMA) to predict segment bitrates for better rate decisions. Extensive simulations demonstrate that HYSA outperforms most of the existing adaptation schemes on overall QoE.



There are no comments yet.


page 1

page 2

page 3

page 4


Adaptive Bitrate Streaming Over Cellular Networks: Rate Adaptation and Data Savings Strategies

Adaptive bitrate streaming (ABR) has become thede factotechnique for vid...

fybrrStream: A WebRTC based Efficient and Scalable P2P Live Streaming Platform

The demand for streaming media and live video conferencing is at peak an...

Performance Analysis of Reliable Video Streaming with Strict Playout Deadline in Multi-Hop Wireless Networks

Motivated by emerging vision-based intelligent services, we consider the...

QARC: Video Quality Aware Rate Control for Real-Time Video Streaming based on Deep Reinforcement Learning

Real-time video streaming is now one of the main applications in all net...

QoE-driven Coupled Uplink and Downlink Rate Adaptation for 360-degree Video Live Streaming

360-degree video provides an immersive 360-degree viewing experience and...

Content-Aware Personalised Rate Adaptation for Adaptive Streaming via Deep Video Analysis

Adaptive bitrate (ABR) streaming is the de facto solution for achieving ...

CBA: Contextual Quality Adaptation for Adaptive Bitrate Video Streaming (Extended Version)

Recent advances in quality adaptation algorithms leave adaptive bitrate ...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1. Introduction

Recent years have seen tremendous growth of live streaming applications. Different from on-demand streaming, live streaming has tight latency constraints. It’s very challenging to reduce the latency while maintaining high video quality and smooth playback. Bitrate adaptation is the most common solution for improving the QoE of video streaming under time-varying network conditions. However, the existing adaptive bitrate algorithms, such as BOLA (Spiteri et al., 2016), RobustMPC (Yin et al., 2015), Pensieve (Mao et al., 2017) and Oboe (Akhtar et al., 2018), haven’t taken the latency into account. In addition to adapting the video bitrate, playback rate control and frame dropping are always utilized to reduce the latency in live streaming. Mingfu Li et al. (Li et al., 2018) employed playback rate adaptation, and Miller et al. (Miller et al., 2017) and Shen Y et al. (Shen et al., 2018) adopted frame dropping to reduce the latency. However, these methods lack the capability to balance between latency and video quality, without fully considering all aspects of QoE.

In this paper, we propose HYSA, an effective hybrid control scheme to realize playback rate adaptation, bitrate adaptation and frame dropping adaptation. First, the playback rate is adaptively adjusted with a buffer-based heuristic method. Then, taking advantage of the playback rate decisions and KAMA-based predicted segment bitrates, we propose the latency-constrained bitrate adaptation scheme to make optimal bitrate decisions for the QoE-oriented frame dropping adaptation scheme in the next step. Our extensive simulation results demonstrate that the proposed HYSA outperforms existing adaptation schemes on the overall QoE.

The rest of the paper is organized as follows. Section 2 introduces the system framework in live streaming scenario and the simulation platform. Section 3 describes the details of the proposed hybrid control scheme. In Section 4, the performance of our method is evaluated comprehensively. Section 5 concludes this paper.

2. System Overview

Figure 1. Architecture of live streaming system and the simulator

Architecture of live streaming system and simulator.

Figure 1 depicts a typical scenario of live streaming. The video frames generated in real-time are uploaded to a transcoding server, which re-encodes the video into multiple representations, each at a different bitrate. These representations are then transmitted to CDN (Content Delivery Network) nodes, which act as edge servers. The client decides which representation to download from one of the CDN nodes based on some state information, such as buffer occupancy and throughput. Besides, the client can adaptively adjust its playback rate and skip some frames to reduce the latency.

The simulator simulates the downloading of video frames under various network conditions and the adaptive playback of a player. The simulator takes video trace, network trace and decisions from the hybrid control scheme as inputs. The video trace records the size of video frames and their time of arrival at CDN, while the network trace simulates throughputs of the downloading network. The simulator collects observations after downloading every frame, then the control scheme makes decisions after downloading a complete group of pictures (GOP) by taking advantage of these observations. Hereinafter we use a segment to refer to a GOP.

3. Hybrid Control Scheme

In this section, we describe the details of the proposed hybrid control scheme – HYSA, which consists of segment bitrate prediction module, playback rate control module, bitrate control module and frame dropping control module, as shown in Figure 2.

3.1. QoE model

Many studies have highlighted the critical role that QoE plays in the design of adaptation schemes. Here we refer to the QoE model specified by the grand challenge in ACM MM 2019111https://www.aitrans.online/MMGC/, which mainly focuses on five performance metrics: video quality, rebuffering, latency, frame skipping and quality switching. Their impacts on QoE are notated by , , , and respectively. The overall QoE is calculated as follows:

where is the total number of frames. The coding bitrate of frame is notated by , while is its length. and denote the rebuffering duration and video length skipped when downloading frame respectively, and is the latency. , , , , are weight factors used to describe the importance of corresponding QoE metrics. describes the quality variation of two adjacent video frames.

Figure 2. Overview of the proposed hybrid control scheme

Architecture of the proposed hybrid control scheme for live streaming

3.2. Segment Bitrate Prediction

The next download duration can be estimated if the next segment’s actual bitrate is known. However, it isn’t realistic to get information about the video that hasn’t been generated in live streaming. Therefore, most of the existing algorithms estimate the next download duration using the coding bitrate of the upcoming segment instead of its actual bitrate, ignoring the fact that segment’s actual bitrate varies significantly for a given coding bitrate, as indicated by Figure

3. Besides, we can observe that for segments in two videos with the same content but different qualities, the ratios of their actual bitrates have a similar trend with the ratios of their coding bitrates.

Assume that each segment is edcoded at different bitrates, and let and be the coding bitrate and the actual bitrate of segment at quality level , satisfying , , . For the -th segment already downloaded at quality , we can estimate its actual bitrates of other quality levels based on the above observation:


Here we employ Kaufman’s Adaptive Moving Average (KAMA) to predict the actual bitrates of the upcoming segment. The bitrate of the next segment at quality level , denoted as , can be predicted as follows:


The smoothing factor is dynamically calculated for every sample, i.e. segment bitrate. To get the smoothing factor, we first set two boundaries for it, based on the method of calculating smoothing factor in Exponential Moving Average (EMA):


The and are the number of samples for the slowest and fastest EMA respectively. Then we calculate the efficiency ratio , which shows the efficiency of sample fluctuations.


where specifies the number of samples used for calculating , and is always between 0 and 1. Using and two boundaries, the can be derived as below:

Figure 3. Actual bitrates of segments at variable coding bitrates

Real Bitrates of Segments at Variable Coding Bitrates

3.3. Playback Rate Control

In the playback rate control module, the target buffer is introduced to help control playback rate . The so-called target buffer is marked as , where and form a buffer interval that player can play buffered video at normal playback rate, and means the target buffer occupancy to resume playback when interruptions occur. When the buffer occupancy is below , the player will slow down the playback rate to 0.95, while speeding up the playback rate to 1.05 when the buffer occupancy is above . In the grand challenge, the target buffer can only be set to 0 or 1, i.e. or , satisfying .

The heuristic playback rate control module decides which target buffer to choose solely depending on current buffer occupancy . Five cases are considered, as described below:

Case 1:, which means there is a substantial risk of interruptions, and whichever target buffer is chosen. To restart playback as soon as possible, the target buffer is set to 0 due to smaller .

Case 2:, which means stalls may be encountered even though it is less possible than Case 1. Therefore, the target buffer is set to 1 to make player slow down the playback rate to 0.95.

Case 3:, which means buffer occupancy remains in a reasonable interval, and whichever target buffer is chosen.

Case 4:, which means the latency is relatively large due to buffered video. Therefore, the target buffer is set to 0 to speed up the playback rate to 1.05.

Case 5:, which means large latency caused by buffered video, and is equal to 1.05 whichever target buffer is chosen.

Based on the discussions above, the playback rate can be adjusted by the target buffer as follows:

HYSA 2424.04 3548.00 -418.92 -608.32 -86.28 -10.44
HYSA-N 2336.33 3398.36 -398.11 -571.27 -82.28 -10.37
MPC 2000.44 3056.23 -255.98 -759.17 -30.23 -10.41
DTTB 2038.34 3472.63 -410.20 -952.57 -63.72 -7.80
Table 1. Performance comparison of the adaptation schemes
(a) Sports Video
(b) Room Video
(c) Game Video
Figure 4. Overall QoE when streaming three types of videos

3.4. Bitrate Control

The latency-constrained bitrate control module makes bitrate decisions based on some state information, such as buffer occupancy, predicted segment bitrate and playback rate derived from the playback rate control module. The optimal bitrate is selected to minimize , the estimated latency after downloading the next segment.

Denoting segment length by , the duration of downloading the upcoming segment n+1 at quality , notated by , can be calculated using the predicted segment bitrate and the estimated network throughput derived by Weighted Moving Average as follows:


During the downloading process, the player will consume buffered video at playback rate if the buffer isn’t drained. The buffer occupancy after downloading, namely , can be calculated as:


To estimate the latency caused by video accumulated at CDN after downloading the next segment, we start with estimating the video accumulation speed at CDN, denoted by , as follows:


where is a predictive factor, and indicates the index of the latest frame at CDN after downloading the -th segment. Then, the latency caused by accumulated video at CDN after next downloading interval, namely , is calculated by:


where represents the index of most recently downloaded frame. The objective of the bitrate adaptive algorithm is to find the quality that results in lowest latency without interruptions, which can be described as the following optimization problem:

where is a warning threshold indicating the upcoming stall event. Since there are finite available bitrates, we can get the optimal quality level by going through all choices.

3.5. Frame Dropping Control

The client can reduce the latency by dropping some frames when current latency is above a specific threshold. We propose a QoE-oriented frame dropping method to adaptively adjust the latency threshold that triggers frame skipping. Assuming that the client skips frames when the latency is above during next segment’s downloading, the positive and negative impact of frame skipping on QoE compared to non-skip, denoted by and , can be estimated based on the given QoE model:


Here, is used to estimate the average latency if frame skipping is not performed when the latency is above . Therefore, when is larger than , frame skipping is a good choice:


Thus, the latency threshold that triggers frame skipping when downloading the next segment can be set as:


4. Evaluation

In this section, we present the extensive evaluations, by which the following questions can be answered: (1) Is the segment bitrate prediction helpful? (2) How does HYSA compare to existing adaptation schemes?

To evaluate the performance of HYSA for streaming different types of videos, video traces in three scenes including room, game and sports are used for the evaluations. These videos are encoded at bitrates in

kbps. Besides, our evaluations use 140 network traces sampled from real network scenario. The average bandwidth of these network traces covers from 0.8Mbps to 2.5Mbps, while variance covers from 0.1Mbps to 2.0Mbps.

First, experiments are conducted to evaluate the accuracy of KAMA-based segment bitrate prediction by calculating the prediction error . The results show that it can reduce the prediction error to 0.22, against the prediction error of 0.258 when using the segment’s coding bitrate for prediction directly.

Then, we compare HYSA to the following adaptation schemes using the simulator mentioned previously: (1) HYSA-N: our baseline scheme where segment bitrate prediction is not included. (2) MPC(Yin et al., 2015): uses buffer occupancy and throughput predictions to select the bitrate which maximizes a given QoE metric over a horizon of five future chunks. (3) DTTB(Xie et al., 2017): selects video bitrate based on a dynamic buffer threshold adapted according to the estimated throughput. Table 1 provides the average value of overall QoE and other QoE metrics that each scheme achieves on the entire network traces and video traces. Figure 4

gives the Cumulative Distribution Function (CDF) of each network trace when streaming three types of videos. From these evaluations, we can easily draw two conclusions. Firstly, the average overall QoE is improved with the KAMA-based segment bitrate predictions rather than the coding bitrates, which demonstrates that the KAMA-based segment bitrate prediction contributes to making better decisions. Secondly, the proposed hybrid control scheme outperforms other schemes with respect to overall QoE when streaming video in different scenes under various networks, because it tends to choose higher quality to improve bandwidth utilization, as illustrated by Table


5. Conclusion

In this paper, we have presented HYSA – an effective hybrid control scheme consisting of heuristic playback rate control, latency-constrained bitrate control and QoE-oriented adaptive frame dropping. Our algorithm adopts Kaufman’s Adaptive Moving Average to predict the segment bitrates, with which we could make the bitrate decisions more accurately. Extensive simulation results have demonstrated that the segment bitrate prediction is advantageous in making better decisions, and HYSA can achieve higher overall QoE than the prior state-of-the-arts.

This work was supported in part by the National Natural Science Foundation of China (Grant No. 61971382)


  • Z. Akhtar, Y. S. Nam, R. Govindan, S. Rao, J. Chen, E. Katz-Bassett, B. Ribeiro, J. Zhan, and H. Zhang (2018) Oboe: auto-tuning video abr algorithms to network conditions. In Proceedings of the 2018 Conference of the ACM Special Interest Group on Data Communication, pp. 44–58. Cited by: §1.
  • M. Li, C. Yeh, and S. Lu (2018) Real-time qoe monitoring system for video streaming services with adaptive media playout. International Journal of Digital Multimedia Broadcasting 2018. Cited by: §1.
  • H. Mao, R. Netravali, and M. Alizadeh (2017) Neural adaptive video streaming with pensieve. In Proceedings of the Conference of the ACM Special Interest Group on Data Communication, pp. 197–210. Cited by: §1.
  • K. Miller, A. Al-Tamimi, and A. Wolisz (2017) QoE-based low-delay live streaming using throughput predictions. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) 13 (1), pp. 4. Cited by: §1.
  • Y. Shen, I. Marcin, J. Tabak, A. Kapoor, J. A. Villatoro, and J. Li (2018) Buffer reduction using frame dropping. Google Patents. Note: US Patent App. 10/015,224 Cited by: §1.
  • K. Spiteri, R. Urgaonkar, and R. K. Sitaraman (2016) BOLA: near-optimal bitrate adaptation for online videos. In IEEE INFOCOM 2016-The 35th Annual IEEE International Conference on Computer Communications, pp. 1–9. Cited by: §1.
  • L. Xie, C. Zhou, X. Zhang, and Z. Guo (2017) Dynamic threshold based rate adaptation for http live streaming. In 2017 IEEE International Symposium on Circuits and Systems (ISCAS), pp. 1–4. Cited by: §4.
  • X. Yin, A. Jindal, V. Sekar, and B. Sinopoli (2015) A control-theoretic approach for dynamic adaptive video streaming over http. In ACM SIGCOMM Computer Communication Review, Vol. 45, pp. 325–338. Cited by: §1, §4.