Cybersickness-aware Tile-based Adaptive 360° Video Streaming

In contrast to traditional videos, the imaging in virtual reality (VR) is 360, and it consumes larger bandwidth to transmit video contents. To reduce bandwidth consumption, tile-based streaming has been proposed to deliver the focused part of the video, instead of the whole one. On the other hand, the techniques to alleviate cybersickness, which is akin to motion sickness and happens when using digital displays, have not been jointly explored with the tile selection in VR. In this paper, we investigate Tile Selection with Cybersickness Control (TSCC) in an adaptive 360 video streaming system with cybersickness alleviation. We propose an m-competitive online algorithm with Cybersickness Indicator (CI) and Video Loss Indicator (VLI) to evaluate instant cybersickness and the total loss of video quality. Moreover, the algorithm exploits Sickness Migration Indicator (SMI) to evaluate the cybersickness accumulated over time and the increase of optical flow to improve the tile quality assignment. Simulations with a real network dataset show that our algorithm outperforms the baselines regarding video quality and cybersickness accumulation.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

11/07/2017

Viewport-aware adaptive 360° video streaming using tiles for virtual reality

360 video is attracting an increasing amount of attention in the context...
03/17/2020

Reinforcement Learning Driven Adaptive VR Streaming with Optical Flow Based QoE

With the merit of containing full panoramic content in one camera, Virtu...
04/11/2021

BurstLink: Techniques for Energy-Efficient Conventional and Virtual Reality Video Display

Conventional planar video streaming is the most popular application in m...
06/27/2019

An Algorithm for Transmitting VR Video Based on Adaptive Modulation

Virtual reality (VR) is making waves around the world recently. However,...
11/09/2017

Estimation of optimal encoding ladders for tiled 360° VR video in adaptive streaming systems

Given the significant industrial growth of demand for virtual reality (V...
05/10/2022

Evaluating the Impact of Tiled User-Adaptive Real-Time Point Cloud Streaming on VR Remote Communication

Remote communication has rapidly become a part of everyday life in both ...
06/12/2021

Evaluating Foveated Video Quality Using Entropic Differencing

Virtual Reality is regaining attention due to recent advancements in har...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

I Introduction

Expedited by the Covid-19 pandemic in 2020, much of the world has been transformed into collaboratively operating in a virtual environment [2]. It markedly accelerates the growth of virtual reality (VR) by necessity. For example, Facebook has launched Facebook Horizon to allow people to engage in a variety of activities like riding roller coasters with different themes. Users can play fast-paced shooting and car racing games with SteamVR.111Facebook Horizon: https://www.oculus.com/facebook-horizon/, SteamVR: https://store.steampowered.com/app/250820/SteamVR/ Equipped with VR head-mounted displays (HMDs), such as Oculus Rift, HTC VIVE, and Google Cardboard,222Oculus Rift: https://www.oculus.com, HTC VIVE: https://www.vive.com, Google Cardboard: https://arvr.google.com/cardboard a user can arbitrarily choose a view of the 360° video displaying the virtual world by moving her head dynamically. However, the bandwidth demand for streaming a 360° video is an order of magnitude larger than traditional static videos (e.g., movies) [21].

To reduce bandwidth consumption, tile-based streaming is proposed to stream a part of a video (i.e., video tiles) displayed [18, 11, 17, 14, 20]. It first cuts a 360° video into multiple tiles spatially and further encodes these tiles into different quality for adaptive streaming. Xie et al. [18]

chose tiles and decided their quality based on viewport distortion, quality variance, and buffer control. Qian

et al. [11] selected tiles according to the tile ranking and classing to maximize the utility function under the consideration of inter-chunk switch, intra-chunk switch, and stall duration. Xiao et al. [17] combined tiles into macro-streaming units and chose quality levels to reduce bandwidth waste. Sun et al. [14] dynamically adjusted the rate allocation and buffer length based on the network bandwidth statistics and viewport prediction. However, these works only focused on bandwidth reduction but did not explore cybersickness during the tile selection and buffer management in VR.

Cybersickness is akin to motion sickness, but it happens when using VR HMDs, and the user may have symptoms of headaches, nausea, and so on [8]. Since these feelings degrade the user’s experience with VR, it is vital to examine cybersickness during the tile selection in VR [21]. Several crucial factors lead to cybersickness. 1) User head movement. Faster head rotations incur more cybersickness [10]. When the user rotates head faster, the Motion-to-photon (MTP) delay increases due to the rapid change of frames. MTP delay is the time difference between the user’s head movement and the VR frame display (reflecting the user’s movement) [21]. 2) Magnitude of optical flow. Optical flow is the velocity distribution of apparent movements and changes in brightness patterns between two similar images. The fast motions and rapid changes of brightness and color of objects induce a large amount of optical flow. A VR scene with a large magnitude of optical flow triggers cybersickness because it contains massive visual information, amplifying the mismatch between the real world and the VR scene [4]. 3) Task duration. Longer task duration to use VR induces more cybersickness since cybersickness accumulates over time [10, 8, 15].

To alleviate cybersickness, field of view (FoV) shrinking [3, 5] and depth of field (DoF) simulation [1, 10] have been demonstrated as two promising approaches. FoV shrinking slightly and temporarily reduces the size of the viewport by masking the outer part of the original video [3, 5]. Smaller FoV induces less cybersickness with the side effect of lowering the sense of immersion, since it reduces the amount of visual information received. On the other hand, DoF simulation further blurs the out-of-focus parts of a video to decrease the magnitude of optical flow when the view changes rapidly [1, 10]. However, these works only verified the effectiveness of FoV shrinking and DoF simulation but did not explore the trade-off between cybersickness and video quality in mobile VR.

In this paper, we explore Tile Selection with Cybersickness Control (TSCC) in an adaptive 360° video streaming system. Unlike previous research of 360° video streaming ignoring the impact of cybersickness [17, 20, 18, 11, 14], TSCC minimizes the video quality loss and cybersickness accumulation by jointly fetching tiles, shrinking the user’s viewport, and determining the activation of DoF simulation. TSCC is challenging due to the following new research issues. 1) Trade-off between cybersickness and video quality. A high-quality video contains a large magnitude of optical flow, which induces severe cybersickness [4]. Hence, when users feel dizzy, it is necessary to constrain cybersickness by slightly limiting the viewport size with FoV shrinking and blurring out-of-focus tiles with DoF simulation to reduce the magnitude of optical flow. The joint configuration of the quality of video tiles (at different regions), FoV shrinking, and DoF simulation is crucial to VR. 2) Cybersickness accumulation. As the user keeps enjoying VR without pauses, cybersickness will aggravate over time since the accumulation is mostly faster than the speed of human adaptation to VR [8, 15]. However, activating FoV shrinking and DoF simulation for a long period will dramatically decrease the user’s experience with the video. Therefore, it is crucial to decide the activation and stop time of FoV shrinking and DoF simulation carefully. 3) Packet queue control. High-quality tiles consume much bandwidth and lead to longer transmission time, which may cause video stalls (i.e., the packet queue underflows). Most research [18, 14] controls the packet queue based on only the network bandwidth and aims to maximize video quality. However, it is necessary to examine user behaviors (e.g., head rotation) when controlling the packet queue in VR since they may induce higher cybersickness (which needs to be compensated by lowering video quality). Therefore, the packet queue management becomes much more complicated to avoid the overflow and underflow of the queue.

To address the above issues, we design an -competitive333 is the ratio of number of tiles chosen by ETSCAA to the optimal policy. online algorithm, named Effective Tile Selection and Cybersickness Alleviation Algorithm (ETSCAA). ETSCAA introduces Cybersickness Indicator (CI) and Video Loss Indicator (VLI) to evaluate the instant cybersickness and the total loss of video quality by carefully assessing the impacts of FoV shrinking and DoF simulation at different tiles. Equipped with CI and VLI, ETSCAA assigns tile quality according to the induced costs (including expected viewport distortion and cybersickness accumulation) and the available bandwidth to control the packet queue growth. Moreover, ETSCAA introduces Sickness Migration Indicator (SMI) to evaluate the cybersickness accumulated over time and the increase of optical flow. ETSCAA adjusts tile quality to jointly alleviate cybersickness and optimize video quality based on SMI.

The rest of this paper is organized as follows. Section II introduces the 360° video streaming system and Section III formulates TSCC. Section IV presents the online algorithm ETSCAA and proves its competitive ratio. Section V shows the simulation results, and Section VI concludes this paper.

Ii System Model

Following [21, 17, 20], a 360° video is first segmented into small chunks for storage. Next, each chunk is cropped into tiles spatially, and each tile is encoded into multiple bitrates by the video compression, such as H.264 or H.265, and stored in a mobile proxy server [21]. When a user request arrives, the server will combine the fetched tiles in the cache to render the required video frames, where the cached tiles were chosen from the server based on the user viewport prediction [21].

To alleviate cybersickness, the system is equipped with FoV shrinking [3, 5] and DoF simulation [1, 10]. After combining the tiles to form a complete video, the requested view is extracted from the video. The bokeh blur filter [1] is then applied to the viewport to create the DoF simulation, followed by slightly masking the upper and lower side of the viewport to shrink FoV. Afterward, the video quality degradation caused by FoV shrinking and DoF simulation can be evaluated according to structural similarity (SSIM) index and bitrate [6, 19].

The system includes a packet queue and a sickness queue maintained on the user side. The packet queue stores the video rendered from the fetched tiles for smooth playback444To avoid cybersickness caused by MTP delay, it is necessary to maintain the occupancy of packet queue (detailed later) [18, 21]. [18, 11, 17, 14], and the queue length increases and decreases when new frames are downloaded and the video is played, respectively. The sickness queue (derived according to the Time-varying Cybersickness Model [8, 15, 4, 1, 3, 5, 10] and detailed in Section III) records cybersickness accumulation when the video is played. According to [10, 8, 15], we measure cybersickness severity by the Simulator Sickness Questionnaire (SSQ) metric, which can be calculated by the magnitude of optical flow and user head rotation speed, monitored by eye and neck movement tracking schemes in VR HMDs [10]. The queue length increases when the magnitude of optical flow and user head rotation speed increase, whereas it constantly decreases due to the user adaptation to VR. The packet queue and the sickness queue affect each other when the available bandwidth, user behavior, or the amount of video optical flow changes.

Iii Problem Formulation

Due to the space constraint, the notation table is provided in [7]. We formulate TSCC for the tile selection and cybersickness control in a 360° video streaming system. Specifically, the system includes a 360° video divided into multiple chunks with the duration [20], and each chunk is spatially cropped into tiles [20], where each tile is encoded into quality levels [20]. Let represent tile with quality . Each tile induces bandwidth consumption , distortion , and the magnitude of optical flow , which are specified in the video metadata [18]. We describe the system status at time slot as , where , , , and are the status of user head rotation, expected viewport distortion, packet queue occupancy, and sickness queue occupancy at time slot , respectively, introduced as follows.

1) The status of user head rotation includes the angular speed of yaw rotation and the angular speed of pitch rotation [18, 11, 17, 14]. Let

be the viewing probability of tile

at time slot , which can be derived by the viewport prediction with the stochastic model and historical data of user head rotation [18, 11].

2) The expected viewport distortion [18] can be expressed as the average video quality distortion represented by the distortion of each tile weighted by its viewing probability at time slot , where is a decision variable indicating whether tile is fetched. The distortion of each tile is derived as the reciprocal of its SSIM index [6], and a tile with higher quality has smaller distortion [6, 19].

3) Let be the bandwidth,555

The bandwidth can be allocated by exploiting machine learning models, such as support vector machine and artificial neural network, that learn the relation between bandwidth allocation and quality of service

[13]. denotes the packet queue capacity,666Due to the short-term constraint of viewport prediction, tile-based streaming keeps a small packet queue capacity, whose unit is second, to store predicted viewports in the next few seconds [18, 20]. and is the bandwidth consumption of the viewport. Since the packet queue length decreases when the download time of video chunk is longer than its duration (i.e., the video consumption rate is faster than download speed), following [18], the packet queue occupancy . is the current occupancy, and represents the amount of occupancy change.777

The decreasing occupancy estimated by the download time is subtracted from the increasing occupancy

evaluated by the video chunk duration.
is a decision variable indicating whether DoF simulation is activated, and is the ratio of the bandwidth consumption reduced by DoF simulation. Following [3, 5], is a decision variable representing the ratio of the shrunken viewport size to the original size when activating FoV shrinking.

4) When the user enjoys VR over time, cybersickness accumulates and the length of the sickness queue increases due to the head rotation [10, 3] and optical flow [4]. However, the user may gradually adapt to VR as time elapses. Let be the consumption rate of the sickness queue (i.e., user adaptation to VR).888According to [9], the values of and for each user can be determined based on age, gender, and familiarity with VR contents. For example, seniors and people that are not familiar with VR contents tend to have a lower tolerance of cybersickness, and smaller and are required. Following [8, 15], the sickness queue occupancy is derived according to i) , ii) increasing occupancy caused by head rotation [10, 3] and optical flow [4], and iii) decreasing occupancy . That is, , where is the sickness queue capacity,footnotemark: and the optical flow of each tile is weighted by its viewing probability. is the normalized head rotation speed [11]. describes the percentage of cybersickness decreased due to FoV shrinking and DoF simulation.999The cybersickness reduction is inversely proportional to the viewport size [5] and the degree of video blurring [1].

TSCC has the following constraints. 1) Fetching constraint. The system fetches only one level for each tile at any instant to avoid tile redundancy, i.e., . 2) Packet queue length constraint. To prevent video rebuffering and control MTP delay, the queue occupancy must be no less than the targeted queue occupancy , i.e., [18, 14]. By properly reformulating the above equation (detailed in [7]), we can transform the packet queue length constraint into the bandwidth constraint for fetching tiles, i.e., . The objective function includes two parts: 1) video quality loss and 2) cybersickness accumulation (i.e., sickness queue occupancy). The video quality deteriorates if the viewport distortion is larger [18] and the viewport size (narrowed down by FoV shrinking) becomes smaller [5]. Also, DoF simulation undermines video quality since it blurs the out-of-focus parts of video [1, 10]. Accordingly, following [18], the video quality loss is . Therefore, the objective is , where and are tuning knobs to trade-off video quality against cybersickness.101010For the videos with highly dynamic contents (e.g., action movies), we can set a larger to focus more on cybersickness since they contain a large amount of optical flow and aggravate cybersickness more easily. For static videos like dramas, we can focus more on video quality and set a larger . TSCC is formulated as follows.

Definition 1 (Tscc).

Given the duration of each video chunk cropped into tiles with quality levels in a frame, video metadata (,,), packet queue occupancy , sickness queue occupancy , and bandwidth , TSCC finds 1) the fetched tiles and their quality, 2) the size of viewport (configured by FoV shrinking), and 3) the activation of DoF simulation, under the 1) fetching, 2) packet queue length, and 3) bandwidth constraints. The objective is to minimize the cost function of video quality loss and cybersickness accumulation over time, i.e., .

Iv Online Algorithm

To minimize the video quality loss, an intuitive approach is to classify the tiles into multiple groups with different priorities according to estimated viewing probability, and a group is assigned a higher quality if it has a higher priority

[17, 11, 14]. However, the above approach does not use FoV shrinking and DoF simulation to alleviate cybersickness. We design an -competitive online algorithm ETSCAA to minimize the long-term video quality loss and cybersickness accumulation by examining Cybersickness Indicator (CI), Video Loss Indicator (VLI), and Sickness Migration Indicator (SMI) in every time slot. CI and VLI respectively evaluate the instant cybersickness and the video quality loss. Equipped with CI and VLI, ETSCAA assigns the tile quality by evaluating the cost of the joint configuration of quality assignment, FoV shrinking, and DoF simulation for each tile in the viewport. When the packet queue overflows, it tends to choose high-quality tiles (requiring a longer transmission time) to slow down packet queue growth, and the faster cybersickness accumulation (due to a larger magnitude of optical flow) also needs to be alleviated by FoV shrinking and DoF simulation. Moreover, ETSCAA introduces SMI to jointly adjust tile quality and control the growth of sickness queue by evaluating the accumulated cybersickness and the increased optical flow. ETSCAA includes three phases: 1) Viewport Prediction and Tile Selection (VPTS), 2) Tile Quality Assignment (TQA), and 3) Cybersickness and Tile Quality Control (CTQC). The pseudocode of ETSCAA is presented in [7] due to the limited space.

Iv-a Viewport Prediction and Tile Selection (VPTS)

To adapt to user head rotation, VPTS derives the viewing probability for all tiles by the following steps: 1) VPTS first predicts the most likely viewport position in the next seconds. Specifically, let be the viewport position at time slot , where and are the horizontal (yaw) and vertical (pitch) angles of the viewport, respectively. VPTS evaluates the viewport position at time slot by applying ° of yaw rotation and ° of pitch rotation from the current viewport position. The predicted viewport position is

. 2) VPTS extracts adjacent viewports to ensure that the fetched tiles can fully cover the user’s view during the head rotation, based on the Gaussian distributions of yaw-axis and pitch-axis with the standard deviations

and , respectively [18]. 3) For each tile , VPTS sums the viewing probability of each predicted viewport position that overlaps with tile to derive . That is, , where is the set of all the predicted viewport positions that overlap with tile . VPTS then constructs a set of fetched tiles and examines the boundary subsets of to identify the tiles unlikely to be seen. Let be the boundary subset of formed by a row (column) of tiles111111Since the viewport, composed of multiple tiles, needs to be rectangular [21], a row (column) of tiles in a set is defined as the tiles sharing the same horizontal (vertical) coordinate, such as tiles in Fig. 1. in . If holds, we remove from , where is the probability threshold to discard tiles.121212Following [14, 20], a smaller allows the mobile proxy to transmit a larger viewport to avoid the interruption caused by head rotation. For example, in Fig. 1, the boundary subset removed is and the fetched tiles are . Due to the space constraint, the detailed calculation of this and the remaining examples are provided in [7].

Fig. 1: An example of VPTS

Iv-B Tile Quality Assignment (TQA)

TQA assigns a quality level to each tile in based on the induced cost of the tile quality assignment and the bandwidth constraint. We introduce Cybersickness Indicator (CI) and Video Loss Indicator (VLI), weighted by the viewing probability , to approximately evaluate the cybersickness and video quality loss. A tile with more visual information has a larger size and more optical flow, which induces cybersickness. To alleviate cybersickness, FoV shrinking and DoF simulation reduce the size and resolution of the viewport such that the visual information (video size) decreases. Therefore, we define to measure the cybersickness induced by each tile regarding the effect of optical flow , FoV shrinking , and DoF simulation .

The video quality becomes worse when the distortion is larger and the viewport size and resolution are smaller (configured by FoV shrinking and DoF simulation). Hence, the video quality drops by the reciprocal of . We define to measure the video quality loss induced by each tile regarding the effect of distortion , FoV shrinking , and DoF simulation (). Therefore, the cost of assigning quality to tile in with a certain configuration of FoV shrinking and DoF simulation 131313We iteratively examine each configuration of , and the number of iterations is small since and there are usually levels of [3, 5]. can be evaluated by . The function trades off the video quality against cybersickness.

The optimal quality of each tile may not be identical because the distortion and magnitude of optical flow of each tile vary. A tile with more visual information has a larger difference in the magnitude of optical flow between adjacent quality levels. Therefore, to jointly minimize the video quality loss and cybersickness accumulation, our idea is to iteratively derive the dynamic programming state of each quality assignment with the joint configuration of FoV shrinking and DoF simulation. Let be the minimum total cost of the quality assignment for the first tiles in under the bandwidth constraint . TQA systematically derives according to the following two cases. 1) When (i.e., no tile needs to be assigned quality), the total cost is . 2) When , TQA considers different quality assignments, which consume bandwidth no more than , for tile . For each tile , TQA recursively examines , which is the minimum total cost of the quality assignment before assigning quality to . Then, it chooses the minimum to minimize the video quality loss and cybersickness accumulation. Accordingly,

(1)

Recall that the total bandwidth for fetching tiles cannot exceed to control the growth of the packet queue (detailed in Section III), and we let . Hence, we can obtain the minimum total cost of the quality assignment for tile set by calculating . Based on (1), TQA extracts a quality assignment to minimize the video quality loss and cybersickness accumulation. TQA first constructs a table to store the bandwidth of each tile. Specifically, TQA stores the bandwidth of tile into the table with a label when it examines during the recursive process. Afterward, TQA iteratively queries the table to construct since the bandwidth of each tile can be mapped to a certain quality and TQA can obtain the bandwidth of by querying the value labeled .

Example 1. Recall that we have obtained in VPTS. Assume that , , , , , and all possible configurations of are , , , and , which results in , , , and , respectively. When , let , , , and . Then, . After the recurrence, the quality assigned to the tile set is , since it induces the minimum total cost . Similarly, for , , and , we have , , and as the quality levels of tile set , respectively.

Iv-C Cybersickness and Tile Quality Control (CTQC)

To alleviate cybersickness accumulated over time, CTQC iteratively adjusts the tile quality assignment to slow down the growth of the sickness queue. When the cybersickness has accumulated massively, even slightly increasing optical flow can lead to the sickness queue overflow (i.e., the user will immediately quit watching 360° videos since the symptoms of cybersickness become unbearable [8, 15]). On the other hand, when the sickness queue occupancy is small, higher tile quality (which leads to more optical flow) can be used to reduce the video quality loss. Therefore, we introduce the Sickness Migration Indicator integrated with VLI to jointly examine cybersickness and video quality, where is the set of quality levels assigned to tile set , is the optical flow of tile at time slot (obtained from the video metadata). SMI evaluates the impact of accumulated cybersickness () and future optical flow on the tile quality assignment, and increases when becomes larger [4].

To adjust the tile quality, we define , a neighbor of , as a quality assignment (satisfying the bandwidth constraint) that only one quality is different from and the level difference is . For example, is a neighbor of if or . In the beginning, we let be the quality assignment obtained in TQA and regard it as the center set for searching its neighbors to adjust tile quality. To avoid re-examining the same assignment in the next few iterations and improve the searching efficiency, CTQC maintains a neighbor search list (NSL) and inserts into it.141414The size of NSL can be adjusted based on the number of neighbors. CTQC first finds the neighbor (of the center set ), which induces the minimum SMI among the neighbors and is not in the NSL, as the center set in the next iteration. Then, CTQC inserts into the NSL to avoid re-evaluating it in the next few iterations. If the list is full, CTQC will remove the oldest assignment. If induces a smaller SMI than , CTQC will replace by . CTQC iteratively examines the above process until a certain quality assignment has been examined for times, where can be adjusted to prevent excessive computation time.

Example 2. Following Example 1, when , , , and , we have , , , and as the quality levels of tile set , respectively. Let and its neighbor . When , the result of CTQC is since it has the minimum SMI. As for the other pairs, the quality assignments obtained from TQA remain the same in CTQC.

Last, ETSCAA repeats TQA and CTQC with all possible FoV shrinking and DoF simulation configurations to minimize the video quality loss and cybersickness accumulation by selecting the tile quality assignment inducing the minimum . Following Example 2, after repeating TQA and CTQC for , , , and , we choose with since it induces the minimum .

Theorem 1.

ETSCAA is -competitive in time, where is the minimum SSIM of all tiles, is the minimum value of , and .

Proof.

The proof is presented in [7] due to the limited space. ∎

Theorem 2.

ETSCAA is -competitive if .

Proof.

The proof is presented in [7] due to the limited space. ∎

V Simulations

Due to the space constraint, some results and the user study are presented in [7].

V-a Simulation Setup

We evaluate the performance of ETSCAA with the video metadata extracted from 4K 360° videos [16]. The network bandwidth traces are extracted from the dataset HSDPA [12], and the mean value of the bandwidth is scaled from 3Mbps to 11Mbps [20]. Following [18, 20], we cut the video into multiple chunks with the same duration of 1 second and split each chunk spatially into tiles in default. Each tile includes 5 quality levels [18, 11, 20] and is encoded with H.264 using Constant Rate Factor (CRF) 43, 38, 33, 28, and 23 [11] for level 1 to 5, respectively. We exploit the viewport prediction technique in [18] by formulating the prediction error with Gaussian distributions. The original viewport size is set to 100°100°in default [20]. Following [18], the capacity of packet queue is set to s and the targeted queue occupancy is set to . is set to 0.1 based on the result of applying DoF simulation to the videos [1, 10]. For the sickness queue, we follow [8] to set the capacity to with the user adaptation to VR set to in default. The default settings of and  are 1 and 2.5, respectively. We compare ETSCAA with three tile selection algorithms for 360° video streaming systems: 1) EPASS360 [20], 2) BAS [17], and 3) Flare [11]. To evaluate ETSCAA, we change the following parameters: 1) network bandwidth and 2) user adaptation to VR (), and we measure the performance metrics: 1) total cost, 2) video quality loss, 3) sickness queue occupancy, and 4) SSIM. Each result is averaged over 100 samples, and each sample includes 1,000+ video downloads.

(a)
(b)
(c)
(d)
(e)
(f)
Fig. 2: Performance evaluation with different parameters.

V-B Simulation Results

In Fig. 2(a), the total cost increases as the bandwidth grows since more tiles with higher quality (i.e., less distortion) are chosen, which leads to more accumulated cybersickness (see Fig. 2(c)). When the bandwidth is sufficient, ETSCAA tends to choose high-quality tiles to reduce the video quality loss. However, it examines CI and VLI to activate FoV shrinking and DoF simulation to slightly mask and blur the outer part of the video for cybersickness alleviation. Therefore, ETSCAA generates a much smaller sickness queue occupancy by degrading only a little video quality in Fig. 2(c). Comparing Figs. 2(b) and 2(c), the baselines lead to a smaller video quality loss but much higher sickness queue occupancy than ETSCAA, because they maximize video quality by choosing the tiles with higher quality and viewing probability. In contrast, ETSCAA exploits CI and VLI to carefully examine FoV shrinking and DoF simulation at different tiles, and they only sacrifice a little video quality (no more than SSIM [6]) in Fig. 2(d) to achieve better cybersickness alleviation in Fig. 2(c). Figs. 2(e) and 2(f) explore the impact of user adaptation to VR (i.e, ). The total cost and sickness queue occupancy plummet as increases since users can adapt to VR more easily. In contrast to the baselines maximizing the video quality, ETSCAA evaluates the change of sickness queue occupancy over time by SMI to deal with the trade-off between cybersickness and video quality. Moreover, it exploits NSL to avoid re-evaluating recent tile quality configurations to ensure a better quality assignment in the next iteration. In general, ETSCAA results in a smaller total cost and can reduce the cybersickness by more than compared to the baselines.

Vi Conclusion

This paper leverages FoV shrinking and DoF simulation to alleviate cybersickness in a 360° video streaming system. We formulate a new optimization problem TSCC to minimize the video quality loss and cybersickness accumulation. To effectively solve TSCC, we propose an -competitive online algorithm ETSCAA with CI, VLI, and SMI to support tile selection and cybersickness control. Simulations with a real network dataset demonstrate that ETSCAA can reduce more than 25% of sickness queue occupancy against the state-of-the-art tile selection algorithms for 360° video streaming systems.

Acknowledgment

We thank to National Center for High-performance Computing (NCHC) of National Applied Research Laboratories (NARLabs) in Taiwan for providing computational and storage resources.

References

  • [1] K. Carnegie and T. Rhee (2015) Reducing visual discomfort with HMDs using dynamic depth of field. IEEE Comput. Graphics Appl. 35 (5), pp. 34–41. Cited by: §I, §II, §II, §III, §V-A, footnote 9.
  • [2] (2020)(Website) External Links: Link Cited by: §I.
  • [3] A. S. Fernandes and S. K. Feiner (2016) Combating VR sickness through subtle dynamic field-of-view modification. In Proc. IEEE 3DUI, Cited by: §I, §II, §II, §III, §III, footnote 13.
  • [4] J. Kim and T. Park (2020) The onset threshold of cybersickness in constant and accelerating optical flow. Applied Sciences 10 (21). External Links: Document Cited by: §I, §I, §II, §III, §IV-C.
  • [5] S. Kim et al. (2018) An effective FoV restriction approach to mitigate VR sickness on mobile devices: an effective approach to mitigate VR sickness. J. Soc. Inf. Disp. 26. Cited by: §I, §II, §II, §III, §III, footnote 13, footnote 9.
  • [6] Y. Li and X. Mou (2021) Joint optimization for SSIM-based CTU-level bit allocation and rate distortion optimization. IEEE Trans. Broadcast. (), pp. 1–12. External Links: Document Cited by: §II, §III, §V-B.
  • [7] C. Lin, C. Wang, D. Yang, and W. Liao (2021) Cybersickness-aware tile-based adaptive 360° video streaming (full version). Technical report National Taiwan University. External Links: Link Cited by: §III, §III, §IV-A, §IV-C, §IV-C, §IV, §V.
  • [8] D. N., S. P., and S. A. (2018) Can simulator sickness be avoided? a review on temporal aspects of simulator sickness.. Frontiers in psychology. Cited by: §I, §I, §II, §III, §IV-C, §V-A.
  • [9] K. Petri, K. Feuerstein, S. Folster, and F. Bariszlovich (2020-11) Effects of age, gender, familiarity with the content, and exposure time on cybersickness in immersive head-mounted display based virtual reality. American Journal of Biomedical Sciences, pp. 107–121. External Links: Document Cited by: footnote 8.
  • [10] T. M. Porcino, E. W. G. Clua, D. G. Trevisan, C. N. Vasconcelos, and L. Valente (2017) Minimizing cyber sickness in head mounted display systems: design guidelines and applications. Proc. IEEE SeGAH. Cited by: §I, §I, §II, §II, §III, §III, §V-A.
  • [11] F. Qian, B. Han, Q. Xiao, and V. Gopalakrishnan (2018) Flare: practical viewport-adaptive 360-degree video streaming for mobile devices. In Proc. ACM Mobicom, Cited by: §I, §I, §II, §III, §III, §IV, §V-A.
  • [12] H. Riiser et al. (2013) Commute path bandwidth traces from 3g networks: analysis and applications. In Proc. ACM MMSys, Cited by: §V-A.
  • [13] L. Ruan, M. P. I. Dias, and E. Wong (2020) Enhancing latency performance through intelligent bandwidth allocation decisions: a survey and comparative study of machine learning techniques. IEEE/OSA J. Opt. Commun. Networking 12 (4), pp. B20–B32. Cited by: footnote 5.
  • [14] L. Sun et al. (2019) A two-tier system for on-demand streaming of 360 degree video over dynamic networks. IEEE J. Emerging Sel. Top. Circuits Syst. 9 (1), pp. 43–57. Cited by: §I, §I, §II, §III, §III, §IV, footnote 12.
  • [15] G. Wang and A. Suh (2019) USER adaptation to cybersickness in virtual reality: a qualitative study. In Proc. ECIS, Cited by: §I, §I, §II, §III, §IV-C.
  • [16] C. Wu, Z. Tan, Z. Wang, and S. Yang (2017) A dataset for exploring user behaviors in vr spherical video streaming. In Proc. ACM MMSys, Cited by: §V-A.
  • [17] M. Xiao, C. Zhou, V. Swaminathan, Y. Liu, and S. Chen (2018) BAS-360°: exploring spatial and temporal adaptability in 360-degree videos over HTTP/2. In Proc. IEEE INFOCOM, Cited by: §I, §I, §II, §II, §III, §IV, §V-A.
  • [18] L. Xie, Z. Xu, Y. Ban, X. Zhang, and Z. Guo (2017) 360ProbDASH: improving QoE of 360 video streaming using tile-based HTTP adaptive streaming. In Proc. ACM Multimedia, Cited by: §I, §I, §II, §III, §III, §III, §III, §III, §IV-A, §V-A, footnote 4, footnote 6.
  • [19] H. Yuan, Q. Wang, Q. Liu, J. Huo, and P. Li (2021) Hybrid distortion-based rate-distortion optimization and rate control for h.265/HEVC. IEEE Trans. Consum. Electron. (), pp. 1–1. Cited by: §II, §III.
  • [20] Y. Zhang et al. (2020) EPASS360: QoE-aware 360-degree video streaming over mobile devices. IEEE Trans. Mob. Comput. (), pp. 1–15. Cited by: §I, §I, §II, §III, §V-A, footnote 12, footnote 6.
  • [21] M. Zink, R. Sitaraman, and K. Nahrstedt (2019) Scalable 360° video stream delivery: challenges, solutions, and opportunities. Proceedings of the IEEE 107 (4), pp. 639–650. Cited by: §I, §I, §II, footnote 11, footnote 4.