I Introduction
Energy harvesting can provide renewable free energy supply for wireless communication networks. With the help of solar cells, thermoelectric and vibration absorption devices, and the like, communication devices are able to gather energy from surrounding environments. Energy harvesting can also help reduce carbon emission and environmental pollution, as well as the consumption of traditional energy resources [1, 2, 3]. In practice, harvested energy arrives in small units at random times and the storage battery usually has limited capacity [4]. Hence, wireless communication systems exclusively powered by energy harvesting devices may not guarantee the users’ quality of service. To provide dependable communication service, reliable energy resources can serve as backup in the case of energy shortage. In this way, efficient mixed usage of the harvested energy and reliable energy provides a key solution to robust wireless green communications [5], an emerging area of critical importance to future wireless development.
In wireless networks, energy efficient transmission has been an everpresent important issue [6, 7, 8]. Subject to the randomness and causality of energy harvesting, the optimal transmission problem has been investigated for an energy harvesting wireless link with batteries of either finite or infinite capacity in [4, 9, 10]. In these works, the authors assumed that the energy harvesting profile (, the arrival times and associated amount of harvested energy) is known before the transmission starts. This line of work has been extended to wireless fading channels [11], broadcast channels [12] and twohop networks [13].
Some other recent works have focused on developing efficient transmission and resource allocation algorithms with different objectives and energy recharging models. For example, a savethentransmit protocol was proposed in [14]
to minimize the delay constrained outage probability by using two alternating batteries, where the battery charging rate is modeled as a random variable. In
[15], a crosslayer resource allocation problem was studied for wireless networks powered by rechargeable batteries, where the amount of replenished energy is assumed to be independent and identically distributed in each time slot. In [16], an optimal energy allocation problem was studied for a wireless link with time varying channel conditions and energy sources. A line of work pertinent to our study focuses on the queueing performance analysis for optimal energy management policies. In particular, different sleep/wakeup strategies in a solarpowered wireless sensor network were studied in [17]. Energy management policies were proposed in [18] to maximize the stable throughput and minimize the mean delay for energy harvesting sensor nodes.While a node can harvest an infinite amount of energy in the long run, harvested energy actually arrives at random times. Due to the energy causality constraint, the node should accumulate a sufficient amount of energy before each packet transmission. Hence, the waiting time could be undesirably long and some packets might be dropped due to delay violation. Intuitively, this situation can be greatly relieved if one Reliable Energy Source (RES) can be used to transmit backlogged packets when needed. At the other extreme, the problem becomes trivial if the system can always transmit using the reliable energy. Hence, there exists a tradeoff between the packet delivery delay and the energy consumption from the reliable source.
In this paper, we investigate the delay optimal scheduling policy for a communication link powered by a capacitylimited battery storing harvested energy and one RES. In our system, the source will first seek energy supply from the capacity limited energy harvesting battery whenever available, and resort to the RES when necessary, but with an average power constraint. In particular, subject to the bursty energy harvesting profile, it transmits with one of the energy supplies according to the data queue status and the energy storage status at the battery. Under the constraint of the average power consumption from the RES, we study the delay optimal scheduling problem, taking into account the match and mismatch between the energy harvesting capabilities and data packet arrival.
To analyze the proposed scheme, we formulate a twodimensional discretetime Markov chain and derive the steadystate probabilities. Based on the Markov chain modeling, we can derive the average delay and the average power consumed from the RES as functions of the steadystate probabilities. Then, by formulating a Linear Programming (LP) problem and analyzing its properties, we are able to characterize the structure of the optimal solution. Moreover, we can obtain an elegant closedform expression for the optimal solution in the case where each unit of harvested energy can support one data packet transmission. We also develop an algorithm to find the optimal solutions in other cases. From the optimal solution, we can determine the optimal probabilistic transmission parameters. It is found that in the face of a depleted battery, the optimal transmission strategy depends on a critical threshold for the data queue length. In particular, the source relies on the harvested energy supply if the data queue length is below the critical threshold, and resorts to the RES otherwise. Our theoretical analysis is verified by simulations.
The rest of this paper is organized as follows. Section II introduces the system model and the stochastic scheduling scheme. In Section III, a twodimensional Markov chain model is constructed for the data and energy packet queueing system. Section IV formulates an LP problem for our scheduling objective. By analyzing the properties of the LP problem, we derive the optimal steadystate probabilities and then determine the optimal transmission parameters in Section V. Section VI demonstrates the simulation results and Section VII concludes this paper. For better illustration and in the interest of space, most proofs for our results are put in the appendices.
Ii System Model
Iia System Description
We consider a communication link which is powered mainly by a battery storing the harvested energy and further by the RES when necessary, as shown in Fig.1. The RES refers to any reliable energy source, either traditional (such as power grid) or newly developed. The source node (e.g. base station) employs a buffer to store the backlogged packets randomly generated from higherlayer applications. Suppose that the data packets arrive at the source buffer according to a Bernoulli arrival process [19] with probability . This simple yet widely adopted traffic model allows tractable analysis, and provides insights for further study. The system is assumed to be timeslotted, and at the beginning instant of each slot, data packets arrive at the data queue with capacity . In this work, is treated as sufficiently large (so no data overflow will incur) and fixed. Let be the length of the data queue at the end of slot , updated as
(1) 
where and denote the number of data packets arriving and served in each time slot , respectively. Without loss of generality, it is assumed that at most one packet is transmitted in each slot due to the capacity limitation of the communication link. Extension to multipacket transmission will be considered in future work.
The harvested energy is generally sporadically and randomly available, and we adopt a probabilistic energy harvesting model similar to [20]. Assume that Joule harvested energy arrives at the beginning of a time slot with probability , which can be used to transmit packets. That is , where (Joule) denotes the amount of energy needed for transmission of one data packet, and is rounded down to the nearest integer. We will consider several interesting combinations of and in this study, and leave the case to future study. The harvested energy is stored in the battery with the maximum capacity Joule, and discarded when the battery is full. The battery storage is modeled as an energy queue with a finite capacity , where one unit of transmission energy is viewed as one energy packet. Let and be the number of energy packets received and consumed in each slot , respectively. At the end of time slot , the length of the energy queue is updated as
(2) 
It is assumed that the packet and energy arrival processes are independent, and the newly harvested energy can be used for data transmission in the same slot. For notational convenience, we set to be the buffer status in the time slot
. Similarly, the arrival and service processes can be characterized by the vectors
and , respectively.IiB Stochastic Scheduling
As we mentioned above, the source node is encouraged to exploit the harvested energy whenever available, and resort to the backup RES when necessary. To this end, the source should always transmit using the energy stored in the battery or newly arriving energy packet when possible, which corresponds to the case or . When the harvested energy is not available, , the source schedules the transmission of data packets with the RES energy according to the data queue status and the data packet arrival status . For generality, we define two sets of parameters: and in our scheduling scheme. In particular, with , if there is new data packet arrival in this slot, , , the source node transmits one data packet with probability with the RES energy and holds from transmission with probability , respectively; If no new data packet arrives, , , it transmits with probability and holds with probability , respectively. As discussed later, these parameters and shall be optimized to achieve the minimum average queueing delay in different cases.
According to the proposed scheduling policy, the service process depends on the queue status and the arrival process , as described below.

Case 1: ()
In this case, the source can transmit a newly arriving data packet using the harvested energy from the battery in the current time slot , and the service process can be expressed as(3) where means ’with the probability’. The notation of is used to denote both and .

Case 2: (, )
In this case, the source can transmit a backlogged packet with the harvested energy. The service process is expressed as(4) 
Case 3:
In this case, when both data and energy packets arrive, the source will transmit with the energy harvested; When new data packets arrive in the absence of energy harvesting, the source shall use the energy from the RES to transmit with probability . Hence, the service process can be expressed as(5) 
Case 4: ()
In this case, the source will transmit definitely using the harvested energy if it is available in the current slot . Otherwise, it will transmit using the RES energy with probability if and with probability if , respectively. The service process is characterized as(6)
The above four cases include all possible scenarios.
IiC Average Delay and Power Consumption
In a queueing system, the average queuing delay is an important metric [21]. From the above description, the queueing system can be modeled as a discretetime Markov chain, where each state represents the buffer status. Let be the state that the data queue length is and the energy queue length is , and denote the steadystate probability of state . By the Little’s law, the average queueing delay is related to the average buffer occupancy, and can be computed as
(7) 
where .
The average transmission power is also an important performance metric in wireless green communication systems. In this work, we focus on the average power consumption from the RES. Denote by the power consumed in the th time slot. If the source transmits using the energy from the RES in time slot , , where denotes the transmission time. Otherwise, . As will discussed below, the source draws one energy packet from the RES depending on the current queueing status . Let denote the probability that the power consumption is equal to conditioned on the queue state
. Using the law of total probability, we obtain the normalized average power consumption (with respect to
) as(8) 
where is the set of states conditioned on which the source may draw the RES energy to transmit one data packet. This normalized quantity can be interpreted as the proportion of the number of time slots in which the source transmits using the power from the RES. From (7) and (8), both the average queueing delay and power consumption are functions of the steadystate probabilities. In this work, we aim to study the delay optimal scheduling policy which minimizes subject to the average power constraint by determining the optimal transmission parameters and . As a key step, we will develop twodimensional Markov chain models for different combinations of and in the next section.
Iii Twodimensional Markov Chain Modeling
To analyze the proposed scheduling scheme, we formulate a twodimensional discretetime Markov chain for the queueing system, as shown in Fig.2.
Let denote the onestep transition probability of the Markov chain, which is homogeneous by the scheme description. For ease of expression, we define four constants as
(9) 
We further define two subsets of as: , , and set , for .
We now describe the onestep transition probabilities in Fig.2(a) in detail, by grouping them into several types. We start with the four transitions among each square unit, for example, those among , , and in Fig.2(a). First, let us examine the transition from to , more generally, from to . This corresponds to the case that there is no data but energy packet arrival, and one backlogged data packet is delivered, so clearly the corresponding probability is When neither data nor energy packets arrive, one data packet stored in the buffer can be transmitted using one energy packet from the battery if there exists. In this case, the state will transfer from to (e.g., from to in Fig.2(a)) with probability for all and . When data packets arrive while no energy is harvested, one data packet will be transmitted using one energy packet if there is energy stored in the battery. That is, the state will transfer from to (e.g., from to in Fig.2(a)) with probability for all . When data and energy packets arrive simultaneously, one data packet is transmitted using one energy packet. In this case, the state will transfer from to (e.g., from to in Fig.2(a)) with probability for .
The case requires special treatment, as the battery is full and the newly harvested energy has to be discarded anyway. With the capacity limit in mind, we have for , , and for all .
We then consider the first row in Fig.2(a). When no data packets arrive and energy packets newly arrive, the state will transfer to with the corresponding transition probability for . We have mentioned that is due to the capacity limitation of the battery. The state remains the same with probability (when neither data nor energy packets arrive).
We now focus our attention on the group of transition probabilities on the first column of Fig. 2, and (), and (), which corresponds to the case that there is no storage of harvested energy in the battery, and can be obtained as
(10) 
In particular, when data packets arrive while no energy is harvested (which happens with probability ), and denote the transition probabilities from state to and , respectively, depending on whether one data packet is delivered with the reliable energy in this slot (with probability ). When neither data nor energy packets arrive (which happens with probability ), and denote the transition probabilities from state to and , respectively, depending on whether one data packet is transmitted using the reliable energy (with probability ).
We order the states as , , , , , , , , , and let denote the transition matrix. We denote by the column vector containing steadystate probabilities, and by the column vector with all the elements equal to one. For notational convenience, we also define two subvectors of as: and , and denote by a row vector with all the elements equal to one. Given a set of parameters and , the steadystate probabilities can be obtained by solving the linear equations and . Note that the transmission parameters and only influence the transition probabilities from the states , . We thus consider , a submatrix of , to exclude the state transition starting from states . In this way, present the local balance equations at the states . For ease of expression, we also denote by the lefttop submatrix of dimensions from .
In the general case with and , the corresponding Markov chain seems not amenable to analysis. In this paper, we mainly focus on three cases: Case I with and , Case II with and , and Case III with and , respectively. These three exemplary cases nonetheless capture some important relations between the data and energy arrival processes, and serve as the basis for further extensions. In the following, we illustrate the Markov chain for each of the three cases.
Iiia Case I: and
In this case, one data packet and one energy packet arrive in each slot with probabilities and , respectively. Accordingly, the simplified Markov chain is shown in Fig.2(b). Essentially all expressions in the general case carry over with the substitution of . For example, the transition from to in Fig.2(a) becomes that from to in Fig.2(b), again with probability . This applies to the states in the first column as well, and as a result, a new notation is needed for the transition from to , which combines and the previous :
(11) 
for all . Also, it is worth noting that in the dashed square, neither queue length can ever increase regardless of the arrival processes, as one data packet transmission happens for sure. As a result, the states with are transient in the following lemma.
Lemma 1.
In Case I with when or , the queue status satisfying is transient.
Proof:
This implies that either the data queue or the energy queue will be exhausted, even if they are not empty initially. Hence, when calculating the steadystate probabilities , the twodimensional Markov chain can be reduced to the onedimensional one, as plotted in Fig.2(c), which consists of the states and for all and .
IiiB Case II and Case III
In Case II, energy packets () arrive at the battery with probability per slot. Hence, the length of the energy queue may increase by or (when one energy packet is consumed in the current slot) each time. The resulting twodimensional Markov chain is shown in Fig.2(d). In Case III, data packets arrive with the probability at each slot, and the twodimensional Markov chain is illustrated in Fig.2(d), where the data queue length could increase by or (when one data packet is transmitted using an energy packet harvested or drawn from the RES in the current slot).
As shown in Fig.2(d), the solid lines present the fixed state transitions while the dotted lines indicate state transitions that vary with different . In particular, the state transfers to with the probability and to with the probability , respectively. Similarly, the state transfers to with the probability , and to with the probability , respectively. Note that the states for all are transient.
Similarly in Fig.2(e), solid and dotted lines are used to present the fixed state transitions and state transitions that vary with different , respectively. Similar to Case I, the state transfers from to with the combined transition probability . For the same reason, the transition probability from to is
(12) 
And the states for all are transient.
Iv LP Problem Formulation
As discussed above, both the average delay and power consumption from the RES are functions of the steadystate probabilities of the corresponding Markov chains, which in turn depend on the transmission parameters and to be designed. To seek the optimal scheduling policy, we adopt a twostep procedure [23]: first we formulate an LP problem only depending on the steadystate probabilities, and obtain the corresponding solution; then from the optimal solution of the LP problem, we determine the optimal transmission parameters.
Our objective is to minimize the average queueing delay subject to the maximum average power constraint from the RES. The corresponding LP problem can be formulated as
(13) 
From the properties of a Markov chain, the last three constraints (c)(e) are straightforward. The original definition of (c.f. (8)) in constraint (a) does depend on the transmission parameters; to facilitate derivation, we will give a new expression for in Lemma 2 below that is only a function of the steadystate probabilities and , . The influence of the transmission parameters on the problem is encapsulated in the constraint (b), which represents the relationship between the steadystate probabilities due to the varying transmission parameters and , as discussed later in Lemma 3. The optimal solution to (13) is denoted by and the minimum average delay by .
Case I with  Case II with and  Case III with and  
Case I with  Case II with  Case III with  
Lemma 2.
In Cases I, II and III, the normalized average power consumption from the RES can be expressed as
(14) 
where the coefficients and are presented in Table I.
Proof:
The proof is deferred to Appendix A. ∎
Remark: By exploiting the local balance equations of states , we can replace all the items and of with the items and . In this way, the average power consumption becomes a linear function of the steadystate probabilities and . Thus, the direct dependence of on the transmission parameters and is removed.
Then, we discuss the constraint (13.b). The basic idea is to vary the transmission parameters and in the full range of , so as to obtain an upper and lower bound for each . In this way, we transform the constraints on and into the relationship between the steadystate probabilities themselves, which allows us to obtain the optimal solution to (13) in terms of first. For ease of illustration, we define several constants as , , and . Let us define .
Lemma 3.
Proof:
The proof is deferred to Appendix B. ∎
Remark: From the proof of Lemma 3, we have at , and at , respectively, in all the three cases ^{3}^{3}3More rigorously, in Case I, holds just when and can be arbitrary. . This lies in the fact that the transmission parameters and determine the relationship between the steadystate probabilities , and vice versa. As listed in Table II, is a linear function of the steadystate probabilities , and is a linear function of .
V Delay Optimal Scheduling Under Power Constraint
In this section, we discuss the optimal solution to Problem (13) by studying its structure with respect to the steadystate probabilities of the corresponding Markov chains.
Va Structure of The Optimal Solution
For ease of discussion, we first consider a scheduling policy strictly based on the threshold : the source waits for the harvested energy when the number of backlogged data packets is less than or equal to a certain threshold and transmits using the reliable energy when the data queue length exceeds . According to the threshold , we use to measure the amount of power drawn from the RES. Since is sufficient for the application of the scheduling policy based on the threshold , but not vise versa, is nonincreasing with the threshold . We will show that the threshold based scheduling policy turns out to be the optimal and the optimal threshold is determined by the power thresholds .
Theorem 4.
The optimal threshold is when , and when , respectively.
Proof:
The proof is deferred to Appendix C. ∎
We notice that the average queueing delay is a weighted summation of the steadystate probabilities . Thus, can be reduced, if we assign a larger value to with a smaller index and vice versa. Based on this intuition, we can reveal that the optimal solution to the LP problem (13) corresponds to a threshold based scheduling policy with the optimal threshold determined by the maximum allowable power consumption from the RES .
Theorem 5.
The optimal solution satisfies
(16) 
where the optimal threshold is obtained as
(17) 
Proof:
The proof is deferred to Appendix D. ∎
Remark: According to Lemma 3, we have or when , and when , respectively. Therefore, associated with (16) is a threshold based scheduling policy that waits for the harvested energy when the number of backlogged data packets is less than a certain threshold , and draws the reliable energy definitely when the harvested energy is not available while the number of backlogged data packets exceeds the threshold ( if there is no new data packet arrival, and if there is new data packet arrival).
Note that the LP problem (13) has an optimal solution only when the queueing system is stable, , when the service rate is greater than the arrival rate, according to Loynes’s theorem [24]. Throughout this paper, the service rate is specialized as the total amount of energy that can be drawn either from the RES or from the battery, . Hence, we will discuss the optimal solution to the LP problem (13) under the assumption that .
VB The Optimal Solution
By exploiting the result in Theorem 5, we continue to derive the optimal steadystate probabilities for Case I, and develop an algorithm to obtain the optimal solutions for Case II and Case III, respectively.
VB1 Case I
In this case, the twodimensional Markov chain is reduced to a onedimensional one, where transitions takes place only between adjacent states, as shown in Fig.2(c). We only need to discuss the optimal steadystate probabilities and for all and . In the sequel, we first show that the optimal is a function of in Lemma 6 and then present the optimal in Corollary 7.
Lemma 6.
In Case I, the optimal steadystate probability is related to as
(18) 
From (18), we get , where
From the results obtained in Theorem 5, we show that the optimal for all are functions of . Further, taking advantage of the dependance of on , we can derive the closedform optimal solution in Corollary 7.
Corollary 7.
In Case I, when