1 Introduction
One of the common drawbacks of all machine learning techniques used in Signal Processing is that none of the methods suggests any convenient way to construct the confidence intervals of the predicted signals. Any forecasting or an estimation technique used for the prediction of a signal always leads to some level of random variations in the values of the prediction at each time point. A statistical way to address such variation is by providing a confidence band of that predicted values. All statistical signal processing techniques address the issue while making their prediction. Machine learning approaches do not have any wellstudied framework to address this critical problem. One of the reasons is such construction of confidence interval requires appropriate modeling of signal noise or asymptotic properties of the estimated parameters. In this paper, we address an innovative way to construct the confidence interval through different types of bootstrap. Finding the bestconstructed confidence interval also requires an effective measure to guess the smallest confidence band given a particular probability. We set some criteria to address the issue, which is capable of comparing different methods.
Predicting the price of a stock is a challenging problem attempted for years. A popular method to predict the stock is technical analysis (PatelShahThakkarKotecha:2015 , SenChaudhuri:2017 ). However plenty of different other methods based on computational intelligence accomplish accurate predictions on stock market [e.g. ZhangZhongDongWangWang:2019 , SongLeeLee:2019 , SenChaudhuri:2017 , PatelShahThakkarKotecha:2015
]. These algorithms include evolutionary computation through the genetic algorithm to support vector machine or various neural networkbased approach [
SainiSharma:2019 , TangDongShi:2019], deep learning methods such as deep belief net coupled with multilinear perceptron [
LongLuCui:2019 , KhoaSakakibaraNishikawa:2006 , JiangTangChenWangHuang:2018 , SelvamuthuKumarMishra:2019 ]. However, these models do not deal with the dependent stream of data. LSTM based model is an alternative popular method HochreiterSchmidhuber:1997used for modeling dependent data. LSTM and its variations are useful in areas such as Natural Language Processing [
ShiTangLong:2019 , ShiTangCui:2018 ], Financial Time series, or some application of speech processing. Many researchers (JiangTangChenWangHuang:2018 , ChenZhouDai:2015 , Jia:2016 ) used an LSTM model to predict the stock price. None of the literature contains any methodology/convenient way to construct an appropriate confidence band in this setup. Also, there is no appropriate notion available to find the best confidence interval of the forecast signals in an LSTM type of model set up.We organize the paper as follows. In Section 1, we provide the details of the data used in this paper. The description of the LSTM model used in this paper is available in Section 2. We describe different types of bootstrap algorithms used for dependent set up in Section 3. In section 4, we discuss the proposed methods to construct confidence intervals in various ways and also the techniques to get the bestconstructed confidence interval. The performance of the proposed architectures is available in Section 5. We conclude the paper in Section 6.
2 Data Set Description
We take two different data sets to carry out our empirical analysis. We study S& P 500 from 28th April 2015 to 27th April 2020, and Google stock price from 8th February 2013 to 7th February 2018. The sample size considered in both cases is 1259. We consider the only closing price of the stock. In an LSTM network, we use inputoutput values as logreturn data. However, we can recalculate the predicted closing price from logreturn data. The main objective of the paper is not an innovation in predicting accurate prices; instead, the paper explores the best confidence band of the predicted stock price via an LSTM model or similar machine learning model. We divide the data set into nearly 64% and 36% i.e., we take the number of training data set as 800 and testing set as 459.
3 Usual Machine learning Algorithms used for Predictions
3.1 Long ShortTerm Memory (LSTM)
Here we introduce the construction of a special kind of cell in LSTM, which took the place of the traditional hidden unit HochreiterSchmidhuber:1997 . This cell takes in the current input , the previous output , and the previous cell state (memory) . Inside the cell, there are three gates, forget gate (), input gate (), and output gate (). The orientation of LSTM enables the important memory of the previous time to carry forward a long way with an intervention of a forget gate at each time step, which helps to keep only the relevant information or memory.
The following equations govern the calculation of the gate vectors:
(1) 
(2) 
(3) 
The new values of cell state and output (along with an intermediatary variable ) are calculated by the following equations:
(4) 
(5) 
(6) 
Here, denotes Hadamard product.
3.2 Some Issues in LSTM model set up
Window feature scaling enhances training because it enables the model to capture the relativevariance in the data in a small neighborhood. The feature scaling is a crucial part of training LSTM models in this context. It normalizes the entire data, featurewise, to convert it to values between 0 and 1 using minmax scaling. Different types of normalization are available in the literature. However in this paper we use the following relation for normalization : Since we deal with a constant time step (lookback) while training LSTM, it makes sense to normalize the data in each lookbacklength window. However, the prediction from the model is later denormalized. We use the Adam optimization algorithm to train the LSTM model. The choice of optimal lookback and batch size are made by trial and error method so that the optimal number minimize RMSE of the test set. Details of choice for the parameter tuning are available in Section 5.
4 Three different Bootstrap for dependent set up
There are multiple methods of bootstrap available in the literature [e.g. KreissaPaparoditis:2011 , RadovanovMarcikić:2014 ]. We choose the following major three methods in this paper for comparison purposes.
4.1 Method 1 : Nonoverlapping Block Bootstrap (NBB)
We take up the following steps :

if nonoverlapping blocks are considered.

Lay the blocks , end to end in the order sampled together and discard the last observations to form a bootstrap pseudo series .
4.2 Method 2 : Moving Block Bootstrap (MBB)
Assume is an integer between 1 to holding a property that . Anyway, a specific description of this method must start from considering the block length as constant. If denotes an ith block of time series, then, the block length begins with , for , where is represented as the number of blocks within the bootstrap sample. In order to form a sample from moving block bootstrap method, it is necessary to choose randomly a certain number of blocks from the set . Therefore represents a random sample with repetitions from the set where each block contains same number of observations.
4.3 Method 3 : Local Block Bootstrap (LBB)
Given the data set , LBB algorithm creates a pseudoseries as follows :

Select an integer block size , and a real number such that is an integer, both l and B are a function of .

For let for where are independent integer valued random variables satisfying . A choice , , .
5 Proposed Method:
In this paper, we contribute in different ways. First, we describe some implementation issues in practice to obtain the optimal bootstrap block length in this setup. Second, we develop a method to construct a confidence interval for the stock price signal received from the LSTM network. Third, we develop a benchmark to judge the best confidence interval in a different setup.
5.1 Benchmark set for best choice of the block length in Bootstrap:
The choice of block length in Bootstrap is an interesting problem. Hall et al. HallHorowitzJing:1995 established that the best block size depends on three factors: the autocorrelation structure, the series length, and the reason for bootstrapping. Most of the papers discussed such choices in the context of estimation or finding sampling distribution of bootstrap mean or variances for dependent set up [NordmanLahiri:2014 , PolitisWhite:2004 , Lahiri:1999 ]. The purpose of any bootstrap is to regenerate the sample, which behaves like the original sample. Therefore it is expected that a regenerated sample should be the best representation of the original sample. Thus the best bootstrapping algorithm mimics the data in the best possible way. Statistically, this is possible when the variance of the generated curve shows a minimal variation for the original curve. However, quantifying this variance is not unique, and thus many algorithms are available. Finding the mean series from the blocks of the original and bootstrapped sample and creating a distance function for the same could be a way to quantify this variation. Demirel and Willemain (DemirelWillemain:2002 ) uses a similar concept in their paper to find the best bootstrap length. His approach uses Higherorder crossings (HOC) to form the benchmark to find out the bootstrap length. This approach does not have a proper theoretical basis for nonstationary data and has many shortcomings.
In our paper, we proposed a novel penalty functionbased approach to select the block length. Instead of a HOC based statistic, we proposed using the following the empirical distance function based on the mean series of the original sample and bootstrapped sample : , where : the number of bootstrap sample drawn, : the series length, : the number of blocks, : the block length and , are mean series of bootstrapped and original sample respectively. Based on our intuition, we can realize that the best block length should be the length, which minimizes the above empirical function.
However we observe numerically that the above empirical function decreases with lots of oscillations with respect to block length. As suggested by Hall et al. HallHorowitzJing:1995 , this oscillation depends on the autocorrelation structure, the series length which makes selection of block length very difficult. If we plan to use a penalty function in addition to this empirical function, convexity of the function depends again with suitable choice of the series length. A separate attention/research is required in this direction. However we propose to use a different trick in this context. We propose to take block length which minimizes the following function : , where . And , are mean series of original data and bootstrapped data when we take original data as logreturns of the prices. This small modification makes the function an excellent convex function for block length. We may need to adjust the value of for exceptional data, as this determines the convergence rate of the above the selector of the block length. However, from our experimental analysis and empirical evidence, we observe it is reasonable to take its value as 2. Fig.1 and Fig.2 reveal the pictorial overview of the convexity of the block length selector in Moving Block Bootstrap when the value of is 2. The figures for other procedures are similar; therefore, we omit them from the paper. They can be made available on request to authors.
The above procedure does not create any problem to fulfill our objective of constructing a confidence interval. We can graphically observe that conversion of the original data into a logreturn series, and a reverse transformation of the bootstrapped logreturn data is capable of mimicking the original data equally well. The above procedure is simple to implement and can be applied even when Higherorder crossings (HOC) fails to capture the block size accurately for a direct, highly volatile, and nonstationary type of data. However, our proposed approach needs more systematic analysis. We pursue the details in some other work.
5.2 Methodology for Construction of Confidence Interval:
Our proposed method to construct the confidence interval in LSTM set up is as follows :

Create bootstrap samples, which mimic the original sample of the input.

For each bootstrap sample, fit an LSTM model separately and predict the test set.

At each time step, find out 95% nonparametric confidence interval to construct the confidence interval of the test set.
In this paper, we propose to construct a slightly different bootstrap sample for closing prices instead of directly extracting bootstrapped samples. We observe that our algorithm depends on the choice of optimal block length that requires transforming the data into the logreturn series. Therefore we first transform the data into logreturns and choose optimal block length based on logreturn series. Different bootstrap techniques are used on the logreturn data using the calculated optimal block length. A reverse transformation helps to create the bootstrap sample for closing prices. We observe graphically that such bootstrap data mimics the original data also very well.
5.3 Benchmark:
Setting a benchmark is crucial in comparing the constructed confidence intervals. We use the sum of confidence width at each time instance as a naive approximation of confidence band, which we consider as comparing factor. We calculate this comparing factor for test set price signal for each bootstrap/or other procedures and use this quantity as a benchmark for comparing different confidence intervals obtained by various methods. The bestconstructed confidence interval would be the one providing a minimum sum of confidence width.
6 Performance of the Architectures and Numerical Results
6.1 Parameter Tuning
While applying the LSTM, we take batch size as 15 and look back number as 5. This choice is made by trial and error, minimizing the RMSE. We take the smoothing window as 200, whereas we take the dropout rate at 0.2. Again optimal no of epochs is kept as 19 since RMSE does not have significantly lower value after 19. Finally, training and testing set RMSE is found at 0.1296 and 25.2169, respectively. We introduce kernel regularizer, which is essentially regularization on the weights of neurons. The optimal choice of block length depends on the dataset we use for empirical analysis. For S&P 500 data, we get block length as 3 for LBB and 4 for NBB and MBB. The optimal block length for Google stock price data is 6 for all bootstrap algorithms. We make all empirical analyses based on 1000 generated bootstrapped samples.
6.2 Graphical Sense of Bootstrap data :
It is also important to figure out how the optimal choice of bootstrap block length enables the resampling technique to mimic the original data set. We take a graphical sense of the fact by plotting original data and bootstrap resampled data obtained through different procedures. Fig.3, Fig.4, Fig.5 show that local block bootstrap best captures the patterns of the original data.
6.3 Results and Comparisons on the Data set
Fig.6, Fig.7, Fig.8 provide the picture of the constructed confidence interval of Google stock price on the test set by Local block bootstrap, Nonoverlapping block bootstrap, Moving block bootstrap, respectively. For S& P 500, we see the prediction bands in Fig.9, Fig.10, Fig.11 respectively. The Prediction in the case of S& P 500, deals with a lesser number of hidden parameters within LSTM nodes. We observe that the predicted curve better matches with actual observations if we increase the number of hidden nodes in each LSTM cell. As S& P 500 includes some recent stock price data, it reflects the behavior in the presence of the current coronavirus pandemic. The last part of the Figure shows a sharp depletion of the stock price, which is the pandemic effect. We can see how the confidence interval constructed captures original data in the presence of the coronavirus shock wave. We also compute the proposed comparing factor for both the data using the three of these bootstrap procedures. For Google stock price, we obtain the values as 47484.094, 53325.1136, and 53767.2914, respectively, whereas for S& P 500, we get the numbers as 163697.7224, 184850.4583, 189109.1758 respectively. As the minimum comparing factor should provide the best procedure for the setup, Local block bootstrap appears to be the best bootstrap procedure to construct a confidence interval in univariate LSTM set up which comes out as the best in terms of comparing factor in S&P 500 as well as Google stock price data set. It makes sense to get Local block bootstrap as the best procedure in all cases, as we observe from Fig.3 that it mimics the original data the best as compared to other algorithms.
7 Conclusion
We verify graphically that the proposed choice of Bootstrap length is capable of mimicking the data set very well. Local block bootstrap has the best ability to mimic the original data set. We tune parameters in such a way that the LSTM model becomes capable of predicting the stock price efficiently. Three bootstrap procedures work quite well to determine a long term confidence band. We found local block and nonoverlapping block bootstrap performs almost equally well and better than the moving block bootstrap method. We can also extend the idea in the Spatiotemporal setup or in a multivariate time series model. The work is in progress.
References
 (1) Chen, K., Zhou, Y., Dai, F.: A lstmbased method for stock returns prediction: A case study of china stock market. In: 2015 IEEE international conference on big data (big data), pp. 2823–2824. IEEE (2015)
 (2) Demirel, O.F., Willemain, T.R.: Generation of simulation input scenarios using bootstrap methods. Journal of the Operational Research Society 53(1), 69–78 (2002)
 (3) Hall, P., Horowitz, J.L., Jing, B.Y.: On blocking rules for the bootstrap with dependent data. Biometrika 82(3), 561–574 (1995)
 (4) Hochreiter, S., Schmidhuber, J.: Long shortterm memory. Neural computation 9(8), 1735–1780 (1997)
 (5) Jia, H.: Investigation into the effectiveness of long short term memory networks for stock price prediction. arXiv preprint arXiv:1603.07893 (2016)
 (6) Jiang, Q., Tang, C., Chen, C., Wang, X., Huang, Q.: Stock price forecast based on lstm neural network. In: International Conference on Management Science and Engineering Management, pp. 393–408. Springer (2018)
 (7) Khoa, N.L.D., Sakakibara, K., Nishikawa, I.: Stock price forecasting using back propagation neural networks with time and profit based adjusted weight factors. In: 2006 SICEICASE International Joint Conference, pp. 5484–5488. IEEE (2006)
 (8) Kreiss, J.P., Paparoditis, E.: Bootstrap methods for dependent data: A review. Journal of the Korean Statistical Society 40(4), 357–378 (2011)
 (9) Lahiri, S.N.: Theoretical comparisons of block bootstrap methods. Annals of Statistics pp. 386–404 (1999)
 (10) Long, W., Lu, Z., Cui, L.: Deep learningbased feature engineering for stock price movement prediction. KnowledgeBased Systems 164, 163–173 (2019)
 (11) Nordman, D.J., Lahiri, S.N., et al.: Convergence rates of empirical block length selectors for block bootstrap. Bernoulli 20(2), 958–978 (2014)
 (12) Patel, J., Shah, S., Thakkar, P., Kotecha, K.: Predicting stock and stock price index movement using trend deterministic data preparation and machine learning techniques. Expert systems with applications 42(1), 259–268 (2015)
 (13) Politis, D.N., White, H.: Automatic blocklength selection for the dependent bootstrap. Econometric Reviews 23(1), 53–70 (2004)
 (14) Radovanov, B., Marcikić, A.: A comparison of four different block bootstrap methods. Croatian Operational Research Review 5(2), 189–202 (2014)

(15)
Saini, A., Sharma, A.: Predicting the unpredictable: An application of machine
learning algorithms in indian stock market.
Annals of Data Science pp. 1–9 (2019)
 (16) Selvamuthu, D., Kumar, V., Mishra, A.: Indian stock market prediction using artificial neural networks on tick data. Financial Innovation 5(1), 16 (2019)
 (17) Sen, J., Chaudhuri, T.: A robust predictive model for stock price forecasting. In: Proceedings of the 5th International Conference on Business Analytics and Intelligence (ICBAI 2017), Indian Institute of Management, Bangalore, INDIA (2017)
 (18) Shi, Y., Tang, Y.r., Cui, L.x., Wen, L.: A test mining based study of investor sentiment and its influence on stock returns. Economic Computation & Economic Cybernetics Studies & Research 52(1) (2018)
 (19) Shi, Y., Tang, Y.r., Long, W.: Sentiment contagion analysis of interacting investors: Evidence from china’s stock forum. Physica A: Statistical Mechanics and its Applications 523, 246–259 (2019)
 (20) Song, Y., Lee, J.W., Lee, J.: A study on novel filtering and relationship between inputfeatures and targetvectors in a deep learning model for stock price prediction. Applied Intelligence 49(3), 897–911 (2019)
 (21) Tang, H., Dong, P., Shi, Y.: A new approach of integrating piecewise linear representation and weighted support vector machine for forecasting stock turning points. Applied Soft Computing 78, 685–696 (2019)
 (22) Zhang, K., Zhong, G., Dong, J., Wang, S., Wang, Y.: Stock market prediction based on generative adversarial network. Procedia computer science 147, 400–406 (2019)
Comments
There are no comments yet.