In signal processing, compressive sensing  is a powerful sensing paradigm to sample sparse signals with much fewer samples than demanded by the Shannon-Nyquist sampling theorem. The Nyquist theorem mandates that the number of data samples to be at least as big as the dimensionality of the signal being sampled. Inherent redundancy present in the real signals like images and videos allows significant compression of data. Compressive sensing exploits this inherent redundancy and enables the sampling to happen at sub-Nyquist rates. As generic video acquisition systems face a trade-off between spatial and temporal resolution due to communication bandwidth, systems that can give higher spatial resolution without compromising temporal resolution are in high demand.
This makes compressive sensing extremely useful for capturing images and videos in systems that cannot afford high data bandwidth as the number of samples needed for the same duration of video is much lower than that of generic imaging systems. The ability to sample very few data points and still be able to reconstruct the original signal helps to create low-power imaging systems. Pixel-wise coding  is one among many ways of compressive sampling the videos. Each pixel is independently exposed for a specific amount of time (bump time) and the start time of that exposure is decided by a random sensing matrix.
Using this controlled exposure, the imaging system can obtain a slower frame rate without having to worry about motion blur. Typically a 100 FPS video can be reconstructed from a 5 FPS video, using pixel-wise coded exposure. Jie Zhang et al. have created an all-CMOS spatio-temporal CS video camera with pixel-wise coded exposure technique at its core.
To reconstruct the original frames of a video from CS frames, one needs to learn an over-complete dictionary or use existing dictionaries such as 3D DCT dictionary to represent the videos in a sparse domain. The reconstruction of each CS frame is done on a patch-by-patch basis. A patch of size around
is taken from the CS frame and the corresponding video patch in the sparse domain of the dictionaries is estimated using algorithms like Orthogonal Matching Pursuit (OMP). Recent works such as Iliadiset al. and Xu et al.
have even succeeded to reconstruct original frames from CS frames using deep learning. Though this line of work is still in progress, reconstruction of the CS frames takes considerable amount of time, which makes it unsuitable for any sort of real-time processing. To overcome this issue, we propose a new approach by eliminating the need to reconstruct the original frames for object detection and localization by performing it directly on the compressed frame. We simulated the images created by a CS camera  by compressing 30 FPS videos with 13x compression using a randomly generated sensing matrix with a bump time of 3 frames. We generated a dataset for the object detection task using the camera simulator.
2.1 Pixel-wise coded exposure
In conventional cameras, there is a global exposure time (shutter speed) for which all the pixels in the sensor are exposed and an image is readout from the sensor. We get frame rate from such sensors. In contrast to this, in pixel-wise coded exposure (PCE) cameras, each pixel is exposed at a random time for a ’single-on’ fixed duration (> ) within the time . But, only one image is read out at the end of . If is set to , we get a compression of times and a frame rate of . This single frame at the end of has all the necessary information of the intermediate frames encoded in it.
Figure 2 shows the process of PCE. Though this creates only frame rate, all the intermediate frames ( ) can be reconstructed using the OMP algorithm based on the random sensing times of each pixel (Sensing Matrix). This makes it suitable for capturing high temporal resolution video without compromising the spatial resolution.
2.2 Sensing Matrix
Consider a spatio-temporal video . A sensing matrix holds the exposure control values for each pixel in the sensor. The sensing matrix is binary, i.e., . The value of is 1 for each pixel for the frames for which it is on or exposed, and 0 for the rest of the frames. There is only one bump in the time.
The position of 1s in S is randomly decided based on a Gaussian normal distribution, i.e, the start time of each pixel’s exposure time is randomly chosen for each pixel and then it is exposed for(Bump Time) number of frames. These values are then summed up along the time dimension. The acquired coded image can be denoted by the equation:
Due to the unavailability of any pre-existing public dataset on temporally compressed videos, we had to create our own dataset. We searched the Youtube 8M 
dataset for videos which had either a single person or a single car present in it. As all the variations were not available, we collected more videos on our own within our university campus for more training data. We made sure that only one object of interest was present in all of the videos captured. We captured videos on multiple days and during different parts of the day and tried to maintain as much variance as possible. The videos include moving objects with a stationary background, stationary objects with moving background and both the object and background moving(Figure3). We also considered the case where the frame (compressive sensing camera) used to capture is also in motion with respect to the scene. This dataset comprises a combination of motion, both with respect to the camera and the object, thus covering the entire spectrum of instances one can witness in the original domain to be addressed in the compressed domain.
Compression Technique using PCE
30 FPS clips were accumulated from various sources and compressed with 13X compression rate. The bump time is set to 3. For every 13 frames in the original video, a single compressed frame is generated using equation1. The random sensing matrix has been varied for each set of 13 frames, so as to generalize compressed sensed frames and to not fit over a particular sensing matrix. We normalize the pixel values after adding them through 13 frames to constrain the pixel value within 255. We only worked with monochrome videos because conventionally only monochromatic CMOS sensors  are available.
Psuedo labels for ground truth
Manually labeling of the CS frames is a difficult job as the edges of moving objects are often indeterminable even to human eye. To avoid this problem, we chose pseudo-labeling of the data. As most of the existing architectures like YOLO v3 detect persons and cars classes with really good accuracy in the original domain, we used trained YOLO v3 to pseudo label the CS frames Fig.3.
As shown in Fig.4 we used a pre-trained YOLO model to generate the bounding boxes of the individual frames. The threshold for detection was kept more than 99%, so as avoid any ambiguous predictions. We fed these pseudo labeled boxes generated by YOLO with the corresponding frames onto VATIC annotation module, where it was manually corrected for the boundary boxes to fit it more compactly. Manual labeling was considered to mitigate the delta difference possessed with the YOLO prediction from the actual ground truth.
We also had to come up with a way to draw bounding boxes in the CS domain. So, we chose to draw a bigger bounding box in the CS frame which enclosed all the individual bounding boxes of the constituent frames. Fig.6 shows the method to merge the bounding boxes of each object across the constituent frames. Then with these boundary box coordinates, we find the minimum and maximum of its X and Y coordinates for the entire clip’s timeline. These coordinates are then used for the bigger bounding box in the CS frame Fig.5. Tracking a single object’s detection in consecutive frames which contain multiple objects of the same class is difficult because YOLO doesn’t order them in any particular fashion. Because of this difficulty to attach each bounding box and track it along the frames to a specific object, for training dataset we collected videos which have only one object of interest. This kind of labeling gave a preemptive measure to easily rectify and provide a tight fit of boundary box for the object in the frame with ease. Thus allowing us to create a large dataset of annotated compressive sensed images, these labelled boundary boxes were then combined into larger bounding boxes that were used as the ground truth values of the bounding boxes in the CS frames Fig.6. We wanted less human intervention for the labeling, the pseudo labeling provided an extra hand for the human introspection for labeling. By providing the pseudo labeling the human effort here was reduced and more attention was drawn towards reiterating the boundary box more tightly to fit the objects more precisely on the compressed frame Fig.6. Approximately 45,472 and 60,708 datapoints samples are available for the class Cars and Person respectively. This dataset is now openly available for the community.
2.4 Model and Training
We used the pre-trained initial layers of YOLO(14 layers), these layers pertain the prior knowledge for the detection and localization of objects in the original RGB domain. We passed the output of these layers through a series of 7 Convolution layers(resembling Layer 15) as described in the Fig. 7, followed by BatchNormalization 
and composed them onto a 256 dimensional vector which was activated by LeakyReLU. Now, the output of the activation layer(LeakyReLU) and the lambda layer are concatenated, this is performed to ensure that the model can comprehend each instance of the object present from the original domain and triangulate the location and the class label of the object present in the compressed domain. This latent space is then fed to a Convolution layer that deciphers the presence of the object and location of the object and the class labels present in the compressed frame. The cars or persons in the CS frames have high variance in their appearance, based on the level of motion present in the 13 frames. Depending on the motion, they can either look like normal cars/persons when stationary or hazy when in motion. Therefore, we needed a pre-trained model with prior knowledge similar to that of YOLO v2 to differentiate the different classes (Cars and Persons being one among them). The model being trained from scratch without pre-trained weights, has the risk of fitting to the blurred patterns of cars and persons. Hence, we generated large training dataset and also started the training with the pre-trained weights.
The model was trained using Tensorflow
. The training converges within 25 epochs, using Adam optimizer for parameter updates with learning rate = 1e-5, beta_1 = 0.9, beta_2 = 0.999, and epsilon = 1e-08. The mini-batch size was 16 for the training. We trained this hybrid network on the dataset with a train and test split ratio of 7:3.
3 Results and Discussion
3.1 Initial Results
Initially, we attempted to train only for a single class of objects. The model gave very poor results in terms of output confidence levels and IOUs. It was not able to differentiate between the moving background and the moving object. However, with both the classes in the training set, the model gave much better results and high output activations when compared to models trained for a single class. This proves that adding more classes increases the capability of the model to learn the differentiating features. Final results of the model on various dynamic scenes of both classes can be seen in Figures 8 & 9.
3.2 Mean Average Precision
We have followed the COCO dataset’s method of evaluating the Mean Average Precision (mAP), by averaging the Average Precisions (APs) over multiple IOU (Intersection over Union ratio of predicted box and ground truth) thresholds from 0.50 to 0.95. The ’Car’ and ’Person’ classes got an mAP of 42.42% and 50.12%, respectively. We also experimented with our trained model to test on CS frames with different compression rates and bump times. We chose sequences of CS frames that gave a good detection with 13x compression (at which the model was trained) and recreated the same sequences with different compression and bump time rates. The AP values at different IOUs for different variations of bump times and compression rates are shown in Table 2 and 3, respectively.
|Model||Mean Average Precision (mAP)|
The results showed no conclusive proof of any trend of decline or ascent in the mAP with respect to the bump time or the compression unless they are the extremes. This shows that our model is generalized and can work with other compression rates and bump times.
|Bump Time||AP at IOU = 0.50||AP at IOU = 0.55||AP at IOU = 0.60||AP at IOU = 0.65||AP at IOU = 0.70||AP at IOU = 0.75||AP at IOU = 0.80||AP at IOU = 0.85||AP at IOU = 0.90||AP at IOU = 0.95||Mean AP|
|Compression Rate||AP at IOU = 0.50||AP at IOU = 0.55||AP at IOU = 0.60||AP at IOU = 0.65||AP at IOU = 0.70||AP at IOU = 0.75||AP at IOU = 0.80||AP at IOU = 0.85||AP at IOU = 0.90||AP at IOU = 0.95||Mean AP|
We can observe from the Tables 3 and 2, that different compression rates of the CS frames do not change the mAP drastically. The reason why other compression rates also work is because, CS frames have high variance in the appearance of the objects, based on the amount of motion in the constituent frames. If the object is stationary and still all through the frames, the object in the corresponding CS frame also appears as seen in Figure 8(e). In such cases, irrespective of the compression or the bump time rates, the CS frames look very alike. The same can happen with combinations of different amounts of motion at different compression rates. Hence, once a model is trained on varying levels of motion at a compression rate, it can also work with other compression rates fairly well. We also compared the performance of our model with a few state of the art object detectors (SNIPER and Yolo V3) on the compressed frames which is listed in Table 1. It is observed that our model displays relatively superior results.
We also observed that the model was capable of detection with different combinations of background and foreground movements Figure [8 and 9]. Our model could detect overlapping moving cars, just like other object-detection models.
Baraunink et al.
explains the connection between compressive sensing and Johnson-Lindenstrauss lemma. The preservation of distances between the higher-dimensional space to the lower-dimensional spaces can be the reason object detection in CS frames is possible using convolutional neural networks. The pre-trained weights of YOLO have equipped the model with the necessary features representations to differentiate different object classes. We have also tested the model on coloured CS frames (Figure10). When trained on colour CS images it might work even better because of the extra features a model can learn. As colour CS CMOS cameras are not a distant reality, our results show that our model can work for them if needed in the future.
To evaluate the model, we deployed the network on NVIDIA TX2 on a typical surveillance use case. Here the the live camera feed from the camera was compressed using the same technique of pixel wise coded exposure mentioned as proposed in the dataset . These compressed frames were fed to the NVIDIA board where the model was deployed, the model took in the compressed frames as input and was able the predict the object which was present in them Figure 4 depicts the process, the model provides the class of the object present with the location of object by surrounding them with a boundary box. We can observe that the compressed frame contains hazy information due to which the privacy concerns raising due to surveillance can be avoided. The compressed frame intactly preserves all the information required for the reconstruction of the frames but at the same time protects the privacy.
In this work, we showed that object detection and localization is possible directly at the compressed frame level in pixel-wise coded images. To our knowledge this is the first time the problem of object detection and localization in the CS domain has been attempted. Our model was also able to detect objects that could not be detected by humans in the CS domain. We also showed that our model works on different compression rates of CS frames. We reduced the time of object searching algorithms in surveillance videos by an order of 20x. As the CS cameras have low readout, real-time object detection is possible with our model. In future, we will continue adding more classes to the dataset and try other architectures to improve the precision of the model. We are looking at using the information of the sensing matrix to refine the results of the model. We envisage that this will be the beginning of object detection and other computer vision tasks in CS domain, which would make significant improvement in surveillance. Due to lack of any better comparison, Table4 shows the comparison of mAP values of existing object detection models in comparison to our model. The mAP of our proposed model suggest that its performing in par with that of the generic models that work on original domain, even though the boundaries and texture of the objects are not so well defined in the CS domain. The proposed model was able to detect and localize all the classes it has been trained on upon the compressed frame.
-  Xiong, Tao and Zhang, Jie and Thakur, Chetan Singh and Rattray, John and Chin, Sang Peter and Tran, Trac D and Etienne-Cummings, Ralph, Real-time segmentation of on-line handwritten arabic script. In 2017 IEEE International Symposium on Circuits and Systems (ISCAS), pages 1–1, IEEE, 2017.
-  Baraniuk, Richard G and Goldstein, Thomas and Sankaranarayanan, Aswin C and Studer, Christoph and Veeraraghavan, Ashok and Wakin, Michael B, Compressive video sensing: algorithms, architectures, and applications In IEEE Signal Processing Magazine, Volume 34, pages 52-66, IEEE, 2017.
-  Hitomi, Yasunobu and Gu, Jinwei and Gupta, Mohit and Mitsunaga, Tomoo and Nayar, Shree K, Video from a single coded exposure photograph using a learned over-complete dictionary In 2011 International Conference on Computer Vision, pages 287–294, IEEE, 2011.
-  Redmon, Joseph and Farhadi, Ali, An incremental improvement In arXiv preprint arXiv:1804.02767 , 2018.
Andreas Geiger and Philip Lenz and Raquel Urtasun,
Are we ready for Autonomous Driving? The KITTI Vision Benchmark Suite
Conference on Computer Vision and Pattern Recognition (CVPR)2012.
-  Vondrick, Carl and Patterson, Donald and Ramanan, Deva, Efficiently scaling up crowdsourced video annotation, In International Journal of Computer Vision, Volume 101, number 1, pages 184–204, Springer, 2013.
-  Abu-El-Haija, Sami and Kothari, Nisarg and Lee, Joonseok and Natsev, Paul and Toderici, George and Varadarajan, Balakrishnan and Vijayanarasimhan, Sudheendra, Youtube-8m: A large-scale video classification benchmark, In arXiv preprint arXiv:1609.08675, 2016.
-  Wu, Yi and Lim, Jongwoo and Yang, Ming-Hsuan, Online Object Tracking: A Benchmark, In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June, 2013.
-  Donoho, David L, For most large underdetermined systems of linear equations the minimal l1-norm solution is also the sparsest solution, In Communications on Pure and Applied Mathematics: A Journal Issued by the Courant Institute of Mathematical Sciences, Volume 59, number 6, page number 797–829, Wiley Online Library, 2006
-  Davenport, M, The fundamentals of compressive sensing, In IEEE Signal Processing Society Online Tutorial Library, 2013
-  Erlich, Yaniv and Chang, Kenneth and Gordon, Assaf and Ronen, Roy and Navon, Oron and Rooks, Michelle and Hannon, Gregory J, DNA Sudoku—harnessing high-throughput sequencing for multiplexed specimen analysis In Genome research, Volume 19, number 7, pages 1243–1253, Cold Spring Harbor Lab, 2009.
-  Common Objects in Context(COCO) 2018 Dataset Evaluation "http://cocodataset.org/#detection-eval"
Abadi, Martín and Barham, Paul and Chen, Jianmin and Chen, Zhifeng and Davis, Andy and Dean, Jeffrey and Devin, Matthieu and Ghemawat, Sanjay and Irving, Geoffrey and Isard, Michael and others
Tensorflow: A system for large-scale machine learning.12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16), 265–283, 2016
-  Baraniuk, Richard and Davenport, Mark and Devore, Ronald and Wakin, Michael The Johnson-Lindenstrauss Lemma Meets Compressed Sensing, 2006
-  Chollet, F Keras "https://github.com/fchollet/keras"
-  Kour, George and Saabne, Raid Real-time segmentation of on-line handwritten arabic script In 2014 14th International Conference on Frontiers in Handwriting Recognition, pages=417–422, year=2014, organization=IEEE
-  Iliadis, Michael and Spinoulas, Leonidas and Katsaggelos, Aggelos K Deepbinarymask: Learning a binary mask for video compressive sensing arXiv preprint arXiv:1607.03343, 2016
-  Ioffe, Sergey and Szegedy, Christian Batch normalization: Accelerating deep network training by reducing internal covariate shift arXiv preprint arXiv:1502.03167, 2015
-  Zhang, Jie and Xiong, Tao and Tran, Trac and Chin, Sang and Etienne-Cummings, Ralph Compact all-CMOS spatiotemporal compressive sensing video camera with pixel-wise coded exposure Optics express, volume 24, number 8, pages 9013–9024, year 2016; Optical Society of America
-  Xu, Kai and Ren, Fengbo Csvideonet: A real-time end-to-end learning framework for high-frame-rate video compressive sensing 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), pages=1680–1688,2018, IEEE
-  Xu, Bing and Wang, Naiyan and Chen, Tianqi and Li, Mu Empirical evaluation of rectified activations in convolutional network In arXiv preprint arXiv:1505.00853,2015
-  Singh, Bharat and Najibi, Mahyar and Davis, Larry S SNIPER: Efficient multi-scale training Advances in Neural Information Processing Systems, pages:9310–9320, 2018
-  Diederik P. Kingma and Jimmy Ba Adam: A Method for Stochastic Optimization arXiv 1412.6980, 2014
-  Sathyaprakash Narayanan and Yeshwanth Bethi and Chetan Singh Thakur A Compressive Sensing Video dataset using Pixel-wise coded exposure, arXiv 1905.10054, 2019