The rise of augmented reality provides an exciting venue for extensive human-machine interactions. In essence, augmented reality overlays computer-generated information with the physical world in a composite view of a user through motion tracking sensors and computer vision. Entertainment, design and navigation are key applications of augmented reality. Nonetheless, the potential of augmented reality can also be harnessed by other innovative applications. In this paper, we explore a novel methodology utilizing augmented reality as an empowering technological tool for more effective building automation, and particularly, HVAC control.
Traditional building management systems usually follow simple pre-set control logic programmed in the hardware controllers. Often, a reductionistic approach is employed, wherein the thermal response behavior of buildings is approximated by first-principle models, and linear time-invariant dynamic processes with 1D parameters by ordinary differential equations (also known as lumped element resistance-capacitance (RC) models)[2, 3, 4]. HVAC control strategies are then devised based on such abstract models, which can only approximate simple building geometry in near-future time horizon. These first-principle models are often difficult to calibrate in a dynamic environment because the physical observable parameters from buildings (e.g., building structures, locations of doors and windows) are not explicitly incorporated in the models. Also, the error accumulates considerably when a longer time horizon of the control strategy is considered. While non-linear models are complicated and impractical, other alternatives based on true physical models of buildings are more viable.
I-a Need for Self-aware Buildings
To accurately and responsively control buildings, more precise representation of building thermal response behavior ought to be considered, particularly the detailed spatial-temporal dynamics of thermal flow. The physical thermal response model of a building can be represented by partial differential equations, which often are employed in sophisticated building simulators. The conventional wisdom is that such building simulators need considerable processing power, and hence are only feasibly utilized in the design stage of buildings. Nonetheless, there have been considerable advances in embedded system technologies, which provide low-cost platforms with powerful processors and sizeable memory storage in a small footprint. There are opportunities to harness such embedded system technologies for real-time building automation systems. Particularly, accurate building thermal response simulations based on physical thermal response models can be executed efficiently in real-time on these embedded systems. Therefore, this paper introduces the notion of self-aware buildings, such that buildings can explicitly construct physical models of themselves (e.g., incorporating building structures, materials, and thermal flow dynamics) to enable accurate real-time simulation-guided control in building automation systems.
I-B Augmented Reality as an Enabling Tool
Despite the promising benefits of self-aware buildings, the question is how to enable self-aware buildings that can automatically acquire dynamic knowledge of themselves. This paper presents a novel approach using augmented reality. The extensive user-environment interactions in augmented reality not only can provide intuitive user interfaces for building management systems, but also allow the systems to capture the physical structures and possibly materials of buildings accurately. Augmented reality is also useful for eliciting interactive user feedback for personalization of building management systems. This paper presents a prototype of building management system incorporating augmented reality (based on Google’s Project Tango platform) and discusses its applications. Our main contribution is to demonstrate that augmented reality can help optimize building control by empowering building management systems to automatically recognize the building structures, and then infer the building material properties and HVAC system parameters. The advantages of incorporating augmented reality also include providing appealing motivation to users for upgrading their building management systems that can integrate with home infotainment systems.
Researchers are beginning to explore the use of augmented reality for building navigation, management, and control. The authors in  present opportunities and challenges involved in the use of 3D information of the indoor building environment for the purpose of indoor navigation. A method for 3D reconstruction of indoor and outdoor environments is proposed in . They perform several filtering steps on monochrome fisheye images to reconstruct scenes directly on a mobile device. A history of augmented reality technology, its challenges, and applications can be found in .
Ii Self-aware Buildings
Before defining the notion of self-aware buildings, we discuss several problems of traditional approaches [4, 3] that are based on abstract thermal response models for HVAC control. First, the true physical state of a building is approximated by a canonical state, which normally represents the average state in the space, without considering the spatial variations of thermal response behavior in different regions of the building. There is a considerable loss of state information in the abstract models, which are unable to incorporate the physical observable parameters from buildings (e.g., building structures, locations of doors and windows). Second, the measurements from the real world do not always match consistently with the outcomes of abstract models. The lack of physical structural information impedes the calibration of such models. Third, the dynamic process in an abstract model does not correspond to the true behavior of the physical world. As a result, the discrepancies of abstract models can grow considerably over time.
On the contrary, self-aware building systems aim to acquire dynamic faithful knowledge of buildings and learn to control themselves adaptively. Specifically, a self-aware building system can perform the following processes in an autonomous manner :
Sensing: Gathering environmental data, such as occupants’ behavioral information through various sensors, as well as temperature, humidity and external weather information. Its purpose is to acquire the knowledge of the current system state of a building.
Recognition: Capturing the physical model of a building by measuring the structures and recognizing the materials. Furthermore, the configurations of HVAC system may be inferred. Its purpose is to create an accurate physical building model.
Simulation: Simulating the dynamics of system behavior (e.g., thermal flow behavior) under various parameters. The accuracy of simulation can be improved by a better physical building model and extensive knowledge of the system state. The simulation should be performed in real-time.
Prediction: Forecasting future environmental data, such as occupants’ behavior and external weather.
Optimization: Deciding the system actions to maximize certain desired objectives (e.g., energy efficiency, comfort).
Diagnosis: Identifying abnormal behavior and adaptively correcting system states.
Control: Applying the actuations of systems in real time.
These processes interact with each other in a way as visualized in Fig. 1. A fully self-aware building system is expected to provide autonomy to these processes, with minimal human assistance. Although automating some of the processes (e.g., control and optimization) have been studied extensively, other processes (e.g., recognition) pose new challenges. This paper aims to shed light particularly on automated recognition for self-aware buildings.
Ii-a Recognition for Self-aware Buildings
To enable self-aware buildings, it is desirable to acquire the following aspects of information of a building:
Structures: Including the dimension and geometry of the building interior.
Materials: Including the types of walls (e.g., cement, wood) and windows (e.g., shaded or not), as well as insulation and conductance properties of built materials.
HVAC Systems: Including the capacity of heating/cooling power and locations of HVAC ducts.
Geography: Including orientation, environmental shading of the building.
While the aforementioned information can be provided by the explicit assistance of users (e.g., constructing 3D model from the BIM models or AutoCAD diagrams of the building, etc.), we aim to develop systems that implicitly gather the information with minimal human assistance.
Iii Augmented Reality
This paper explores the possibilities of using augmented reality, an interactive technology with the physical world by overlaying digital information, as an enabling tool for self-aware buildings. The extensive user-environment interactions in augmented reality allow us to implicitly capture the structures of physical buildings accurately, which can be used for real-time building simulations.
To illustrate the concept, we present several user interfaces of an augmented reality enabled building management system in Fig. 2. For example, the system will automatically recognize the dimension and geometry of building interior, as visualized in Fig. 2(a). It will identify the locations and dimensions of windows and doors. Also, it will detect the orientation of a building by magnetic sensing. Hence, a faithful representation of building will be acquired for the construction of physical building model. On the other hand, to increase users’ awareness of the energy efficiency of HVAC units, the system will overlay simulated thermal flow information with the physical view of the building, as visualized in Fig. 2(b). Furthermore, users are able to input and modify the information used by building management system through an intuitive interactive user interface. For example, users can provide feedback on occupancy behavior, as visualized in Fig. 2(c). User input information can improve the system knowledge, and enable a personalized system to address the needs of specific users.
While there are other options to capture physical building models (e.g., LIDAR), augmented reality is more convenient and cost-effective, providing rich user-building interaction opportunities.
Iii-a Project Tango Platform
Augmented reality has been a mature technology, with an increasing number of available platforms and commercial products, and emerging new developments in software systems and hardware sensors. Among the state-of-the-art augmented reality platforms, Project Tango is a popular option developed by Google, which will be deployed extensively on enhanced Android tablets or smartphones with depth sensors that are able to acquire 3D information of a physical world in indoor built environments [10, 11, 12].
Project Tango platform integrates several technologies of augmented reality, such as depth-sensing based on infra-red light, motion tracking camera, and environmental landmark learning . We tested on the first commercial Tango phone (Lenovo Phab pro 2 shown in Fig. 3), which shows good accuracy of capturing 3D structures in indoor built environments. A Tango phone provides a collection of pre-installed mobile apps for typical augmented reality experiences, such as Measure app (for remote measurement of object dimensions), Constructor app (for capturing 3D scenes) and several augmented reality games. Google also provides specified Tango SDK for advanced augmented reality app development.
Iv System Prototype
In this section, we present a system prototype incorporating augmented reality into building management system. In particular, we demonstrate our system for creating an accurate 3D building model by recognizing the building structures, and then inferring the building material properties and certain HVAC system parameters.
Iv-a Recognizing Building Structures
We first capture the 3D scene of a room by Tango Constructor app, which is able to capture approximate building spatial structures and texture information (depicted in Fig. 4(a)). Then, we convert the 3D scene to a 3D point cloud dataset (depicted in Fig. 4
(b)) for advanced processing in MATLAB, which has extensive Computer Vision System Toolbox. To obtain structural information (i.e., planes corresponding to walls and floor) in the 3D point cloud dataset, we employ the M-estimator Sample Consensus (MSAC) algorithm. MSAC is a modified version of the Ransom Sample Consensus (RANSAC)  algorithm for improved surface detection. MSAC is available in MATLAB as function pcfitplane().
The result of plane detection by MSAC on point cloud dataset is depicted in Fig. 4(c). Initially, MSAC extracts the floor dimensions by detecting the horizontal plane in the point cloud. Then, MSAC is applied iteratively to the point cloud dataset until all vertical planes are detected. Each vertical plane whose height is within a certain threshold of the maximum detected height is labeled as a wall. The x-axis, y-axis, and z-axis respectively represent the length, width, and height of the room that were detected by MSAC. Finally, the extracted geometry will be used to construct a 3D model for building simulation in EnergyPlus (depicted in Fig. 4(d)).
Iv-B Inferring Building Materials & HVAC
Unlike the physical structures of a building that can be visually perceptible, the building materials are more difficult to be discerned. Although one may use advanced techniques like machine learning and image recognition, we employ a simpler method in this paper. With the accurate 3D models of buildings constructed based on augmented reality technology, it is possible to infer the building material properties and certain HVAC system parameters based on iterative comparisons between simulations (with appropriate calibrations of simulation model parameters) and empirical observations of the real-world building behavior.
The basic idea is that we first identify a collection of key parameters in the building simulator (see Table I), and then calibrate these parameters to match as close as possible with the observed thermal response measurements under various external weather conditions and HVAC operations. We employ EnergyPlus , a popular building simulation program. EnergyPlus provides detailed building models, incorporating building structures (e.g., locations of doors and windows, etc.) and material properties, which makes them easier to calibrate than first-principle or non-linear models. By iterative calibration of the parameters of building simulation model, we seek to minimize the discrepancy between the actual and simulated indoor temperature values. The calibration of simulation model is performed as follows:
The calibration algorithm sets the initial material properties and HVAC system parameters.
EnergyPlus runs a simulation with adjusted parameters and returns simulated indoor temperature.
The algorithm tracks the changes in the discrepancy between actual and simulated temperature and applies gradient descent to adjust the respective parameters. The process repeats iteratively until the discrepancy is within a certain threshold.
A final calibrated building model will provide an estimation of the building material properties and certain HVAC system parameters. This calibration process is carried out using co-simulation, where the EnergyPlus building model is coupled with a calibration algorithm implemented in Python using BCVTB (Building Controls Virtual Test Bed) as software interface . It is worth noting that gradient descent may arrive at parameter values that agree with the current set of observations but these might not be the true values of the parameters. For this reason, the calibration algorithm continuously checks the model in the background, and re-calibrates it whenever the discrepancy between the actual and simulated temperature exceeds the threshold. This way, the model always provides a good estimation of the parameter values.
A demonstration using the 3D model from Fig. 4(d) is illustrated in Fig. 5, showing the thermal response of the initial uncalibrated model and of the iteratively calibrated model. After calibration, the simulated temperature of the calibrated EnergyPlus model can closely match the actual temperature. Fig. 5 also depicts the observed status of HVAC system during different times of the day, as in the bottom strips. Finally, Table I presents the inferred parameter values of building simulation model, including building material properties and HVAC system.
|Air Flow Rate||0.384m/sec|
This paper first presented a notion of self-aware building that is capable of recognizing of building structures and properties to construct accurate simulation models. Then, we demonstrated a system prototype using augmented reality to capture the build structures and then to infer building materials and HVAC system parameters. In future work, we will implement a full system that incorporates simulation visualization, user feedback elicitation, and automatic real-time simulation-guided control for HVAC operations.
-  D. Van Krevelen and R. Poelman, “A survey of augmented reality technologies, applications and limitations,” International journal of virtual reality, vol. 9, no. 2, p. 1, 2010.
-  M. Aftab, S. C.-K. Chau, and P. Armstrong, “Smart air-conditioning control by wireless sensors: An online optimization approach,” in Proceedings of the Fourth International Conference on Future Energy Systems, ser. e-Energy ’13. New York, NY, USA: ACM, 2013, pp. 225–236. [Online]. Available: http://doi.acm.org/10.1145/2487166.2487192
-  J. Hu and P. Karava, “A state-space modeling approach and multi-level optimization algorithm for predictive control of multi-zone buildings with mixed-mode cooling,” Building and Environment, vol. 80, pp. 259 – 273, 2014.
-  R. Balan, J. Cooper, K.-M. Chao, S. Stan, and R. Donca, “Parameter identification and model based predictive control of temperature inside a house,” Energy and Buildings, vol. 43, no. 2 - 3, pp. 748 – 758, 2011.
-  D. B. Crawley, J. W. Hand, M. Kummert, and B. T. Griffith, “Contrasting the capabilities of building energy performance simulation programs,” Building and Environment, vol. 43, no. 4, pp. 661 – 673, 2008.
-  A. A. Diakité and S. Zlatanova, “First experiments with the tango tablet for indoor scanning,” ISPRS Annals of Photogrammetry, Remote Sensing and Spatial Information Sciences, pp. 67–72, 2016.
-  T. Schöps, T. Sattler, C. Häne, and M. Pollefeys, “3d modeling on the go: Interactive 3d reconstruction of large-scale scenes on mobile devices,” in 3D Vision (3DV), 2015 International Conference on. IEEE, 2015, pp. 291–299.
-  D. Van Krevelen and R. Poelman, “Augmented reality: Technologies, applications, and limitations,” 2007.
-  A. Mahdavi, R. Brahme, and S. Gupta, “Self-aware buildings: a simulation-based approach,” 2001.
-  X. Blanco, “Google’s project tango tablet is ripe for development,” 2015. [Online]. Available: https://www.cnet.com/products/google-project-tango/preview/
-  Lenovo, “Lenovo phab 2 pro,” 2018. [Online]. Available: https://www3.lenovo.com/in/en/tablets/android-tablets/tablet-phab-series/Lenovo-Phab-2-Pro/p/WMD00000220
-  Asus, “Zenfone ar,” 2018. [Online]. Available: https://www.asus.com/in/Phone/ZenFone-AR-ZS571KL/
-  P. H. Torr and A. Zisserman, “Mlesac: A new robust estimator with application to estimating image geometry,” Computer Vision and Image Understanding, vol. 78, no. 1, pp. 138–156, 2000.
-  M. A. Fischler and R. C. Bolles, “Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography,” Communications of the ACM, vol. 24, no. 6, pp. 381–395, 1981.
-  M. Aftab, C. Chen, C.-K. Chau, and T. Rahwan, “Automatic hvac control with real-time occupancy recognition and simulation-guided model predictive control in low-cost embedded system,” Energy and Buildings, vol. 154, pp. 141 – 156, 2017. [Online]. Available: http://www.sciencedirect.com/science/article/pii/S0378778817305091
-  D. B. Crawley, L. K. Lawrie, F. C. Winkelmann, W. Buhl, Y. Huang, C. O. Pedersen, R. K. Strand, R. J. Liesen, D. E. Fisher, M. J. Witte, and J. Glazer, “Energyplus: a new-generation building energy simulation program,” Energy and Buildings, vol. 33, no. 4, pp. 319 – 331, 2001.
-  M. Wetter, “Co-simulation of building energy and control systems with the building controls virtual test bed,” Journal of Building Performance Simulation, vol. 4, no. 3, pp. 185–203, 2011.