Real-Time Energy Audit of Built Environments: Simultaneous Localization and Thermal Mapping
Publication: Journal of Infrastructure Systems
Volume 24, Issue 3
Abstract
Leveraging thermography for managing built environments has become prevalent as a robust tool for detecting, analyzing, and reporting their performance in a nondestructive manner. Despite many documented benefits of thermographic inspection for better characterizing the conditions of built environments, current thermographic inspections still have several inefficiencies. Inspectors typically collect and store large numbers of thermal images or long-sequence thermal videos to support decision making on the maintenance and rehabilitation of built environments. However, more importantly, these large-scale visual data are typically unordered and not localized (a term used to define a process of finding locations or relative positions of a camera). This paper proposes and compares two approaches for simultaneous localization and thermal mapping. These methods estimate a camera’s pose and map the environment in three dimensions in real-time. Case studies using an off-the-shelf hardware configuration using solely thermal cameras and an author-customized configuration using both thermal and red, green, and blue (RGB) cameras are conducted and the results are compared with a well-established offline three-dimensional mapping tool. The results show that combining information from RGB and thermal cameras provides significant benefits to real-time localization compared with using solely thermal cameras and that the proposed real-time methods have localization performance comparable to the offline tool.
Get full access to this article
View all available purchase options and get full access to this article.
Acknowledgments
The authors gratefully acknowledge the support of NVIDIA Corporation with the donation of the Tesla K40 graphics processing units used for this research. The authors thank Khashayar Asadi and Zexi Chen for providing computing equipment and for valuable input in the object-oriented design of ORB-SLAM2 modifications, respectively. Any opinions, findings, conclusions, or recommendations presented in this paper are those of the authors and do not reflect the views of NVIDIA or the individuals acknowledged previously.
References
An, Y.-K., Y. Kwon, and H. Sohn. 2013. “Noncontact laser ultrasonic crack detection for plates with additional structural complexities.” Struct. Health Monit. 12 (5–6): 522–538. https://doi.org/10.1177/1475921713500515.
Borrmann, D., A. Nüchter, M. Ðakulović, I. Maurović, I. Petrović, D. Osmanković, and J. Velagić. 2014. “A mobile robot based system for fully automated thermal 3D mapping.” Adv. Eng. Inf. 28 (4): 425–440. https://doi.org/10.1016/j.aei.2014.06.002.
Bouguet, J.-Y. 2004. Camera calibration toolbox for Matlab. Pasadena, CA: Caltech Technical Report.
Cerdeira, F., M. Vázquez, J. Collazo, and E. Granada. 2011. “Applicability of infrared thermography to the study of the behaviour of stone panels as building envelopes.” Energy Build. 43 (8): 1845–1851. https://doi.org/10.1016/j.enbuild.2011.03.029.
Cho, Y. K., Y. Ham, and M. Golpavar-Fard. 2015. “3D as-is building energy modeling and diagnostics: A review of the state-of-the-art.” Adv. Eng. Inf. 29 (2): 184–195. https://doi.org/10.1016/j.aei.2015.03.004.
Dubrofsky, E. 2009. “Homography estimation.” Ph.D. thesis, Univ. of British Columbia.
Endres, F., J. Hess, J. Sturm, D. Cremers, and W. Burgard. 2014. “3-D mapping with an RGB-D camera.” IEEE Trans. Rob. 30 (1): 177–187. https://doi.org/10.1109/TRO.2013.2279412.
FHWA (Federal Highway Administration). 2006. Status of the nation’s highways, bridges, and transit: Conditions and performance. Washington, DC: US Dept. of Transportation.
Fox, M., D. Coley, S. Goodhew, and P. de Wilde. 2014. “Thermography methodologies for detecting energy related building defects.” Renewable Sustainable Energy Rev. 40: 296–310. https://doi.org/10.1016/j.rser.2014.07.188.
Furukawa, Y., and J. Ponce. 2010. “Accurate, dense, and robust multiview stereopsis.” IEEE Trans. Pattern Anal. Mach. Intell. 32 (8): 1362–1376. https://doi.org/10.1109/TPAMI.2009.161.
Golparvar-Fard, M., and Y. Ham. 2014. “Automated diagnostics and visualization of potential energy performance problems in existing buildings using energy performance augmented reality models.” J. Comput. Civ. Eng. 28 (1): 17–29. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000311.
Ham, Y., and M. Golparvar-Fard. 2013. “An automated vision-based method for rapid 3D energy performance modeling of existing buildings using thermal and digital imagery.” Adv. Eng. Inf. 27 (3): 395–409. https://doi.org/10.1016/j.aei.2013.03.005.
Ham, Y., and M. Golparvar-Fard. 2014. “3D visualization of thermal resistance and condensation problems using infrared thermography for building energy diagnostics.” Visual. Eng. 2 (1): 12. https://doi.org/10.1186/s40327-014-0012-0.
Ham, Y., and M. Golparvar-Fard. 2015. “Mapping actual thermal properties to building elements in gbXML-based BIM for reliable building energy performance modeling.” Autom. Constr. 49: 214–224. https://doi.org/10.1016/j.autcon.2014.07.009.
Ham, Y., H. Yoon, and B. F. Spencer Jr. 2016. “Improving situational awareness of the as-is building conditions through multi-modal sensing and analytics using thermal camera-equipped smartphones.” In Proc., Construction Research Congress 2016, 898–908. Reston, VA: ASCE.
Heikkila, J., and O. Silven. 1997. “A four-step camera calibration procedure with implicit image correction.” In Proc., 1997 IEEE Computer Society Conf. on Computer Vision and Pattern Recognition, 1997, 1106–1112. Piscataway, NJ: IEEE.
Hiasa, S. 2016. “Investigation of infrared thermography for subsurface damage detection of concrete structures.” Ph.D. thesis, Univ. of Central Florida.
Horn, B. K. 1987. “Closed-form solution of absolute orientation using unit quaternions.” JOSA A 4 (4): 629–642. https://doi.org/10.1364/JOSAA.4.000629.
Kylili, A., P. A. Fokaides, P. Christou, and S. A. Kalogirou. 2014. “Infrared thermography (IRT) applications for building diagnostics: A review.” Appl. Energy 134: 531–549. https://doi.org/10.1016/j.apenergy.2014.08.005.
Lee, J.-S. 1980. “Digital image enhancement and noise filtering by use of local statistics.” IEEE Trans. Pattern Anal. Mach. Intell. PAMI-2 (2): 165–168. https://doi.org/10.1109/TPAMI.1980.4766994.
Lowe, D. G. 1999. “Object recognition from local scale-invariant features.” In Vol. 2 of Proc., 7th IEEE Int. Conf. on Computer Vision, 1999, 1150–1157. Piscataway, NJ: IEEE.
MarketsandMarkets. 2015. “Infrared thermography market in building inspection by application (energy auditing, building performance, & others), building type (residential, commercial, & others), platform (hardware & software), & geography—Forecast to 2020.” Accessed June 1, 2015. http://www.marketsandmarkets.com/Market-Reports/thermography-building-inspection-market-194389420.html.
Mur-Artal, R., J. M. M. Montiel, and J. D. Tardos. 2015. “ORB-SLAM: A versatile and accurate monocular SLAM system.” IEEE Trans. Rob. 31 (5): 1147–1163. https://doi.org/10.1109/TRO.2015.2463671.
Mur-Artal, R., and J. D. Tardós. 2017. “ORB-SLAM2: An open-source SLAM system for monocular, stereo and RGB-D cameras.” IEEE Trans. Rob. 33 (5): 1255–1262. https://doi.org/10.1109/TRO.2017.2705103.
Nagatani, K., K. Otake, and K. Yoshida. 2014. Three-dimensional thermography mapping for mobile rescue robots, 49–63. Berlin, Germany: Springer.
Omondi, B., D. G. Aggelis, H. Sol, and C. Sitters. 2016. “Improved crack monitoring in structural concrete by combined acoustic emission and digital image correlation techniques.” Struct. Health Monit. 15 (3): 359–378. https://doi.org/10.1177/1475921716636806.
Plum, R., and T. Ummenhofer. 2013. “Use of ultrasound excited thermography applied to massive steel components: Emerging crack detection methodology.” J. Bridge Eng. 18 (6): 455–463. https://doi.org/10.1061/(ASCE)BE.1943-5592.0000355.
Ramachandra, B., J. Monroe, P. Nawathe, K. Han, Y. Ham, and R. R. Vatsavai. 2017. “Energy audit of built environments through simultaneous localization and thermal mapping.” In Proc., Int. Conf. on Maintenance and Rehabilitation of Constructed Infrastructure Facilities. Iowa City, IA: International Society for Maintenance and Rehabilitation of Transportation Infrastructures.
Rosten, E., and T. Drummond. 2006. “Machine learning for high-speed corner detection.” In European Conf. on Computer Vision, 430–443. Berlin, Germany: Springer.
Rublee, E., V. Rabaud, K. Konolige, and G. Bradski. 2011. “ORB: An efficient alternative to SIFT or SURF.” In Proc., IEEE Int. Conf. on Computer Vision (ICCV), 2011, 2564–2571. Piscataway, NJ: IEEE.
Shi, P., X. Fan, J. Ni, and G. Wang. 2016. “A detection and classification approach for underwater dam cracks.” Struct. Health Monit. 15 (5): 541–554. https://doi.org/10.1177/1475921716651039.
Snavely, N., S. M. Seitz, and R. Szeliski. 2006. “Photo tourism: Exploring photo collections in 3D.” ACM Trans. Graphics 25 (3): 835–846. https://doi.org/10.1145/1141911.1141964.
Thrun, S., and J. J. Leonard. 2008. “Simultaneous localization and mapping.” In Springer handbook of robotics, 871–889. New York, NY: Springer.
Triggs, B., P. F. McLauchlan, R. I. Hartley, and A. W. Fitzgibbon. 1999. “Bundle adjustment—A modern synthesis.” In International workshop on vision algorithms, 298–372. New York, NY: Springer.
Vaghefi, K., H. D. M. Melo e Silva, D. Harris, and T. Ahlborn. 2011. “Application of thermal IR imagery for concrete bridge inspection.” In Proc., PCI National Bridge Conference (PCI/NBC ‘11), Salt Lake City, UT.
Vidas, S., P. Moghadam, and M. Bosse. 2013. “3D thermal mapping of building interiors using an RGB-D and thermal camera.” In Proc., 2013 IEEE Int. Conf. on Robotics and Automation, 2311–2318. Piscataway, NJ: IEEE.
Vidas, S., P. Moghadam, and S. Sridharan. 2015. “Real-time mobile 3D temperature mapping.” IEEE Sens. J. 15 (2): 1145–1152. https://doi.org/10.1109/JSEN.2014.2360709.
Vidas, S., and S. Sridharan. 2012. “Hand-held monocular slam in thermal-infrared.” In Proc., 12th Int. Conf. on Control Automation Robotics Vision (ICARCV), 859–864. Piscataway, NJ: IEEE.
Washer, G. 2013. Field evaluation of thermographic bridge concrete inspection techniques. Rolla, MO: Center for Transportation Infrastructure and Safety, Missouri Univ. of Science and Technology.
Washer, G., N. Bolleni, and R. Fenwick. 2010. “Thermographic imaging of subsurface deterioration in concrete bridges.” Transp. Res. Rec. 2201: 27–33. https://doi.org/10.3141/2201-04.
Winkler, J., G. Fischer, and C. T. Georgakis. 2014. “Measurement of local deformations in steel monostrands using digital image correlation.” J. Bridge Eng. 19 (10): 04014042. https://doi.org/10.1061/(ASCE)BE.1943-5592.0000615.
Wu, C. 2011. VisualSFM: A visual structure from motion system. Seattle, WA: Univ. of Washington.
Information & Authors
Information
Published In
Journal of Infrastructure Systems
Volume 24 • Issue 3 • September 2018
Copyright
©2018 American Society of Civil Engineers.
History
Received: May 15, 2017
Accepted: Mar 9, 2018
Published online: Jun 13, 2018
Published in print: Sep 1, 2018
Discussion open until: Nov 13, 2018
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.