Multiobjective Optimization of Reality Capture Plans for Computer Vision–Driven Construction Monitoring with Camera-Equipped UAVs
Publication: Journal of Computing in Civil Engineering
Volume 36, Issue 5
Abstract
The exponential growth in reality capture and Building Information Modeling (BIM)-enabled construction workflows has created a surge in computer vision–driven solutions that automatically model and compare as-built conditions against BIM, offering project teams actionable insight into construction progress and quality. Despite their significant impact, the performance of these methods heavily relies on the accuracy and completeness of the reality capture. In addition, and especially in the case of reality captures conducted with camera-equipped unmanned aerial vehicles (UAVs), operational requirements—including battery capacity and operator’s line of sight (LOS)—should be carefully considered for safe flight execution. Accounting for these technical and operational requirements during reality capture planning requires expertise. In addition, it involves a significant amount of manual tweaking that does not scale well to ongoing changes due to progress on construction projects. To address these limitations, this paper presents a novel multiobjective optimization method to improve reality capture plans aiming to maximize (1) visual coverage of the monitored structure, (2) redundant observation of the structure’s components in the collected frames, (3) resolution of the structure’s elements in the captured data, (4) canonical camera viewpoints to the structure’s topology, and (5) stability of three-dimensional (3D) reconstruction algorithms used to process the data altogether, while (6) reducing the data collection duration. The objectives are also set to meet other technical and operational requirements, particularly for camera-equipped UAVs. Furthermore, a client-server system architecture is presented to visualize, simulate, and optimize reality capture missions in a web-based 3D environment using four-dimensional (4D) BIM to indicate the as-planned expected changes. Five conducted experiments using real-world data demonstrated the method’s capability to enhance the quality of user-created reality capture plans. The optimization process resulted in a 7.65% improvement in visual coverage, 30.89% enhancement in the structure’s resolution, and 8.95% more stable 3D reconstruction while ensuring the flight paths meet operational requirements.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
Some or all data, models, or code that support the findings of this study are available from the corresponding author upon reasonable request. This includes the reality capture planning code, the evaluation model, the optimization model, and the initial and optimized reality capture plans.
Acknowledgments
The authors would like to acknowledge the financial support of National Science Foundation (NSF) Grant Nos. 1446765 and 1544999. The authors also appreciate the support of Reconstruct Inc. and all other construction and construction technology companies who offered the authors access to real-world project data. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the authors. They do not necessarily reflect the view of the NSF, industry partners, or professionals mentioned above.
References
Aryan, A., F. Bosché, and P. Tang. 2021. “Planning for terrestrial laser scanning in construction: A review.” Autom. Constr. 125 (May): 103551. https://doi.org/10.1016/j.autcon.2021.103551.
Asadi, K., A. Kalkunte Suresh, A. Ender, S. Gotad, S. Maniyar, S. Anand, M. Noghabaei, K. Han, E. Lobaton, and T. Wu. 2020. “An integrated UGV-UAV system for construction site data collection.” Autom. Constr. 112 (Apr): 103068. https://doi.org/10.1016/j.autcon.2019.103068.
Baik, H., and J. Valenzuela. 2019. “Unmanned aircraft system path planning for visually inspecting electric transmission towers.” J. Intell. Robot. Syst., Theory Appl. 95 (3–4): 1097–1111. https://doi.org/10.1007/s10846-018-0947-9.
Bircher, A., K. Alexis, M. Burri, P. Oettershagen, S. Omari, T. Mantel, and R. Siegwart. 2015. “Structural inspection path planning via iterative viewpoint resampling with application to aerial robotics.” In Proc., IEEE Int. Conf. on Robotics and Automation (ICRA), 6423–6430. Piscataway, NJ: IEEE. https://doi.org/10.1109/ICRA.2015.7140101.
Bosché, F., A. Forster, and E. Valero. 2015. 3D surveying technologies and applications: Point clouds and beyond. Edinburgh, Scotland: Heriot-Watt Univ.
Braun, A., S. Tuttas, A. Borrmann, and U. Stilla. 2020. “Improving progress monitoring by fusing point clouds, semantic data and computer vision.” Autom. Constr. 116 (Aug): 103210. https://doi.org/10.1016/j.autcon.2020.103210.
Chen, M., E. Koc, Z. Shi, and L. Soibelman. 2018. “Proactive 2D model-based scan planning for existing buildings.” Autom. Constr. 93 (Sep): 165–177. https://doi.org/10.1016/j.autcon.2018.05.010.
Daftry, S., C. Hoppe, and H. Bischof. 2015. “Building with drones: Accurate 3D facade reconstruction using MAVs.” In Proc., IEEE Int. Conf. on Robotics and Automation (ICRA), 3487–3494. Piscataway, NJ: IEEE. https://doi.org/10.1109/ICRA.2015.7139681.
Deb, K., A. Pratap, S. Agarwal, and T. Meyarivan. 2002. “A fast and elitist multiobjective genetic algorithm: NSGA-II.” IEEE Trans. Evol. Comput. 6 (2): 182–197. https://doi.org/10.1109/4235.996017.
Degol, J., M. Golparvar-Fard, and D. Hoiem. 2016. “Geometry-informed material recognition.” In Proc., IEEE Computer Society Conf. on Computer Vision and Pattern Recognition, 1554–1562. Piscataway, NJ: IEEE. https://doi.org/10.1109/CVPR.2016.172.
Degol, J., J. Y. Lee, R. Kataria, D. Yuan, T. Bretl, and D. Hoiem. 2018. “FEATS: Synthetic feature tracks for structure from motion evaluation.” In Proc., Int. Conf. on 3D Vision, 3DV 2018, 352–361. Verona, Italy: Institute of Electrical and Electronics Engineers.
Denisov, E., A. Sagitov, K. Yakovlev, K. L. Su, M. Svinin, and E. Magid. 2019. “Towards total coverage in autonomous exploration for UGV in 2.5D dense clutter environment.” In Proc., 16th Int. Conf. on Informatics in Control, Automation and Robotics: ICINCO 2019, 2(Icinco), 409–416. Berlin: Springer. https://doi.org/10.5220/0007923304090416.
Figueira, J., S. Greco, and M. Ehrogott. 2005. Multiple criteria decision analysis: State of the art surveys. New York: Springer.
Fortin, F.-A., F.-M. De Rainville, M.-A. Gardner, M. Parizeau, and C. Gagné. 2012. “DEAP: Evolutionary algorithms made easy.” J. Mach. Learn. Res. 13 (1): 2171–2175.
Freimuth, H., and M. König. 2018. “Planning and executing construction inspections with unmanned aerial vehicles.” Autom. Constr. 96 (Dec): 540–553. https://doi.org/10.1016/j.autcon.2018.10.016.
Giorgini, M., S. Marini, R. Monica, and J. Aleotti. 2019. “Sensor-based optimization of terrestrial laser scanning measurement setup on GPU.” IEEE Geosci. Remote Sens. Lett. 16 (9): 1452–1456. https://doi.org/10.1109/LGRS.2019.2899681.
Hamieh, A., A. Ben Makhlouf, B. Louhichi, and D. Deneux. 2020. “A BIM-based method to plan indoor paths.” Autom. Constr. 113 (May): 103120. https://doi.org/10.1016/j.autcon.2020.103120.
Hamledari, H., S. Sajedi, B. McCabe, and M. Fischer. 2021. “Automation of inspection mission planning using 4D BIMs and in support of unmanned aerial vehicle–based data collection.” J. Constr. Eng. Manage. 147 (3): 04020179. https://doi.org/10.1061/(ASCE)CO.1943-7862.0001995.
Han, K., J. Degol, and M. Golparvar-Fard. 2017. “Geometry- and appearance-based reasoning of construction progress monitoring.” J. Constr. Eng. Manage. 144 (2): 04017110. https://doi.org/10.1061/(ASCE)CO.1943-7862.0001428.
Han, K. K., and M. Golparvar-Fard. 2015. “Appearance-based material classification for monitoring of operation-level construction progress using 4D BIM and site photologs.” Autom. Constr. 53 (May): 44–57. https://doi.org/10.1016/j.autcon.2015.02.007.
Ibrahim, A., and M. Golparvar-Fard. 2019. “4D BIM based optimal flight planning for construction monitoring applications using camera-equipped UAVs.” In Computing in civil engineering 2019, 217–224. Reston, VA: ASCE.
Ibrahim, A., M. Golparvar-Fard, T. Bretl, and K. El-Rayes. 2017a. “Model-driven visual data capture on construction sites: Method and metrics of success.” In Proc., Int. Workshop for Computing in Civil Engineering (IWCCE 2017), 109–116. Reston, VA: ASCE. https://doi.org/10.1061/9780784480847.014.
Ibrahim, A., M. Golparvar-Fard, and K. El-Rayes. 2021. “Demo for multiobjective optimization of reality capture plans for computer vision construction monitoring with camera-equipped UAVs.” Vimeo. Accessed October 26, 2021. https://vimeo.com/639266815.
Ibrahim, A., M. Golparvar-Fard, and K. El-Rayes. 2022. “Metrics and methods for evaluating model-driven reality capture plans.” Comput.-Aided Civ. Infrastruct. Eng. 37 (1): 55–72. https://doi.org/10.1111/mice.12693.
Ibrahim, A., D. Roberts, M. Golparvar-Fard, and T. Bretl. 2017b. “An interactive model-driven path planning and data capture system for camera-equipped aerial robots on construction sites.” In Proc., Int. Workshop for Computing in Civil Engineering (IWCCE 2017), 117–124. Reston, VA: ASCE. https://doi.org/10.1061/9780784480847.015.
Jan, G. E., C. Luo, H.-T. Lin, and K. Fung. 2019. “Complete area coverage path-planning with arbitrary shape obstacles.” Autom. Control Eng. 7 (2): 80–85. https://doi.org/10.18178/joace.7.2.80-85.
Jia, F., and D. D. Lichti. 2019. “A model-based design system for terrestrial laser scanning networks in complex sites.” Remote Sens. 11 (15): 1749. https://doi.org/10.3390/rs11151749.
Jiang, W., Y. Zhou, L. Ding, C. Zhou, and X. Ning. 2020. “UAV-based 3D reconstruction for hoist site mapping and layout planning in petrochemical construction.” Autom. Constr. 113 (May): 103137. https://doi.org/10.1016/j.autcon.2020.103137.
Kim, P., J. Park, Y. K. Cho, and J. Kang. 2019. “UAV-assisted autonomous mobile robot navigation for as-is 3D data collection and registration in cluttered environments.” Autom. Constr. 106 (Oct): 102918. https://doi.org/10.1016/j.autcon.2019.102918.
Lin, J. J., and M. Golparvar-Fard. 2020. “Construction progress monitoring using cyber-physical systems.” In Cyber-physical systems in the built environment, 6387. Cham, Switzerland: Springer.
Lin, J. J., A. Ibrahim, S. Sarwade, and M. Golparvar-Fard. 2021. “Bridge inspection with aerial robots: Automating the entire pipeline of visual data capture, 3D mapping, defect detection, analysis, and reporting.” J. Comput. Civ. Eng. 35 (2): 04020064. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000954.
Lin, W. Y. 2020. “Automatic generation of high-accuracy stair paths for straight, spiral, and winder stairs using IFC-based models.” ISPRS Int. J. Geo-Inf. 9 (4): 22–26. https://doi.org/10.3390/ijgi9040215.
Lindner, S., C. Garbe, and K. Mombaur. 2019. “Optimization based multi-view coverage path planning for autonomous structure from motion recordings.” IEEE Rob. Autom. Lett. 4 (4): 3278–3285. https://doi.org/10.1109/LRA.2019.2926216.
Lowe, D. G. 2004. “Distinctive image features from scale invariant keypoints.” Int. J. Comput. Vision 60 (2): 91–110. https://doi.org/10.1023/B:VISI.0000029664.99615.94.
Majeed, A., and S. Lee. 2019. “A new coverage flight path planning algorithm based on footprint sweep fitting for unmanned aerial vehicle navigation in urban environments.” Appl. Sci. (Switzerland) 9 (7): 1–17. https://doi.org/10.3390/app9071470.
Marica, V., C. D. Curiac, P. Y. M. Quentel, C. S. Stangaciu, and M. V. Micea. 2019. “Static coverage path planning for UAVs with conical field of view when monitoring rectangular ground areas.” In Proc., 23rd Int. Conf. on System Theory, Control and Computing, ICSTCC 2019, 510–514. New York: IEEE. https://doi.org/10.1109/ICSTCC.2019.8885559.
Mora, O. E., A. Suleiman, J. Chen, D. Pluta, M. H. Okubo, and R. Josenhans. 2019. “Comparing sUAS photogrammetrically-derived point clouds with GNSS measurements and terrestrial laser scanning for topographic mapping.” Drones 3 (3): 64. https://doi.org/10.3390/drones3030064.
Ortega, A. G., J. Reyes-Muñoz, M. McGee, A. Choudhuri, and A. Flores-Abad. 2020. “Drone inspection flight path generation from 3d cad models: Power plant boiler case study.” In Proc., AIAA Scitech 2020 Forum, 1–16. Reston, VA: American Institute of Aeronautics. https://doi.org/10.2514/6.2020-1091.
Phung, M. D., C. H. Quach, T. H. Dinh, and Q. Ha. 2017. “Enhanced discrete particle swarm optimization path planning for UAV vision-based surface inspection.” Autom. Constr. 81 (Sep): 25–33. https://doi.org/10.1016/j.autcon.2017.04.013.
Qu, C., W. Gai, J. Zhang, and M. Zhong. 2020. “A novel hybrid grey wolf optimizer algorithm for unmanned aerial vehicle (UAV) path planning.” Appl. Soft Comput. 89 (Apr): 1–12. https://doi.org/10.1016/j.asoc.2020.106099.
Ramanagopal, M. S., A. P. V. Nguyen, and J. Le Ny. 2018. “A motion planning strategy for the active vision-based mapping of ground-level structures.” IEEE Trans. Autom. Sci. Eng. 15 (1): 356–368. https://doi.org/10.1109/TASE.2017.2762088.
Szeliski, R. 2020. Computer vision: Algorithms and applications. New York: Springer.
Theile, M., H. Bayerlein, R. Nai, D. Gesbert, and M. Caccamo. 2020. “UAV coverage path planning under varying power constraints using deep reinforcement learning.” In Proc., IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS). New York: IEEE. https://doi.org/10.1109/IROS45743.2020.9340934.
Wang, L., J. Kan, J. Guo, and C. Wang. 2019. “Improved ant colony optimization for ground robot 3D path planning.” In Proc., 2018 Int. Conf. on Cyber-Enabled Distributed Computing and Knowledge Discovery, CyberC 2018, 106–112. New York: IEEE.
Xu, Z., R. Kang, and R. Lu. 2020. “3D reconstruction and measurement of surface defects in prefabricated elements using point clouds.” J. Comput. Civ. Eng. 34 (5): 04020033. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000920.
Zhang, C., V. S. Kalasapudi, and P. Tang. 2016. “Rapid data quality oriented laser scan planning for dynamic construction environments.” Adv. Eng. Inf. 30 (2): 218–232. https://doi.org/10.1016/j.aei.2016.03.004.
Information & Authors
Information
Published In
Journal of Computing in Civil Engineering
Volume 36 • Issue 5 • September 2022
Copyright
© 2022 American Society of Civil Engineers.
History
Received: Oct 29, 2021
Accepted: Mar 17, 2022
Published online: Jun 9, 2022
Published in print: Sep 1, 2022
Discussion open until: Nov 9, 2022
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
Cited by
- Yuxiang Zhao, Binyao Guo, Ishfaq Aziz, Mohamad Alipour, An Optimization Framework for UAS-Based Infrastructure Inspection Path Planning, Computing in Civil Engineering 2023, 10.1061/9780784485248.107, (890-898), (2024).
- Jack C. P. Cheng, Changhao Song, Xiao Zhang, Zhengyi Chen, Pose Graph Relocalization with Deep Object Detection and BIM-Supported Object Landmark Dictionary, Journal of Computing in Civil Engineering, 10.1061/JCCEE5.CPENG-5301, 37, 5, (2023).
- Abdul Hannan Qureshi, Wesam Salah Alaloul, Wong Kai Wing, Syed Saad, Muhammad Ali Musarat, Syed Ammad, Ahmed Farouk Kineber, Automated progress monitoring technological model for construction projects, Ain Shams Engineering Journal, 10.1016/j.asej.2023.102165, (102165), (2023).
- Changhao Song, Zhengyi Chen, Kai Wang, Han Luo, Jack C.P. Cheng, BIM-supported scan and flight planning for fully autonomous LiDAR-carrying UAVs, Automation in Construction, 10.1016/j.autcon.2022.104533, 142, (104533), (2022).