Construction Photo Localization in 3D Reality Models for Vision-Based Automated Daily Project Monitoring
Publication: Journal of Computing in Civil Engineering
Volume 37, Issue 6
Abstract
Recent research has focused on visualizing and analyzing massive visual data (images/videos) captured at construction sites to improve coordination, communication, and planning. One key approach is to generate reality models (point clouds) from ordered visual data using three-dimensional (3D) reconstruction pipelines and then compare them with the as-planned four-dimensional (4D) building information model (BIM). However, ordered photo collection requires a strict capture plan and happens only after a specific time interval. Additionally, the reality models demand a considerable amount of processing time. As a result, the construction project status is often unreported between the intervals. The random photos captured daily by construction practitioners from different parts of the projects are helpful in filling this void. Localizing them in the 3D reality model helps to extract valuable information on time for effective construction monitoring. This study develops a system that localizes random photos in reality models using computer vision and deep learning–based approaches. Specifically, given a set of photos and a point cloud pair, a deep learning network for a six-degrees-of-freedom (6DOF) camera pose regression is trained to take any random photo within the point cloud region and estimate its position and orientation. The network performance is enhanced through data augmentation by another generative adversarial network: the pix2pix generative adversarial network (GAN). Finally, the poses are refined through traditional vision methods such as Perspective-n-Point (PnP) pose computation with random sample consensus (RANSAC). The proposed method was evaluated on a construction site. The system could localize random images captured during construction engineers’ daily work with as low as 0.04 m position error and 0.70° orientation error. In the end, this paper indicates further applications of construction image localization in the context of progress, quality, and safety monitoring.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
Some data, models, or codes that support the findings of this study are available from the corresponding author upon reasonable request (images, point cloud models, and camera poses).
Acknowledgments
The authors would like to thank the National Science and Technology Council (NSTC), Taiwan, for supporting this research through Grant Nos. MOST-110-2222-E-002-002-MY3 and MOST-109-2622-E-002-027.
References
Abdel-Aziz, Y. I., H. Karara, and M. Hauck. 2015. “Direct linear transformation from comparator coordinates into object space coordinates in close-range photogrammetry.” Photogramm. Eng. Remote Sens. 81 (2): 103–107. https://doi.org/10.14358/PERS.81.2.103.
Acharya, D., K. Khoshelham, and S. Winter. 2019. “BIM-posenet: Indoor camera localisation using a 3D indoor model and deep learning from synthetic images.” ISPRS J. Photogramm. Remote Sens. 150 (Apr): 245–258. https://doi.org/10.1016/j.isprsjprs.2019.02.020.
Acharya, D., R. Tennakoon, S. Muthu, K. Khoshelham, R. Hoseinnezhad, and A. Bab-Hadiashar. 2022. “Single-image localisation using 3D models: Combining hierarchical edge maps and semantic segmentation for domain adaptation.” Autom. Constr. 136 (Apr): 104152. https://doi.org/10.1016/j.autcon.2022.104152.
Asadi, K., H. Ramshankar, M. Noghabaei, and K. Han. 2019. “Real-time image localization and registration with BIM using perspective alignment for indoor monitoring of construction.” J. Comput. Civ. Eng. 33 (5): 04019031. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000847.
Bae, H., M. Golparvar-Fard, and J. White. 2015. “Image-based localization and content authoring in structure-from-motion point cloud models for real-time field reporting applications.” J. Comput. Civ. Eng. 29 (4): B4014008. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000392.
Besl, P. J., and N. D. McKay. 1992. “A method for registration of 3-D shapes.” IEEE Trans. Pattern Anal. Mach. Intell. 14 (2): 239–256. https://doi.org/10.1109/34.121791.
Brilakis, I., and L. Soibelman. 2006. “Multimodal image retrieval from construction databases and model-based systems.” J. Constr. Eng. Manage. 132 (7): 777–785. https://doi.org/10.1061/(ASCE)0733-9364(2006)132:7(777).
Brilakis, I. K., and L. Soibelman. 2008. “Shape-based retrieval of construction site photographs.” J. Comput. Civ. Eng. 22 (1): 14–20. https://doi.org/10.1061/(ASCE)0887-3801(2008)22:1(14).
Chen, C., B. Wang, C. X. Lu, N. Trigoni, and A. Markham. 2020. “A survey on deep learning for localization and mapping: Towards the age of spatial machine intelligence.” Preprint, submitted June 20, 2022. http://arxiv.org/abs/2006.12567.
Diebel, J. 2006. “Representing attitude: Euler angles, unit quaternions, and rotation vectors.” Matrix 58 (15–16): 1–35.
Fang, Q., H. Li, X. Luo, C. Li, and W. An. 2020. “A sematic and prior-knowledge-aided monocular localization method for construction-related entities.” Comput.-Aided Civ. Infrastruct. Eng. 35 (9): 979–996. https://doi.org/10.1111/mice.12541.
Fathi, H., and I. Brilakis. 2013. “A videogrammetric as-built data collection method for digital fabrication of sheet metal roof panels.” Adv. Eng. Inf. 27 (4): 466–476. https://doi.org/10.1016/j.aei.2013.04.006.
Fischler, M. A., and R. C. Bolles. 1981. “Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography.” Commun. ACM 24 (6): 381–395. https://doi.org/10.1145/358669.358692.
Fuentes-Pacheco, J., J. Ruiz-Ascencio, and J. M. Rendón-Mancha. 2015. “Visual simultaneous localization and mapping: A survey.” Artif. Intell. Rev. 43 (1): 55–81. https://doi.org/10.1007/s10462-012-9365-8.
Gao, X. S., X. R. Hou, J. Tang, and H. F. Cheng. 2003. “Complete solution classification for the perspective-three-point problem.” IEEE Trans. Pattern Anal. Mach. Intell. 25 (8): 930–943. https://doi.org/10.1109/TPAMI.2003.1217599.
Gil, D., G. Lee, and K. Jeon. 2018. “Classification of images from construction sites using a deep-learning algorithm.” In Proc, 35th Int. Symp. on Automation and Robotics in Construction (ISARC), 176–181. Edinburgh, Scotland: International Association for Automation and Robotics in Construction.
Golparvar-Fard, M., J. Bohn, J. Teizer, S. Savarese, and F. Peña-Mora. 2011. “Evaluation of image-based modeling and laser scanning accuracy for emerging automated performance monitoring techniques.” Autom. Constr. 20 (8): 1143–1155. https://doi.org/10.1016/j.autcon.2011.04.016.
Golparvar-Fard, M., F. Peña-Mora, C. A. Arboleda, and S. Lee. 2009. “Visualization of construction progress monitoring with 4D simulation model overlaid on time-lapsed photographs.” J. Comput. Civ. Eng. 23 (6): 391–404. https://doi.org/10.1061/(ASCE)0887-3801(2009)23:6(391).
Golparvar-Fard, M., F. Peña-Mora, and S. Savarese. 2015. “Automated progress monitoring using unordered daily construction photographs and IFC-based building information models.” J. Comput. Civ. Eng. 29 (1): 1–19. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000205.
Ham, Y., K. K. Han, J. J. Lin, and M. Golparvar-Fard. 2016. “Visual monitoring of civil infrastructure systems via camera-equipped unmanned aerial vehicles (UAVS): A review of related works.” Visualization Eng. 4 (1): 1–8. https://doi.org/10.1186/s40327-015-0029-z.
Han, K., J. Degol, and M. Golparvar-Fard. 2018. “Geometry-and appearance-based reasoning of construction progress monitoring.” J. Constr. Eng. Manage. 144 (2): 1–12. https://doi.org/10.1061/(ASCE)CO.1943-7862.0001428.
Han, K. K., and M. Golparvar-Fard. 2015. “Appearance-based material classification for monitoring of operation-level construction progress using 4D BIM and site photologs.” Autom. Constr. 53 (May): 44–57. https://doi.org/10.1016/j.autcon.2015.02.007.
Hochreiter, S., and J. Schmidhuber. 1997. “Long short-term memory.” Neural Comput. 9 (8): 1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735.
Huang, G., Z. Liu, L. Van Der Maaten, and K. Q. Weinberger. 2017. “Densely connected convolutional networks.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2261–2269. New York: IEEE.
Irschara, A., C. Zach, J.-M. Frahm, and H. Bischof. 2009. “From SFM point clouds to fast location recognition.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2599–2606. New York: IEEE.
Isola, P., J. Y. Zhu, T. Zhou, and A. A. Efros. 2017. “Image-to-image translation with conditional adversarial networks.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition, 1125–1134. New York: IEEE.
Jeelani, I., K. Asadi, H. Ramshankar, K. Han, and A. Albert. 2021. “Real-time vision-based worker localization & hazard detection for construction.” Autom. Constr. 121 (Jan): 103448. https://doi.org/10.1016/j.autcon.2020.103448.
Johansen, K. W., R. P. de Figueiredo, O. Golovina, and J. Teizer. 2021. “Autonomous safety barrier inspection in construction: An approach using unmanned aerial vehicles and safe BIM.” In Proc., 38th Int. Symp. on Automation and Robotics in Construction (ISARC), 629–636. Edinburgh, Scotland: International Association for Automation and Robotics in Construction.
Karthikeyan, T., P. Manikandaprabhu, and S. Nithya. 2014. “A survey on text and content based image retrieval system for image mining.” Int. J. Eng. 3 (3): 509–512.
Kendall, A., and R. Cipolla. 2017. “Geometric loss functions for camera pose regression with deep learning.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition, 5974–5983. New York: IEEE.
Kendall, A., M. Grimes, and R. Cipolla. 2015. “PoseNet: A convolutional network for real-time 6-DOF camera relocalization.” In Proc., IEEE Int. Conf. on Computer Vision, 2938–2946. New York: IEEE.
Kolar, Z., H. Chen, and X. Luo. 2018. “Transfer learning and deep convolutional neural networks for safety guardrail detection in 2D images.” Autom. Constr. 89 (May): 58–70. https://doi.org/10.1016/j.autcon.2018.01.003.
Kropp, C., C. Koch, and M. König. 2018. “Interior construction state recognition with 4D BIM registered image sequences.” Autom. Constr. 86 (Feb): 11–32. https://doi.org/10.1016/j.autcon.2017.10.027.
Lepetit, V., F. Moreno-Noguer, and P. Fua. 2009. “EPnP: An accurate O() solution to the PnP problem.” Int. J. Comput. Vis. 81 (2): 155. https://doi.org/10.1007/s11263-008-0152-6.
Lin, J. J., and M. Golparvar-Fard. 2018. “Visual data and predictive analytics for proactive project controls on construction sites.” In Proc., Workshop of the European Group for Intelligent Computing in Engineering, 412–430. Berlin: Springer.
Lin, J. J., K. K. Han, and M. Golparvar-Fard. 2015. “A framework for model-driven acquisition and analytics of visual data using UAVS for automated construction progress monitoring.” In Computing in civil engineering, 156–164. Reston, VA: ASCE.
Lin, J. J., A. Ibrahim, S. Sarwade, and M. Golparvar-Fard. 2021. “Bridge inspection with aerial robots: Automating the entire pipeline of visual data capture, 3D mapping, defect detection, analysis, and reporting.” J. Comput. Civ. Eng. 35 (2): 04020064. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000954.
Maalek, R., D. D. Lichti, and J. Y. Ruwanpura. 2019. “Automatic recognition of common structural elements from point clouds for automated progress monitoring and dimensional quality control in reinforced concrete construction.” Remote Sens. 11 (9): 1102. https://doi.org/10.3390/rs11091102.
Meng, D., W. Li, and W. Bangfeng. 2011. “Vision-based estimation of relative pose in autonomous aerial refueling.” Chin. J. Aeronaut. 24 (6): 807–815. https://doi.org/10.1016/S1000-9361(11)60095-2.
Mueller, M. S., A. Metzger, and B. Jutzi. 2018. “CNN-based initial localization improved by data augmentation.” ISPRS Ann. Photogramm. Remote Sens. Spatial Inf. Sci. 4 (Sep): 117–124. https://doi.org/10.5194/isprs-annals-IV-1-117-2018.
Nakano, G. 2021. “Efficient DLT-based method for solving PNP, PNPF, and PNPFR problems.” IEICE Trans. Inf. Syst. 104 (9): 1467–1477. https://doi.org/10.1587/transinf.2020EDP7208.
Naseer, T., and W. Burgard. 2017. “SVS-Pose: Deep regression for monocular camera-based 6-DoF global localization in outdoor environments.” In Proc., IEEE Int. Conf. on Intelligent Robots and Systems, 1525–1530. New York: IEEE.
Özyeşil, O., V. Voroninski, R. Basri, and A. Singer. 2017. “A survey of structure from motion.” Acta Numer. 26 (May): 305–364. https://doi.org/10.1017/S096249291700006X.
Pal, A., and S. H. Hsieh. 2021. “Deep-learning-based visual data analytics for smart construction management.” Autom. Constr. 131 (Nov): 103892. https://doi.org/10.1016/j.autcon.2021.103892.
Pal, A., J. J. Lin, and S. H. Hsieh. 2021a. “A framework for automated daily construction progress monitoring leveraging unordered site photographs.” In Proc., Int. Conf. on Computing in Civil Engineering 2021 (i3CE2021), 538–545. Reston, VA: ASCE.
Pal, A., J. J. Lin, and S. H. Hsieh. 2021b. “A framework for vision-based progress monitoring through localization and analysis of unorganized onsite photographs.” In Proc., 26th Conf. on Computer Applications in Civil and Hydraulic Engineering, 1–9. Taipei City, Taiwan: Chinese Institute of Civil and Hydraulic Engineering.
Pal, A., J. J. Lin, and S. H. Hsieh. 2022. “Automated construction progress monitoring of partially completed building elements leveraging geometry modeling and appearance detection with deep learning.” In Proc., Construction Research Congress, 708–717. Reston, VA: Construction Research Council and the Construction Institute of ASCE.
Sattler, T., Q. Zhou, M. Pollefeys, and L. Leal-Taixe. 2019. “Understanding the limitations of CNN-based absolute camera pose regression.” In Proc., IEEE/CVF Conf. on Computer Vision and Pattern Recognition, 3302–3312. New York: IEEE.
Schönberger, J. L., and J. Frahm. 2016. “Structure-from-motion revisited.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition, 4104–4113. New York: IEEE.
Szegedy, C., W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. 2015. “Going deeper with convolutions.” In Proc., IEEE Computer Society Conf. on Computer Vision and Pattern Recognition, 1–9. New York: IEEE.
Szegedy, C., V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna. 2016. “Rethinking the inception architecture for computer vision.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition, 2818–2826. New York: IEEE.
Trzeciak, M., K. Pluta, Y. Fathy, L. Alcalde, S. Chee, A. Bromley, I. Brilakis, and P. Alliez. 2023. “Conslam: Construction data set for slam.” J. Comput. Civ. Eng. 37 (3): 04023009. https://doi.org/10.1061/JCCEE5.CPENG-5212.
Vassilieva, N. 2009. “Content-based image retrieval methods.” Program. Comput. Software 35 (3): 158–180. https://doi.org/10.1134/S0361768809030049.
Walch, F., C. Hazirbas, L. Leal-Taixe, T. Sattler, S. Hilsenbeck, and D. Cremers. 2017. “LSTM PoseNet: Image-based localization using LSTMs for structured feature correlation.” In Proc., IEEE Int. Conf. on Computer Vision, 627–637. New York: IEEE.
Wan, Y., W. Gao, S. Han, and Y. Wu. 2020. “Boosting image-based localization via randomly geometric data augmentation.” In Proc., IEEE Int. Conf. on Image Processing (ICIP), 688–692. New York: IEEE.
Wang, Z., Q. Zhang, B. Yang, T. Wu, K. Lei, B. Zhang, and T. Fang. 2021. “Vision-based framework for automatic progress monitoring of precast walls by using surveillance videos during the construction phase.” J. Comput. Civ. Eng. 35 (1): 04020056. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000933.
Wei, Y., and B. Akinci. 2018. “End-to-end image-based indoor localization for facility operation and management.” In Proc., 35th Int. Symp. on Automation and Robotics in Construction (ISARC), 1125–1133. Edinburgh, Scotland: International Association for Automation and Robotics in Construction.
Wei, Y., and B. Akinci. 2019. “A vision and learning-based indoor localization and semantic mapping framework for facility operations and management.” Autom. Constr. 107 (Nov): 102915. https://doi.org/10.1016/j.autcon.2019.102915.
Wei, Y., and B. Akinci. 2022. “Panorama-to-model registration through integration of image retrieval and semantic reprojection.” Autom. Constr. 140 (Aug): 104356. https://doi.org/10.1016/j.autcon.2022.104356.
Wu, Y., and Z. Hu. 2006. “PnP problem revisited.” J. Math. Imaging Vision 24 (1): 131–141. https://doi.org/10.1007/s10851-005-3617-z.
Xu, L., C. Feng, V. R. Kamat, and C. C. Menassa. 2019. “An occupancy grid mapping enhanced visual slam for real-time locating applications in indoor GPS-denied environments.” Autom. Constr. 104 (Aug): 230–245. https://doi.org/10.1016/j.autcon.2019.04.011.
Xu, L., C. Feng, V. R. Kamat, and C. C. Menassa. 2020. “A scene-adaptive descriptor for visual slam-based locating applications in built environments.” Autom. Constr. 112 (Apr): 103067. https://doi.org/10.1016/j.autcon.2019.103067.
Information & Authors
Information
Published In
Journal of Computing in Civil Engineering
Volume 37 • Issue 6 • November 2023
Copyright
© 2023 American Society of Civil Engineers.
History
Received: Jan 29, 2023
Accepted: Jun 6, 2023
Published online: Aug 7, 2023
Published in print: Nov 1, 2023
Discussion open until: Jan 7, 2024
ASCE Technical Topics:
- Architectural engineering
- Building information modeling
- Building management
- Computer models
- Computer networks
- Computing in civil engineering
- Construction engineering
- Construction management
- Construction sites
- Engineering fundamentals
- Geotechnical engineering
- Geotechnical investigation
- Models (by type)
- Project management
- Site investigation
- Three-dimensional models
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
Cited by
- Seau Chen Houng, Aritra Pal, Jacob J. Lin, 4D BIM and Reality Model–Driven Camera Placement Optimization for Construction Monitoring, Journal of Construction Engineering and Management, 10.1061/JCEMD4.COENG-14600, 150, 6, (2024).
- Houhao Liang, Justin K. W. Yeoh, David K. H. Chua, Robust Alignment of UGV Perspectives with BIM for Inspection in Indoor Environments, Journal of Computing in Civil Engineering, 10.1061/JCCEE5.CPENG-5761, 38, 4, (2024).