Pose Graph Relocalization with Deep Object Detection and BIM-Supported Object Landmark Dictionary
Publication: Journal of Computing in Civil Engineering
Volume 37, Issue 5
Abstract
Indoor localization is a prerequisite for autonomous robot applications in the construction industry. However, traditional localization techniques rely on low-level features and do not exploit construction-related semantics. They also are sensitive to environmental factors such as illumination and reflection rate, and therefore suffer from unexpected drifts and failures. This study proposes a pose graph relocalization framework that utilizes object-level landmarks to enhance a traditional visual localization system. The proposed framework builds an object landmark dictionary from Building Information Model (BIM) as prior knowledge. Then a multimodal deep neural network (DNN) is proposed to realize 3D object detection in real time, followed by instance-level object association with false-positive rejection, and relative pose estimation with outlier removal. Finally, a keyframe-based graph optimization is performed to rectify the drifts of traditional visual localization. The proposed framework was validated using a mobile platform with red-green-blue-depth (RGB-D) and inertial sensors, and the test scene was an indoor office environment with furnishing elements. The object detection model achieved 62.9% mean average precision (mAP). The relocalization technique reduced translational drifts by 64.67% and rotational drifts by 41.59% compared with traditional visual–inertial odometry.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
Some or all data, models, or code generated or used during the study are proprietary or confidential in nature and may only be provided with restrictions (e.g., BIM of the campus).
Acknowledgments
This work was supported by the Key-Area Research and Development Program of Guangdong Province (2020B090928001), the HKUST-BDR Joint Research Institute Fund (OKT23EG01), and the Foshan HKUST Project (FSUST21-HKUST05C).
References
Agarwal, S., and K. Mierle. 2019. “Ceres solver.” Accessed May 14, 2022. http://ceres-solver.org/.
Alby, C. 2020. “Revit furniture families.” Accessed August 1, 2022. https://measuredsurvey365.co.uk/revit-furniture-families-download/#stol.
Asadi, K., H. Ramshankar, M. Noghabaei, and K. Han. 2019. “Real-time image localization and registration with BIM using perspective alignment for indoor monitoring of construction.” J. Comput. Civ. Eng. 33 (5): 04019031. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000847.
Blum, H., J. Stiefel, C. Cadena, R. Siegwart, and A. Gawel. 2021. “Precise robot localization in architectural 3D plans.” In Proc., 38th Int. Symp. on Automation and Robotics in Construction (ISARC), 755–762. Edinburgh, UK: IAARC.
Bowman, S. L., N. Atanasov, K. Daniilidis, and G. J. Pappas. 2017. “Probabilistic data association for semantic SLAM.” In Proc., IEEE Int. Conf. Robot Autom, 1722–1729. New York: IEEE.
Cai, J., A. Du, X. Liang, and S. Li. 2023. “Prediction-based path planning for safe and efficient human–robot collaboration in construction via deep reinforcement learning.” J. Comput. Civ. Eng. 37 (1): 04022046. https://doi.org/10.1061/(ASCE)CP.1943-5487.0001056.
Chen, Z., K. Chen, C. Song, X. Zhang, J. C. P. Cheng, and D. Li. 2022. “Global path planning based on BIM and physics engine for UGVs in indoor environments.” Autom. Constr. 139 (Jul): 104263. https://doi.org/10.1016/j.autcon.2022.104263.
Deng, Y., J. C. P. Cheng, and C. Anumba. 2016. “Mapping between BIM and 3D GIS in different levels of detail using schema mediation and instance comparison.” Autom. Constr. 67 (Jul): 1–21. https://doi.org/10.1016/j.autcon.2016.03.006.
Dreher, M., H. Blum, R. Siegwart, and G. Abel. 2021. “Global localization in meshes.” In Proc., 38th Int. Symp. on Automation and Robotics in Construction (ISARC), 747–754. Edinburgh, UK: IAARC.
Gálvez-López, D., and J. D. Tardós. 2012. “Bags of binary words for fast place recognition in image sequences.” IEEE Trans. Rob. 28 (5): 1188–1197. https://doi.org/10.1109/TRO.2012.2197158.
Garrido-Jurado, S., R. Muñoz-Salinas, F. J. Madrid-Cuevas, and M. J. Marín-Jiménez. 2014. “Automatic generation and detection of highly reliable fiducial markers under occlusion.” Pattern Recognit. 47 (6): 2280–2292. https://doi.org/10.1016/j.patcog.2014.01.005.
Geiger, A., P. Lenz, C. Stiller, and R. Urtasun. 2013. “Vision meets robotics: The KITTI dataset.” Int. J. Rob. Res. 32 (11): 1231–1237. https://doi.org/10.1177/0278364913491297.
Girshick, R. 2015. “Fast R-CNN.” In Proc., 2015 IEEE Int. Conf. on Computer Vision (ICCV), 1440–1448. New York: IEEE.
Girshick, R., J. Donahue, T. Darrell, and J. Malik. 2014. “Rich feature hierarchies for accurate object detection and semantic segmentation.” In Proc., 2014 IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 580–587. New York: IEEE.
Grandio, J., B. Riveiro, M. Soilán, and P. Arias. 2022. “Point cloud semantic segmentation of complex railway environments using deep learning.” Autom. Constr. 141 (Sep): 104425. https://doi.org/10.1016/j.autcon.2022.104425.
Grinvald, M., F. Furrer, T. Novkovic, J. J. Chung, C. Cadena, R. Siegwart, and J. Nieto. 2019. “Volumetric instance-aware semantic mapping and 3D object discovery.” IEEE Robot. Autom. Lett. 4 (3): 3037–3044. https://doi.org/10.1109/LRA.2019.2923960.
Grupp, M. 2017. “evo: Python package for the evaluation of odometry and SLAM.” Accessed August 26, 2022. https://github.com/MichaelGrupp/evo.
Ha, I., H. Kim, S. Park, and H. Kim. 2018. “Image retrieval using BIM and features from pretrained VGG network for indoor localization.” Build. Environ. 140 (May): 23–31. https://doi.org/10.1016/j.buildenv.2018.05.026.
Hamledari, H., S. Sajedi, B. McCabe, and M. Fischer. 2021. “Automation of inspection mission planning using 4D BIMs and in support of unmanned aerial vehicle–based data collection.” J. Constr. Eng. Manage. 147 (3): 04020179. https://doi.org/10.1061/(ASCE)CO.1943-7862.0001995.
He, K., G. Gkioxari, P. Dollar, and R. Girshick. 2020. “Mask R-CNN.” IEEE Trans. Pattern Anal. Mach. Intell. 42 (2): 386–397. https://doi.org/10.1109/TPAMI.2018.2844175.
Ibrahim, A., M. Golparvar-Fard, and K. El-Rayes. 2022. “Multiobjective optimization of reality capture plans for computer vision–driven construction monitoring with camera-equipped UAVs.” J. Comput. Civ. Eng. 36 (5): 04022018. https://doi.org/10.1061/(ASCE)CP.1943-5487.0001032.
Ibrahim, A., A. Sabet, and M. Golparvar-Fard. 2019. “BIM-driven mission planning and navigation for automatic indoor construction progress detection using robotic ground platform.” In Proc., 2019 European Conf. on Computing in Construction, 182–189. Rhodes, Greece: European Council on Computing in Construction.
Kim, S., M. Peavy, P.-C. Huang, and K. Kim. 2021. “Development of BIM-integrated construction robot task planning and simulation system.” Autom. Constr. 127 (Apr): 103720. https://doi.org/10.1016/j.autcon.2021.103720.
Krizhevsky, A., I. Sutskever, and G. E. Hinton. 2012. “ImageNet classification with deep convolutional neural networks.” Commun. ACM 60 (6): 84–90. https://doi.org/10.1145/3065386.
Li, C. T., J. C. P. Cheng, and K. Chen. 2020. “Top 10 technologies for indoor positioning on construction sites.” Autom. Constr. 118 (Jun): 103309. https://doi.org/10.1016/j.autcon.2020.103309.
Li, G., L. Yu, and S. Fei. 2021. “A deep-learning real-time visual SLAM system based on multi-task feature extraction network and self-supervised feature points.” Measurement 168 (Jan): 108403. https://doi.org/10.1016/j.measurement.2020.108403.
Lin, T. Y., M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. 2014. “Microsoft COCO: Common objects in context.” In Lecture notes in computer science (including subseries lecture notes in artificial intelligence and lecture notes in bioinformatics), 740–755. Cham, Switzerland: Springer.
Liu, D., X. Xia, J. Chen, and S. Li. 2021. “Integrating building information model and augmented reality for drone-based building inspection.” J. Comput. Civ. Eng. 35 (2): 04020073. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000958.
Lowe, D. G. 2004. “Distinctive image features from scale-invariant keypoints.” Int. J. Comput. Vision 60 (2): 91–110. https://doi.org/10.1023/B:VISI.0000029664.99615.94.
Lu, T., S. Tervola, X. Lü, C. J. Kibert, Q. Zhang, T. Li, and Z. Yao. 2021. “A novel methodology for the path alignment of visual SLAM in indoor construction inspection.” Autom. Constr. 127 (Jul): 103723. https://doi.org/10.1016/j.autcon.2021.103723.
Luperto, M., M. Antonazzi, F. Amigoni, and N. A. Borghese. 2020. “Robot exploration of indoor environments using incomplete and inaccurate prior knowledge.” Rob. Auton. Syst. 133 (Nov): 103622. https://doi.org/10.1016/j.robot.2020.103622.
McCormac, J., A. Handa, A. Davison, and S. Leutenegger. 2017. “SemanticFusion: Dense 3D semantic mapping with convolutional neural networks.” In Proc., IEEE Int. Conf. Robot Autom, 4628–4635. New York: IEEE. https://doi.org/10.1109/ICRA.2017.7989538.
Moura, M. S., C. Rizzo, and D. Serrano. 2021. “BIM-based localization and mapping for mobile robots in construction.” In Proc., 2021 IEEE Int. Conf. on Autonomous Robot Systems and Competitions (ICARSC), 12–18. New York: IEEE.
Mur-Artal, R., J. M. M. Montiel, and J. D. Tardos. 2015. “ORB-SLAM: A versatile and accurate monocular SLAM system.” IEEE Trans. Rob. 31 (5): 1147–1163. https://doi.org/10.1109/TRO.2015.2463671.
Ojha, A., Y. Liu, S. Shayesteh, H. Jebelli, and W. E. Sitzabee. 2023. “Affordable multiagent robotic system for same-level fall hazard detection in indoor construction environments.” J. Comput. Civ. Eng. 37 (1): 04022042. https://doi.org/10.1061/(ASCE)CP.1943-5487.0001052.
Qi, C. R., X. Chen, O. Litany, and L. J. Guibas. 2020. “ImVoteNet: Boosting 3D object detection in point clouds with image votes.” In Proc., IEEE Computer Society Conf. on Computer Vision and Pattern Recognition (CVPR), 4403–4412. New York: IEEE.
Qi, C. R., O. Litany, K. He, and L. Guibas. 2019. “Deep hough voting for 3D object detection in point clouds.” In Proc., IEEE/CVF Int. Conf. on Computer Vision, 9276–9285. New York: IEEE.
Qi, C. R., W. Liu, C. Wu, H. Su, and L. J. Guibas. 2018. “Frustum PointNets for 3D object detection from RGB-D data.” In Proc., IEEE Computer Society Conf. on Computer Vision and Pattern Recognition (CVPR), 918–927. New York: IEEE.
Qi, C. R., H. Su, K. Mo, and L. J. Guibas. 2016. “PointNet: Deep learning on point sets for 3D classification and segmentation.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 652–660. New York: IEEE.
Qi, C. R., L. Yi, H. Su, and L. J. Guibas. 2017. “PointNet++: Deep hierarchical feature learning on point sets in a metric space.” In Proc., Conf. on Neural Information Processing Systems (NIPS). New York: Curran Associates.
Qin, T., P. Li, and S. Shen. 2018. “VINS-mono: A robust and versatile monocular visual-inertial state estimator.” IEEE Trans. Rob. 34 (4): 1004–1020. https://doi.org/10.1109/TRO.2018.2853729.
Redmon, J., S. Divvala, R. Girshick, and A. Farhadi. 2016. “You only look once: Unified, real-time object detection.” In Proc., IEEE Computer Society Conf. on Computer Vision and Pattern Recognition (CVPR), 779–788. New York: IEEE.
Ren, S., K. He, R. Girshick, and J. Sun. 2015. “Faster R-CNN: Towards real-time object detection with region proposal networks.” In Advances in neural information processing systems. New York: Curran Associates.
Rosinol, A., M. Abate, Y. Chang, and L. Carlone. 2020. “Kimera: An open-source library for real-time metric-semantic localization and mapping.” In Proc., IEEE Int. Conf. Robot Autom, 1689–1696. New York: IEEE. https://doi.org/10.1109/ICRA40945.2020.9196885.
Rukhovich, D., A. Vorontsova, and A. Konushin. 2022. “ImVoxelNet: Image to voxels projection for monocular and multi-view general-purpose 3D object detection.” In Proc., 2022 IEEE/CVF Winter Conf. on Applications of Computer Vision (WACV), 1265–1274. New York: IEEE Computer Society.
Salas-Moreno, R. F., R. A. Newcombe, H. Strasdat, P. H. J. Kelly, and A. J. Davison. 2013. “SLAM++: Simultaneous localisation and mapping at the level of objects.” In Proc., IEEE Computer Society Conf. on Computer Vision and Pattern Recognition (CVPR), 1352–1359. New York: IEEE.
Shirowzhan, S., S. M. E. Sepasgozar, D. J. Edwards, H. Li, and C. Wang. 2020. “BIM compatibility and its differentiation with interoperability challenges as an innovation factor.” Autom. Constr. 112 (Apr): 103086. https://doi.org/10.1016/j.autcon.2020.103086.
Simonyan, K., and A. Zisserman. 2014. “Very deep convolutional networks for large-scale image recognition.” Preprint, submitted May 4, 2023. https://arxiv.org/abs/1610.02583.
Song, C. 2022. “Pose graph re-localization.” Accessed September 2, 2022. https://www.youtube.com/watch?v=wsF0kLocgaM.
Song, C., Z. Chen, K. Wang, H. Luo, and J. C. P. Cheng. 2022. “BIM-supported scan and flight planning for fully autonomous LiDAR-carrying UAVs.” Autom. Constr. 142 (Oct): 104533. https://doi.org/10.1016/j.autcon.2022.104533.
Song, C., K. Wang, and J. C. P. Cheng. 2020. “BIM-aided scanning path planning for autonomous surveillance UAVs with LiDAR.” In Proc., 37th Int. Symp. on Automation and Robotics in Construction, ISARC 2020, 1195–1202. Edinburgh, UK: IAARC.
Song, S., S. P. Lichtenberg, and J. Xiao. 2015. “SUN RGB-D: A RGB-D scene understanding benchmark suite.” In Proc., 2015 IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 567–576. New York: IEEE.
Strasdat, H., J. M. M. Montiel, and A. J. Davison. 2010. “Real-time monocular SLAM: Why filter?” In Proc., IEEE Int. Conf. Robot Autom, 2657–2664. New York: IEEE.
Su, H., S. Maji, E. Kalogerakis, and E. Learned-Miller. 2015. “Multi-view convolutional neural networks for 3D shape recognition.” In Proc., 2015 IEEE Int. Conf. on Computer Vision (ICCV), 945–953. New York: IEEE.
Taketomi, T., H. Uchiyama, and S. Ikeda. 2017. “Visual SLAM algorithms: A survey from 2010 to 2016.” IPSJ Trans. Comput. Vision Appl. 9 (1): 1–11. https://doi.org/10.1186/s41074-017-0027-2.
Tan, L., T. Huangfu, and L. Wu. 2021. “Comparison of YOLO v3, Faster R-CNN, and SSD for real-time pill identification.” Accessed August 7, 2022. https://www.researchsquare.com/article/rs-668895/v1.
Taneja, S., B. Akinci, J. H. Garrett Jr., and L. Soibelman. 2016. “Algorithms for automated generation of navigation models from building information models to support indoor map-matching.” Autom. Constr. 61 (Jan): 24–41. https://doi.org/10.1016/j.autcon.2015.09.010.
Tseng, P. Y., J. J. Lin, Y.-C. Chan, and A. Y. Chen. 2022. “Real-time indoor localization with visual SLAM for in-building emergency response.” Autom. Constr. 140 (Aug): 104319. https://doi.org/10.1016/j.autcon.2022.104319.
Wang, J., M. Runz, and L. Agapito. 2021. “DSP-SLAM: Object oriented SLAM with deep shape priors.” In Proc., 2021 Int. Conf. on 3D Vision (3DV), 1362–1371. New York: IEEE.
Wang, K., F. Gao, and S. Shen. 2019. “Real-time scalable dense surfel mapping.” In Proc., 2019 Int. Conf. on Robotics and Automation (ICRA), 6919–6925. New York: IEEE.
Xu, F., P. Xia, H. You, and J. Du. 2022. “Robotic cross-platform sensor fusion and augmented visualization for large indoor space reality capture.” J. Comput. Civ. Eng. 36 (6): 04022036. https://doi.org/10.1061/(ASCE)CP.1943-5487.0001047.
Xu, L., C. Feng, V. R. Kamat, and C. C. Menassa. 2019. “An occupancy grid mapping enhanced visual SLAM for real-time locating applications in indoor GPS-denied environments.” Autom. Constr. 104 (Aug): 230–245. https://doi.org/10.1016/j.autcon.2019.04.011.
Xu, L., C. Feng, V. R. Kamat, and C. C. Menassa. 2020. “A scene-adaptive descriptor for visual SLAM-based locating applications in built environments.” Autom. Constr. 112 (Apr): 103067. https://doi.org/10.1016/j.autcon.2019.103067.
Xu, W., and F. Zhang. 2021. “FAST-LIO: A fast, robust LiDAR-inertial odometry package by tightly-coupled iterated Kalman filter.” IEEE Rob. Autom. Lett. 6 (2): 3317–3324. https://doi.org/10.1109/LRA.2021.3064227.
Xu, Y., V. John, S. Mita, H. Tehrani, K. Ishimaru, and S. Nishino. 2017. “3D point cloud map based vehicle localization using stereo camera.” In Proc., IEEE Intelligent Vehicles Symp., Proc., 487–492. New York: IEEE. https://doi.org/10.1109/IVS.2017.7995765.
Yang, J., H. Li, D. Campbell, and Y. Jia. 2016. “Go-ICP: A globally optimal solution to 3D ICP point-set registration.” IEEE Trans. Pattern Anal. Mach. Intell. 38 (11): 2241–2254. https://doi.org/10.1109/TPAMI.2015.2513405.
Yin, C., B. Wang, V. J. L. Gan, M. Wang, and J. C. P. Cheng. 2021. “Semantic localization on BIM-generated maps using a 3D LiDAR sensor.” Autom. Constr. 130 (Oct): 103874. https://doi.org/10.1016/j.autcon.2021.103874.
Yin, H., Z. Lin, and J. K. W. Yeoh. 2022. “Semantic localization in BIM using a 3D LiDAR sensor.” Autom. Constr. 146 (Feb): 104641. https://doi.org/10.1016/j.autcon.2022.104641.
Zeng, A., S. Song, M. Niessner, M. Fisher, J. Xiao, and T. Funkhouser. 2017. “3DMatch: Learning local geometric descriptors from RGB-D reconstructions.” In Proc., 2017 IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 199–208. New York: IEEE.
Zhang, J., and S. Singh. 2014. “LOAM: Lidar odometry and mapping in real-time.” In Proc., Robotics: Science and Systems (RSS’14). Berkeley, CA: Robotics: Science and Systems Foundation.
Zhao, X., and C. C. Cheah. 2023. “BIM-based indoor mobile robot initialization for construction automation using object detection.” Autom. Constr. 146 (Feb): 104647. https://doi.org/10.1016/j.autcon.2022.104647.
Zhou, X., Q. Xie, M. Guo, J. Zhao, and J. Wang. 2020. “Accurate and efficient indoor pathfinding based on building information modeling data.” IEEE Trans. Ind. Inf. 16 (12): 7459–7468. https://doi.org/10.1109/TII.2020.2974252.
Zhou, Y., and O. Tuzel. 2018. “VoxelNet: End-to-end learning for point cloud based 3D object detection.” In Proc., 2018 IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), 4490–4499. New York: IEEE Computer Society.
Information & Authors
Information
Published In
Journal of Computing in Civil Engineering
Volume 37 • Issue 5 • September 2023
Copyright
© 2023 American Society of Civil Engineers.
History
Received: Dec 9, 2022
Accepted: Mar 5, 2023
Published online: May 19, 2023
Published in print: Sep 1, 2023
Discussion open until: Oct 19, 2023
ASCE Technical Topics:
- Analysis (by type)
- Architectural engineering
- Automation and robotics
- Building information modeling
- Building management
- Building systems
- Construction engineering
- Construction industry
- Construction management
- Construction methods
- Engineering fundamentals
- Failure analysis
- History and Heritage
- Light (artificial)
- Practice and Profession
- Systems engineering
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.