Improved Boundary Identification of Stacked Objects with Sparse LiDAR Augmentation Scanning
Publication: Journal of Construction Engineering and Management
Volume 149, Issue 11
Abstract
Vision-based sensors have been widely used in reality capture and the corresponding scene understanding tasks such as object detection. Given the increasing complexity of built environments, geometric features from the raw scanning data can become too vague for effective object detection. One example challenge is stacked object recognition, i.e., the segmentation, detection, and recognition of objects being stacked together with similar geometric features or occlusions. Previous methods propose to use high-resolution sensors to capture more detailed geometry information to highlight the boundaries between adjacent objects, which increase the deployment cost and computing needs. This paper proposes a novel data augmentation and voting method for stacked object detection with only low-cost sparse sensors. Several locomotion strategies were used to focus on filling the gaps of the sparse light detection and ranging (LiDAR) sensor. A modified LiDAR odometry and mapping (LOAM) method was used to register and augment raw point cloud data from multiple scans in real time. Then a voxel-based density voting method was applied to centralize the points in enhanced scan for a more accurate clustering. Finally, the clustered points were grouped and applied to generate three-dimensional (3D) bounding boxes for object boundary identification. A pilot test was performed to show the improved results of the proposed methods. A series of benchmarking studies were also performed to identify the minimum acceptable density level for the proposed method.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
Some or all data, models, or code generated or used during the study are available from the corresponding author by request.
Acknowledgments
This material is supported by the National Institute of Standards and Technology (NIST) under Grant 70NANB21H045. Any opinions, findings, conclusions, or recommendations expressed in this article are those of the authors and do not reflect the views of the NIST.
References
Ariyachandra, M. R. M. F., and I. Brilakis. 2020. “Detection of railway masts in airborne LiDAR data.” J. Constr. Eng. Manage. 146 (9): 04020105. https://doi.org/10.1061/(ASCE)CO.1943-7862.0001894.
Bai, L., Y. Li, M. Cen, and F. Hu. 2021. “3D instance segmentation and object detection framework based on the fusion of LiDAR remote sensing and optical image sensing.” Remote Sens. 13 (16): 3288. https://doi.org/10.3390/rs13163288.
Bandyapadhyay, S., and K. Varadarajan. 2015. “On variants of k-means clustering.” Preprint, submitted December 9, 2015. http://arxiv.org/abs/1512.02985.
Bay, H., T. Tuytelaars, and L. V. Gool. 2006. “Surf: Speeded up robust features.” In Proc., 9th European Conf. on Computer Vision. New York: Springer.
Calonder, M., V. Lepetit, C. Strecha, and P. Fua. 2010. “Brief: Binary robust independent elementary features.” In Proc., 11th European Conf. on Computer Vision, 778–792. Berlin: Springer.
Charles, R. Q., H. Su, M. Kaichun, and L. J. Guibas. 2017. “PointNet: Deep learning on point sets for 3D classification and segmentations.” In Proc., 2017 IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 77–858. New York: IEEE. https://arxiv.org/abs/1912.00036.
Chen, J., Y. Fang, and Y. K. Cho. 2018. “Performance evaluation of 3D descriptors for object recognition in construction applications.” Autom. Constr. 86 (Feb): 44–52. https://doi.org/10.1016/j.autcon.2017.10.033.
Chen, J., Z. Kira, and Y. K. Cho. 2019. “Deep learning approach to point cloud scene understanding for automated scan to 3D reconstruction.” J. Comput. Civil Eng. 33 (4): 04019027. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000842.
Chen, L.-C., T.-H. Nguyen, and S.-T. Lin. 2015. “Independent 3D object segmentation for randomly stacked objects using optical object detection.” Meas. Sci. Technol. 26 (10): 105202. https://doi.org/10.1088/0957-0233/26/10/105202.
Cheng, L., and Z. Yang. 2020. “GRCNN: Graph Recognition Convolutional Neural Network for synthesizing programs from flow charts.” Preprint, submitted November 11, 2020. https://doi.org/10.48550/arXiv.1706.02413.
Dai, A., C. Diller, and M. Nießner. 2020. “SG-NN: Sparse generative neural networks for self-supervised scene completion of RGB-D scans.” In Proc., IEEE/CVF Conference on Computer Vision and Pattern Recognition, 849–858. New York: IEEE. https://arxiv.org/abs/1912.00036.
Dalal, N., and B. Triggs. 2005. “Histograms of oriented gradients for human detection.” In Proc., IEEE Computer Society Conf. on Computer Vision and Pattern Recognition (CVPR’05). New York: IEEE.
Das, S. D., V. Bain, and P. Rakshit. 2018. “Energy optimized robot arm path planning using differential evolution in dynamic environment.” In Proc., 2nd Int. Conf. on Intelligent Computing and Control Systems (ICICCS). New York: IEEE.
Davison, A. J., I. D. Reid, N. D. Molton, and O. Stasse. 2007. “MonoSLAM: Real-time single camera SLAM.” IEEE Trans. Pattern Anal. Mach. Intell. 29 (6): 1052–1067. https://doi.org/10.1109/TPAMI.2007.1049.
De Geyter, S., J. Vermandere, H. De Winter, M. Bassier, and M. Vergauwen. 2022. “Point cloud validation: On the impact of laser scanning technologies on the semantic segmentation for BIM modeling and evaluation.” Remote Sens. 14 (3): 582. https://doi.org/10.3390/rs14030582.
Dino, I. G., A. E. Sari, O. K. Iseri, S. Akin, E. Kalfaoglu, B. Erdogan, S. Kalkan, and A. A. Alatan. 2020. “Image-based construction of building energy models using computer vision.” Autom. Constr. 116 (Aug): 103231. https://doi.org/10.1016/j.autcon.2020.103231.
Dong, P., and Q. Chen. 2017. LiDAR remote sensing and applications. Boca Raton, FL: CRC Press.
Ekanayake, B., J. K.-W. Wong, A. A. F. Fini, and P. Smith. 2021. “Computer vision-based interior construction progress monitoring: A literature review and future research directions.” Autom. Constr. 127 (Jul): 103705. https://doi.org/10.1016/j.autcon.2021.103705.
Ester, M., H.-P. Kriegel, J. Sander, and X. Xu. 1996. “A density-based algorithm for discovering clusters in large spatial databases with noise.” In Proc., 2nd Int. Conf. on Knowledge Discovery and Data Mining. Washington, DC: Association for the Advancement of Artificial Intelligence.
Fang, W., L. Ding, P. E. Love, H. Luo, H. Li, F. Pena-Mora, B. Zhong, and C. Zhou. 2020. “Computer vision applications in construction safety assurance.” Autom. Constr. 110 (Feb): 103013. https://doi.org/10.1016/j.autcon.2019.103013.
Fathi, H., F. Dai, and M. Lourakis. 2015. “Automated as-built 3D reconstruction of civil infrastructure using computer vision: Achievements, opportunities, and challenges.” Adv. Eng. Inf. 29 (2): 149–161. https://doi.org/10.1016/j.aei.2015.01.012.
Fuentes-Pacheco, J., J. Ruiz-Ascencio, and J. M. Rendón-Mancha. 2015. “Visual simultaneous localization and mapping: A survey.” Artif. Intell. Rev. 43 (1): 55–81. https://doi.org/10.1007/s10462-012-9365-8.
Garg, S., and R. Jain. 2006. “Variations of k-mean algorithm: A study for high-dimensional large data sets.” Inf. Technol. J. 5 (6): 1132–1135 https://doi.org/10.3923/itj.2006.1132.1135.
Gargoum, S. A., K. El-Basyouny, and J. Sabbagh. 2018. “Assessing stopping and passing sight distance on highways using mobile LiDAR data.” J. Comput. Civ. Eng. 32 (4): 04018025. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000753.
Geneva, P., J. Maley, and G. Huang. 2019. “An efficient Schmidt-EKF for 3D visual-inertial SLAM.” In Proc., IEEE/CVF Conf. on Computer Vision and Pattern Recognition. New York: IEEE.
Kamari, M., and Y. Ham. 2021. “Vision-based volumetric measurements via deep learning-based point cloud segmentation for material management in jobsites.” Autom. Constr. 121: 103430. https://doi.org/10.1016/j.autcon.2020.103430.
Kim, J., J. Kim, and J. Cho. 2019. “An advanced object classification strategy using YOLO through camera and LiDAR sensor fusion.” In Proc., 13th Int. Conf. on Signal Processing and Communication Systems (ICSPCS). New York: IEEE.
Kim, P., J. Chen, and Y. K. Cho. 2018. “SLAM-driven robotic mapping and registration of 3D point clouds.” Autom. Constr. 89 (May): 38–48. https://doi.org/10.1016/j.autcon.2018.01.009.
Li, L., S. Khan, and N. Barnes. 2019. “Silhouette-assisted 3D object instance reconstruction from a cluttered scene.” In Proc., IEEE/CVF Int. Conf. on Computer Vision Workshops. New York: IEEE.
Lowe, D. G. 1999. “Object recognition from local scale-invariant features.” In Proc., 7th IEEE Int. Conf. on Computer Vision. New York: IEEE.
Mascaro, R., M. Wermelinger, M. Hutter, and M. Chli. 2021. “Towards automating construction tasks: Large-scale object mapping, segmentation, and manipulation.” J. Field Rob. 38 (5): 684–699. https://doi.org/10.1002/rob.22007.
Matsuno, D., R. Hachiuma, H. Saito, J. Sugano, and H. Adachi. 2020. “Pose estimation of stacked rectangular objects from depth images.” In Proc., 2020 IEEE 29th Int. Symposium on Industrial Electronics (ISIE), 1409–1414. New York: IEEE. https://doi.org/10.1109/ISIE45063.2020.9152510.
Paneru, S., and I. Jeelani. 2021. “Computer vision applications in construction: Current state, opportunities & challenges.” Autom. Constr. 132 (Dec): 103940. https://doi.org/10.1016/j.autcon.2021.103940.
Pratama, A. R. P., B. S. B. Dewantara, D. M. Sari, and D. Pramadihanto. 2022. “Density-based clustering for 3D stacked pipe object recognition using directly-given point cloud data on convolutional neural network.” EMITTER Int. J. Eng. Technol. 10 (1): 153–169. https://doi.org/10.24003/emitter.v10i1.704.
Previtali, M., M. Scaioni, L. Barazzetti, and R. Brumana. 2014. “A flexible methodology for outdoor/indoor building reconstruction from occluded point clouds.” ISPRS Ann. Photogramm. Remote Sens. Spatial Inf. Sci. II-3 (3): 119. https://doi.org/10.5194/isprsannals-II-3-119-2014.
Qi, C. R., L. Yi, H. Su, and L. J. Guibas. 2017. “Pointnet++: Deep hierarchical feature learning on point sets in a metric space.” Adv. Neural Inf. Process. Syst. 30. https://doi.org/10.48550/arXiv.1706.02413.
Roca, D., S. Lagüela, L. Díaz-Vilariño, J. Armesto, and P. Arias. 2013. “Low-cost aerial unit for outdoor inspection of building façades.” Autom. Constr. 36 (Dec): 128–135. https://doi.org/10.1016/j.autcon.2013.08.020.
Rosas-Cervantes, V., and S.-G. Lee. 2020. “3D localization of a mobile robot by using Monte Carlo algorithm and 2D features of 3D point cloud.” Int. J. Control Autom. Syst. 18 (11): 2955–2965. https://doi.org/10.1007/s12555-019-0313-0.
Rosten, E., and T. Drummond. 2006. “Machine learning for high-speed corner detection.” In Proc., 9th European Conf. on Computer Vision. Berlin: Springer.
Rublee, E., V. Rabaud, K. Konolige, and G. Bradski. 2011. “ORB: An efficient alternative to SIFT or SURF.” In Proc., Int. Conf. on Computer Vision. New York: IEEE.
Sajjan, S., M. Moore, M. Pan, G. Nagaraja, J. Lee, A. Zeng, and S. Song. 2020. “Clear grasp: 3d shape estimation of transparent objects for manipulation.” In Proc., 2020 IEEE International Conference on Robotics and Automation (ICRA), 3634–3642. New York: IEEE. https://doi.org/10.1109/ICRA40945.2020.9197518.
Shao, L., Z. Cai, L. Liu, and K. Lu. 2017. “Performance evaluation of deep feature learning for RGB-D image/video classification.” Inf. Sci. 385: 266–283. https://doi.org/10.1016/j.ins.2017.01.013.
Shi, S., C. Guo, L. Jiang, Z. Wang, J. Shi, X. Wang, and H. Li. 2020. “PV-RCNN: Point-voxel feature set abstraction for 3D object detection.” In Proc., IEEE/CVF Conf. on Computer Vision and Pattern Recognition, 10529–10538. New York: IEEE.
Shi, S., L. Jiang, J. Deng, Z. Wang, C. Guo, J. Shi, X. Wang, and H. Li. 2023. “PV-RCNN++: Point-voxel feature set abstraction with local vector representation for 3D object detection.” Int. J. Comput. Vision 131 (2): 531–551. https://doi.org/10.48550/arXiv.1706.02413.
Shi, W., M. B. Alawieh, X. Li, and H. Yu. 2017. “Algorithm and hardware implementation for visual perception system in autonomous vehicle: A survey.” Integration 59 (Sep): 148–156. https://doi.org/10.1016/j.vlsi.2017.07.007.
Son, H., H. Choi, H. Seong, and C. Kim. 2019. “Detection of construction workers under varying poses and changing background in image sequences via very deep residual networks.” Autom. Constr. 99 (Mar): 27–38. https://doi.org/10.1016/j.autcon.2018.11.033.
Song, S., S. P. Lichtenberg, and J. Xiao. 2015. “SUN RGB-D: A RGB-D scene understanding benchmark suite.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition. New York: IEEE.
Thammasorn, P., S. Boonchu, and A. Kawewong. 2013. “Real-time method for counting unseen stacked objects in mobile.” In Proc., 2013 IEEE Int. Conf. on Image Processing, 4103–4107. New York: IEEE. https://doi.org/10.1109/ICIP.2013.6738845.
Wang, C., Y. K. Cho, and M. Gai. 2013. “As-is 3D thermal modeling for existing building envelopes using a hybrid LiDAR system.” J. Comput. Civ. Eng. 27 (6): 645–656. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000273.
Wang, R., J. Peethambaran, and D. Chen. 2018. “LiDAR point clouds to 3-D urban models: A review.” IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 11 (2): 606–627. https://doi.org/10.1109/JSTARS.2017.2781132.
Xu, Y., S. Tuttas, L. Hoegner, and U. Stilla. 2018. “Voxel-based segmentation of 3D point clouds from construction sites using a probabilistic connectivity model.” Pattern Recognit. Lett. 102: 67–74. https://doi.org/10.1016/j.patrec.2017.12.016.
Yi, C., Y. Zhang, Q. Wu, Y. Xu, O. Remil, M. Wei, and J. Wang. 2017. “Urban building reconstruction from raw LiDAR point data.” Comput. Aided Des. 93 (Dec): 1–14. https://doi.org/10.1016/j.cad.2017.07.005.
Yin, C., B. Wang, V. J. Gan, M. Wang, and J. C. Cheng. 2021. “Automated semantic segmentation of industrial point clouds using ResPointNet++.” Autom. Constr. 130: 103874. https://doi.org/10.1016/j.autcon.2021.103874.
Zhang, J., and S. Singh. 2014. “LOAM: LiDAR odometry and mapping in real-time.” In Vol. 2 of Proc., Robotics: Science and Systems Conf., 1–9. New York: IEEE. https://doi.org/10.15607/rss.2014.x.007.
Zhang, T., R. Ramakrishnan, and M. Livny. 1996. “BIRCH: An efficient data clustering method for very large databases.” ACM Sigmod Rec. 25 (2): 103–114. https://doi.org/10.1145/235968.233324.
Zhao, Q., Y. Wu, X. Li, J. Xu, and Q. Meng. 2017. “A method of measuring stacked objects volume based on laser sensing.” Meas. Sci. and Technol. 28 (10): 105002. https://doi.org/10.1088/1361-6501/aa7e8b.
Zhao, Y., K. Zhang, H. Yu, Y. Zhang, D. Zheng, and J. Han. 2022. “Indoor simultaneous localization and mapping based on fringe projection profilometry.” Preprint, submitted May 17, 2023. http://arxiv.org/abs/2204.11020.
Zhou, Y., H. Guo, L. Ma, Z. Zhang, and M. Skitmore. 2021. “Image-based onsite object recognition for automatic crane lifting tasks.” Autom. Constr. 123 (Mar): 103527. https://doi.org/10.1016/j.autcon.2020.103527.
Zhou, Y., and O. Tuzel. 2018. “Voxelnet: End-to-end learning for point cloud based 3D object detection.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition. New York: IEEE.
Żywanowski, K., A. Banaszczyk, and M. R. Nowicki. 2020. “Comparison of camera-based and 3D LiDAR-based place recognition across weather conditions.” In Proc., 16th Int. Conf. on Control, Automation, Robotics and Vision (ICARCV). New York: IEEE.
Information & Authors
Information
Published In
Journal of Construction Engineering and Management
Volume 149 • Issue 11 • November 2023
Copyright
© 2023 American Society of Civil Engineers.
History
Received: Feb 5, 2023
Accepted: Jun 26, 2023
Published online: Aug 17, 2023
Published in print: Nov 1, 2023
Discussion open until: Jan 17, 2024
ASCE Technical Topics:
- Benefit cost ratios
- Buildings
- Business management
- Computer vision and image processing
- Computing in civil engineering
- Design (by type)
- Detection methods
- Domain boundary
- Engineering fundamentals
- Equipment and machinery
- Financial management
- Geometrics
- Highway and road design
- Mathematics
- Methodology (by type)
- Practice and Profession
- Probe instruments
- Structural engineering
- Structures (by type)
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.