Adaptive Scanning for Improved Stacked Object Detection with RGB and LiDAR
Publication: Construction Research Congress 2024
ABSTRACT
The increasing requirements of robots in construction have brought great challenges related to understanding the complex environment and completing the downstream tasks with timelines. Focusing on stacked object detection, which is a generally complicated scenario in construction sites, this paper proposes a novel framework by using both RGB and LiDAR for object clustering with low storage occupation and high detection speed that support real-time implementation. An RGB camera is first used to capture an image of the overall scene, and a pre-trained CNN network is applied to give the rough prediction of region-of-interests (ROIs) along with their confidence scores. The ROIs are linearly sorted based on their scores to select the potential stacked areas with low confidence. The center locations of ROIs are then transferred into the LiDAR system with the calibration matrix, and a Velodyne-16 scanner is used to perform adaptive scanning on the ROIs for detailed object clustering and detection. The result shows that given the pre-detected ROIs from RGB, the scanning time and computational time of clustering could be largely reduced. Furthermore, a confidence-based criterion is illustrated to linearly determine the required scanning frames to get desired detection results.
Get full access to this article
View all available purchase options and get full access to this chapter.
REFERENCES
Agüera-Vega, F., Agüera-Puntas, M., Martínez-Carricondo, P., Mancini, F., and Carvajal, F. (2020). “Effects of point cloud density, interpolation method and grid size on derived Digital Terrain Model accuracy at micro topography level.” International Journal of Remote Sensing, 41(21), 8281–8299.
Chen, J., Kira, Z., and Cho, Y. K. (2019). “Deep learning approach to point cloud scene understanding for automated scan to 3D reconstruction.” Journal of Computing in Civil Engineering, 33(4), 04019027.
Dara, S., and Tumma, P. (2018). Feature extraction by using deep learning: A survey. 2018 Second international conference on electronics, communication and aerospace technology (ICECA), IEEE.
Debeunne, C., and Vivet, D. (2020). “A review of visual-LiDAR fusion based simultaneous localization and mapping.” Sensors, 20(7), 2068.
Fan, Z., et al. (2021). Depth Ranging Performance Evaluation and Improvement for RGB-D Cameras on Field-Based High-Throughput Phenotyping Robots. 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE.
Fang, Q., Li, H., Luo, X., Li, C., and An, W. (2020). “A sematic and prior‐knowledge‐aided monocular localization method for construction‐related entities.” Computer‐Aided Civil and Infrastructure Engineering, 35(9), 979–996.
Fayyad, J., Jaradat, M. A., Gruyer, D., and Najjaran, H. (2020). “Deep learning sensor fusion for autonomous vehicle perception and localization: A review.” Sensors, 20(15), 4220.
Fung, M. L., et al. (2017). Sensor fusion: A review of methods and applications. 2017 29th Chinese Control And Decision Conference (CCDC), IEEE.
Kang, X., Li, J., Fan, X., and Wan, W. (2019). “Real-time rgb-d simultaneous localization and mapping guided by terrestrial lidar point cloud for indoor 3-d reconstruction and camera pose estimation.” Applied Sciences, 9(16), 3264.
Li, X., Han, S., Gül, M., and Al-Hussein, M. (2019). “Automated post-3D visualization ergonomic analysis system for rapid workplace design in modular construction.” Automation in Construction, 98, 160–174.
Liu, K., et al. (2022). D-lc-nets: Robust denoising and loop closing networks for lidar slam in complicated circumstances with noisy point clouds. 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE.
Liu, W., et al. (2016). Ssd: Single shot multibox detector. Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part I 14, Springer.
Liu, W., Sun, J., Li, W., Hu, T., and Wang, P. (2019). “Deep learning on point clouds and its application: A survey.” Sensors, 19(19), 4188.
Luo, H., Wang, M., Wong, P. K.-Y., and Cheng, J. C. (2020). “Full body pose estimation of construction equipment using computer vision and deep learning techniques.” Automation in construction, 110, 103016.
McMillan, B. (1953). “The basic theorems of information theory.” The Annals of mathematical statistics, 196–219.
Paneru, S., and Jeelani, I. (2021). “Computer vision applications in construction: Current state, opportunities & challenges.” Automation in Construction, 132, 103940.
Rao, A. S., Radanovic, M., Liu, Y., Hu, S., Fang, Y., Khoshelham, K., Palaniswami, M., and Ngo, T. (2022). “Real-time monitoring of construction sites: Sensors, methods, and applications.” Automation in Construction, 136, 104099.
Srivastava, R. K., Greff, K., and Schmidhuber, J. (2015). “Training very deep networks.” Advances in neural information processing systems, 28.
Tsai, D., et al. (2021). Optimising the selection of samples for robust lidar camera calibration. 2021 IEEE International Intelligent Transportation Systems Conference (ITSC), IEEE.
Xiao, X., Liu, B., Warnell, G., and Stone, P. (2022). “Motion planning and control for mobile robot navigation using machine learning: a survey.” Autonomous Robots, 46(5), 569–597.
Xu, F., Xia, P., You, H., and Du, J. (2022). “Robotic Cross-Platform Sensor Fusion and Augmented Visualization for Large Indoor Space Reality Capture.” Journal of Computing in Civil Engineering, 36(6), 04022036.
Yang, S., Zhao, C., Wu, Z., Wang, Y., Wang, G., and Li, D. (2022). “Visual SLAM based on semantic segmentation and geometric constraints for dynamic indoor environments.” IEEE Access, 10, 69636–69649.
Yeong, D. J., Velasco-Hernandez, G., Barry, J., and Walsh, J. (2021). “Sensor and sensor fusion technology in autonomous vehicles: A review.” Sensors, 21(6), 2140.
Yi, C., Zhang, Y., Wu, Q., Xu, Y., Remil, O., Wei, M., and Wang, J. (2017). “Urban building reconstruction from raw LiDAR point data.” Computer-Aided Design, 93, 1–14.
You, H., Xu, F., and Du, E. (2022). “Robot-Based Real-Time Point Cloud Digital Twin Modeling in Augmented Reality.” Transforming Construction with Reality Capture Technologies.
Information & Authors
Information
Published In
Construction Research Congress 2024
Pages: 1107 - 1116
History
Published online: Mar 18, 2024
ASCE Technical Topics:
- Adaptive systems
- Automation and robotics
- Computer models
- Computer vision and image processing
- Construction engineering
- Construction management
- Construction sites
- Engineering fundamentals
- Linear functions
- Mathematical functions
- Mathematics
- Methodology (by type)
- Models (by type)
- Systems engineering
- Systems management
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.