A Deep Learning and Vision-Based Solution for Material Volume Estimation Considering Devices’ Applications
Publication: Journal of Computing in Civil Engineering
Volume 38, Issue 1
Abstract
The estimation of material volume in a construction vehicle’s bucket is a crucial prerequisite for automation, as well as for productivity assessment and efficient material transport. Although some studies have been conducted in this field, the accuracy and speed of inference have been suboptimal, and specific implementation strategies have not been proposed. To address these issues, this paper proposes a new approach. The proposed approach has three main components. First, a novel image preprocessing framework based on three-dimensional (3D) grayscale terrain is presented. Second, a semantic mask-level data set is constructed to facilitate future research in this area. Third, a combined neural network and probabilistic approach is proposed to estimate the material volume, with speed and accuracy as metrics. Transfer learning is introduced to improve training efficiency and accuracy. The proposed material volume estimation method is implemented on three different devices, addressing the problem from the development phase to the application phase. The advantages and disadvantages of each device are discussed in depth. The results demonstrate that the proposed approach achieves an impressive average accuracy of 98.20% on all three devices, with real-time or semi–real-time volume estimation feasible on each. In summary, this paper proposes a new approach to estimate the material volume in a construction vehicle’s bucket, addressing issues of accuracy and speed of inference and providing specific implementation strategies. The results demonstrate the effectiveness of the proposed approach, which has potential applications in automation and productivity assessment in the construction industry.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
Some or all data, models, or code that support the findings of this study are available from the corresponding author upon reasonable request, including some images used for testing, model prototypes, and illustrative code for the postprocessing algorithm.
Acknowledgments
This work was supported by the National Natural Science Foundation of China (Grant/Award No. 52375098) and 2023 Jilin University Graduate Student Innovation Research Program Project (Grant/Award No. 451230411085) and National Natural Science Foundation of China (Grant/Award No. 52305275) and Fundamental Research Program of Shanxi Province (TZLH20230818004).
References
Adeli, H. 2001. “Neural networks in civil engineering: 1989–2000.” Comput.-Aided Civ. Infrastruct. Eng. 16 (2): 126–142. https://doi.org/10.1111/0885-9507.00219.
Anwar, H., S. M. Abbas, A. Muhammad, and K. Berns. 2014. “Volumetric estimation of contained soil using 3D sensors.” In Commercial Vehicle Technology 2014—Proc. of the Commercial Vehicle Technology Symp. (CVT), 11–13. Berlin: Springer.
Bugler, M., A. Borrmann, G. Ogunmakin, P. A. Vela, and J. Teizer. 2017. “Fusion of photogrammetry and video analysis for productivity assessment of earthwork processes.” Comput.-Aided Civ. Infrastruct. Eng. 32 (2): 107–123. https://doi.org/10.1111/mice.12235.
Concha-Meyer, A., J. Eifert, H. Wang, and G. Sanglay. 2018. “Volume estimation of strawberries, mushrooms, and tomatoes with a machine vision system.” Int. J. Food Prop. 21 (1): 1867–1874. https://doi.org/10.1080/10942912.2018.1508156.
Correia, B. A. B., R. Davies, F. D. Carvalho, F. C. Rodrigues, and D. W. Braggins. 1993. “Computer vision system for the automatic measurement of volumes of wood.” In Vol. 1989 of Computer vision for industry, 206–214. Bellingham, WA: International Society for Optical Engineering.
Dadhich, S., U. Bodin, and U. Andersson. 2016. “Key challenges in automation of earth-moving machines.” Autom. Constr. 68 (Aug): 212–222. https://doi.org/10.1016/j.autcon.2016.05.009.
Dadhich, S., F. Sandin, U. Bodin, U. Andersson, and T. Martinsson. 2019. “Field test of neural-network based automatic bucket-filling algorithm for wheel-loaders.” Autom. Constr. 97 (Jan): 1–12. https://doi.org/10.1016/j.autcon.2018.10.013.
Dehais, J., M. Anthimopoulos, S. Shevchik, and S. Mougiakakou. 2017. “Two-view 3D reconstruction for food volume estimation.” IEEE Trans. Multimedia 19 (5): 1090–1099. https://doi.org/10.1109/TMM.2016.2642792.
Ding, L., W. Fang, H. Luo, P. E. D. Love, B. Zhong, and X. Ouyang. 2018. “A deep hybrid learning model to detect unsafe behavior: Integrating convolution neural networks and long short-term memory.” Autom. Constr. 86 (Feb): 118–124. https://doi.org/10.1016/j.autcon.2017.11.002.
Fan, M., S. Lai, J. Huang, X. Wei, Z. Chai, J. Luo, and X. Wei. 2021. “Rethinking BiSeNet for real-time semantic segmentation.” In Proc., IEEE/CVF Conf. on Computer Vision and Pattern Recognition, 9711–9720. New York: IEEE.
Guevara, J., T. Arevalo-Ramirez, F. Yandun, M. Torres-Torriti, and F. A. Cheein. 2020. “Point cloud-based estimation of effective payload volume for earthmoving loaders.” Autom. Constr. 117 (Sep): 103207. https://doi.org/10.1016/j.autcon.2020.103207.
He, K., X. Zhang, S. Ren, and J. Sun. 2016. “Deep residual learning for image recognition.” In Proc., 2016 IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 770–778. New York: IEEE.
Hola, B., and K. Schabowicz. 2010. “Estimation of earthworks execution time cost by means of artificial neural networks.” Autom. Constr. 19 (5): 570–579. https://doi.org/10.1016/j.autcon.2010.02.004.
Holt, G. D., and D. Edwards. 2015. “Analysis of interrelationships among excavator productivity modifying factors.” Int. J. Prod. Perform. Manage. 64 (6): 853–869. https://doi.org/10.1108/IJPPM-02-2014-0026.
Horn, R., and M. J. D. Lebert. 1994. “Soil compactability and compressibility.” In Vol. 11 of Developments in agricultural engineering, 45–69. New York: Elsevier.
Howard, A., et al. 2019. “Searching for MobileNetV3.” In Proc., IEEE/CVF Int. Conf. on Computer Vision (ICCV), 1314–1324. New York: IEEE.
Howard, A. G., M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam. 2017. “MobileNets: Efficient convolutional neural networks for mobile vision applications.” Preprint, submitted April 17, 2017. https://arxiv.org/abs/1704.04861.
Huang, S.-C., F.-C. Cheng, and Y.-S. Chiu. 2013. “Efficient contrast enhancement using adaptive gamma correction with weighting distribution.” IEEE Trans. Image Process. 22 (3): 1032–1041. https://doi.org/10.1109/TIP.2012.2226047.
Kim, J., J. Hwang, S. Chi, and J. Seo. 2020. “Towards database-free vision-based monitoring on construction sites: A deep active learning approach.” Autom. Constr. 120 (Dec): 103376. https://doi.org/10.1016/j.autcon.2020.103376.
Kim, M., and M. G. Chung. 2008. “Recursively separated and weighted histogram equalization for brightness preservation and contrast enhancement.” IEEE Trans. Consum. Electron. 54 (3): 1389–1397. https://doi.org/10.1109/TCE.2008.4637632.
Kong, T., W. Fang, P. E. D. Love, H. Luo, S. Xu, and H. Li. 2021. “Computer vision and long short-term memory: Learning to predict unsafe behaviour in construction.” Adv. Eng. Inf. 50 (Oct): 101400. https://doi.org/10.1016/j.aei.2021.101400.
Lu, J., Q. Bi, Y. Li, and X. Li. 2020. “Estimation of fill factor for earth-moving machines based on 3D point clouds.” Measurement 165 (Dec): 108114. https://doi.org/10.1016/j.measurement.2020.108114.
Lu, J., Z. Yao, Q. Bi, and X. Li. 2021. “A neural network–based approach for fill factor estimation and bucket detection on construction vehicles.” Comput.-Aided Civ. Infrastruct. Eng. 36 (12): 1600–1618. https://doi.org/10.1111/mice.12675.
Ma, N., X. Zhang, H.-T. Zheng, and J. Sun. 2018. “Shufflenet V2: Practical guidelines for efficient CNN architecture design.” In Proc., 15th European Conf. on Computer Vision (ECCV), 122–138. New York: Springer.
Pfeiffer, S. A., J. Guevara, F. A. Cheein, and R. Sanz. 2018. “Mechatronic terrestrial LiDAR for canopy porosity and crown surface estimation.” Comput. Electron. Agric. 146 (Mar): 104–113. https://doi.org/10.1016/j.compag.2018.01.022.
Russakovsky, O., et al. 2015. “Imagenet large scale visual recognition challenge.” Int. J. Comput. Vision 115 (3): 211–252. https://doi.org/10.1007/s11263-015-0816-y.
Sanchez, M. C., M. Torres-Torriti, and F. A. Cheein. 2020. “Online inertial parameter estimation for robotic loaders.” In Proc., 21st IFAC World Congress on Automatic Control–Meeting Societal Challenges, 8763–8770. Amsterdam, Netherlands: Elsevier.
Sandler, M., A. Howard, M. Zhu, A. Zhmoginov, and L. C. Chen. 2018. Inverted residuals and linear bottlenecks: Mobile networks for classification, detection and segmentation. New York: IEEE. https://doi.org /10.1109/CVPR.2018.00474.
Shim, S., J. Kim, S.-W. Lee, and G.-C. Cho. 2021. “Road surface damage detection based on hierarchical architecture using lightweight auto-encoder network.” Autom. Constr. 130 (Oct): 103833. https://doi.org/10.1016/j.autcon.2021.103833.
Simonyan, K., and A. J. C. S. Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. Vienna, Austria: International Conference on Learning Representations.
Szegedy, C., W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. 2015. “Going deeper with convolutions.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 1–9. New York: IEEE.
Tan, M., B. Chen, R. Pang, V. Vasudevan, M. Sandier, A. Howard, and Q. V. Le. 2019. “MnasNet: Platform-aware neural architecture search for mobile.” In Proc., 32nd IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), 2815–2823. New York: IEEE.
Vallejos, R., F. Yandun, M. A. San Martin, V. Escobar, C. Roman, and F. A. Cheein. 2018. “Assessing the estimation of trawling catches using LiDAR sensor technology.” Ocean Coastal Manage. 165 (Nov): 99–108. https://doi.org/10.1016/j.ocecoaman.2018.08.016.
Wang, D. Y., and K. Wang. 2010. “Realization for dynamic weight system of loader.” In Proc., 3rd IEEE Int. Conf. on Computer Science and Information Technology (ICCSIT), 656–658. New York: IEEE.
Woolard, J. W., and J. D. Colby. 2002. “Spatial characterization, resolution, and volumetric change of coastal dunes using airborne LiDAR: Cape Hatteras, North Carolina.” Geomorphology 48 (1–3): 269–287. https://doi.org/10.1016/S0169-555X(02)00185-X.
Yang, D., Y. Wu, F. Sun, J. Chen, D. Zhai, and C. Fu. 2021. “Freeway accident detection and classification based on the multi-vehicle trajectory data and deep learning model.” Transp. Res. Part C Emerging Technol. 130 (Sep): 103303. https://doi.org/10.1016/j.trc.2021.103303.
Yu, C., J. Wang, C. Peng, C. Gao, G. Yu, and N. Sang. 2018. “BiSeNet: Bilateral segmentation network for real-time semantic segmentation.” In Proc., 15th European Conf. on Computer Vision (ECCV), 334–349. New York: Springer.
Zhang, D., M. Li, D. Tian, L. Song, and Y. Shen. 2022. “Intelligent text recognition based on multi-feature channels network for construction quality control.” Adv. Eng. Inf. 53 (Aug): 101669. https://doi.org/10.1016/j.aei.2022.101669.
Zhao, Y., Q. Chen, W. Cao, J. Yang, J. Xiong, and G. Gui. 2019. “Deep learning for risk detection and trajectory tracking at construction sites.” IEEE Access 7 (Mar): 30905–30912. https://doi.org/10.1109/ACCESS.2019.2902658.
Information & Authors
Information
Published In
Journal of Computing in Civil Engineering
Volume 38 • Issue 1 • January 2024
Copyright
© 2023 American Society of Civil Engineers.
History
Received: Mar 28, 2023
Accepted: Aug 5, 2023
Published online: Oct 11, 2023
Published in print: Jan 1, 2024
Discussion open until: Mar 11, 2024
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.