Monocular Vision–Enabled 3D Truck Reconstruction: A Novel Optimization Approach Based on Parametric Modeling and Graphics Rendering
Publication: Journal of Computing in Civil Engineering
Volume 36, Issue 5
Abstract
Three-dimensional (3D) truck information, e.g., geometry, orientation, and position, can enable various smart construction applications such as monitoring earthwork, enhancing construction safety, and promoting productivity. Whereas stereo cameras have been explored extensively, the use of monocular vision (MV) for object 3D reconstruction still lacks substantial documentation. This study advances the field of MV-enabled 3D truck reconstruction by formulating it as an optimization problem. First, the general geometry of trucks was conceptualized and used to form a truck parametric model (TPM). Then the TPM was rendered by a computer graphics engine to generate synthetic views of the truck. Finally, an optimization algorithm is proposed to calibrate variables of the TPM progressively to maximize the alignment of the synthetic views with a target truck image. The proposed approach, called Mono-Truck, was evaluated by both lab tests and field experiments. The lab tests demonstrated an average error of 10.1%, 6.7 mm, and 0.7° in estimating the truck’s dimensions, position, and orientation, respectively. In the field experiments, Mono-Truck performed well compared with the baseline. This study contributes to the knowledge body by opening a new avenue to the monocular 3D truck reconstruction problem from an optimization perspective. The proposed approach can be generalized further to other types of construction machinery (e.g., excavators, cranes, and bulldozers) for their 3D reconstruction and smart applications.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
Select data and code generated during the study are available from the corresponding author by request, e.g., the graphics rendering code and the implementation of the optimization algorithm.
Acknowledgments
This research is jointly supported by the Strategic Public Policy Research (SPPR) Funding Scheme (Project No. S2018.A8.010.18S) and the Environment and Conservation Fund (ECF) (Project No. ECF 2019-111) of Hong Kong SAR.
References
Aguirre-Jofré, H., M. Eyre, S. Valerio, and D. Vogt. 2021. “Low-cost internet of things (IoT) for monitoring and optimising mining small-scale trucks and surface mining shovels.” Autom. Constr. 131 (May): 103918. https://doi.org/10.1016/j.autcon.2021.103918.
Ahn, C. R., S. Lee, and F. Peña-Mora. 2015. “Application of low-cost accelerometers for measuring the operational efficiency of a construction equipment fleet.” J. Comput. Civ. Eng. 29 (2): 04014042. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000337.
Akinci, B., F. Boukamp, C. Gordon, D. Huber, C. Lyons, and K. Park. 2006. “A formalism for utilization of sensor systems and integrated project models for active construction quality control.” Autom. Constr. 15 (2): 124–138. https://doi.org/10.1016/j.autcon.2005.01.008.
Azar, E. R., S. Dickinson, and B. McCabe. 2013. “Server-customer interaction tracker: Computer vision–based system to estimate dirt-loading cycles.” J. Constr. Eng. Manage. 139 (7): 785–794. https://doi.org/10.1061/(ASCE)CO.1943-7862.0000652.
Azar, E. R., and B. McCabe. 2012. “Automated visual recognition of dump trucks in construction videos.” J. Comput. Civ. Eng. 26 (6): 769–781. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000179.
Behzadan, A. H., Z. Aziz, C. J. Anumba, and V. R. Kamat. 2008. “Ubiquitous location tracking for context-specific information delivery on construction sites.” Autom. Constr. 17 (6): 737–748. https://doi.org/10.1016/j.autcon.2008.02.002.
Bender, F. A., S. Göltz, T. Bräunl, and O. Sawodny. 2017. “Modeling and offset-free model predictive control of a hydraulic mini excavator.” IEEE Trans. Autom. Sci. Eng. 14 (4): 1682–1694. https://doi.org/10.1109/TASE.2017.2700407.
Brilakis, I. 2016. “Entity matching across stereo cameras for tracking construction workers.” In Proc., 33rd Int. Symp. on Automation and Robotics in Construction (ISARC), 669–677. Berlin: International Association for Automation and Robotics in Construction.
Brilakis, I., M.-W. Park, and G. Jog. 2011. “Automated vision tracking of project related entities.” Adv. Eng. Inf. 25 (4): 713–724. https://doi.org/10.1016/j.aei.2011.01.003.
Brookshire, J. D. 2014. Articulated pose estimation via over-parametrization and noise projection. Cambridge, MA: Massachusetts Institute of Technology.
Brownlee, J. 2018. “Analytical vs numerical solutions in machine learning.” Accessed December 17, 2021. https://machinelearningmastery.com/analytical-vs-numerical-solutions-in-machine-learning/.
Caldas, C. H., D. G. Torrent, and C. T. Haas. 2006. “Using global positioning system to improve materials-locating processes on industrial projects.” J. Constr. Eng. Manage. 132 (7): 741–749. https://doi.org/10.1061/(ASCE)0733-9364(2006)132:7(741).
Chen, J., W. Lu, L. Yuan, Y. Wu, and F. Xue. 2022. “Estimating construction waste truck payload volume using monocular vision.” Resour. Conserv. Recycl. 177 (21): 106013. https://doi.org/10.1016/j.resconrec.2021.106013.
Chen, L.-C., G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille. 2017. “DeepLab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFS.” IEEE Trans. Pattern Anal. Mach. Intell. 40 (4): 834–848. https://doi.org/10.1109/TPAMI.2017.2699184.
Cho, Y. K., and M. Gai. 2014. “Projection-recognition-projection method for automatic object recognition and registration for dynamic heavy equipment operations.” J. Comput. Civ. Eng. 28 (5): A4014002. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000332.
Clancey, O., and M. Witten. 2011. “A memetic algorithm for dosimetric optimization in CyberKnife robotic radiosurgical treatment planning.” In Proc., 2011 IEEE Congress of Evolutionary Computation (CEC), 880–885. New York: IEEE.
Deere & Company. 2021. “Grade management.” Accessed November 16, 2021. https://www.deere.com/en/construction/construction-technology/grade-control/.
Fang, Q., H. Li, X. Luo, C. Li, and W. An. 2020. “A sematic and prior-knowledge-aided monocular localization method for construction-related entities.” Comput.-Aided Civ. Infrastruct. Eng. 35 (9): 979–996. https://doi.org/10.1111/mice.12541.
Feng, C., S. Dong, K. Lundeen, Y. Xiao, and V. Kamat. 2015. “Vision-based articulated machine pose estimation for excavation monitoring and guidance.” In Proc., ISARC Int. Symp. on Automation and Robotics in Construction, 1–10. Bogota, Colombia: IAARC Publications.
Feng, C., V. R. Kamat, and H. Cai. 2018. “Camera marker networks for articulated machine pose estimation.” Autom. Constr. 96 (3): 148–160. https://doi.org/10.1016/j.autcon.2018.09.004.
Golparvar-Fard, M., A. Heydarian, and J. C. Niebles. 2013. “Vision-based action recognition of earthmoving equipment using spatio-temporal features and support vector machine classifiers.” Adv. Eng. Inf. 27 (4): 652–663. https://doi.org/10.1016/j.aei.2013.09.001.
Gong, J., and C. H. Caldas. 2011. “An object recognition, tracking, and contextual reasoning-based video interpretation method for rapid productivity analysis of construction operations.” Autom. Constr. 20 (8): 1211–1226. https://doi.org/10.1016/j.autcon.2011.05.005.
Hassan, R., B. Cohanim, O. De Weck, and G. Venter. 2005. “A comparison of particle swarm optimization and the genetic algorithm.” In Proc., 46th AIAA/ASME/ASCE/AHS/ASC Structures, Structural Dynamics and Materials Conf. Reston, VA: American Institute of Aeronautics and Astronautics.
He, K., G. Gkioxari, P. Dollár, and R. Girshick. 2017. “Mask r-CNN.” In Proc., IEEE Int. Conf. on Computer Vision, 2961–2969. New York: IEEE.
Heydarian, A., M. Golparvar-Fard, and J. C. Niebles. 2012. “Automated visual recognition of construction equipment actions using spatio-temporal features and multiple binary support vector machines.” In Proc., Construction Research Congress 2012, 889–898. Reston, VA: ASCE.
Ibrahim, M., and O. Moselhi. 2014. “Automated productivity assessment of earthmoving operations.” J. Inf. Technol. Constr. 19 (9): 169–184.
Jo, B.-W., Y.-S. Lee, J.-H. Kim, D.-K. Kim, and P.-H. Choi. 2017. “Proximity warning and excavator control system for prevention of collision accidents.” Sustainability 9 (8): 1488. https://doi.org/10.3390/su9081488.
Kassem, E., W. Liu, T. Scullion, E. Masad, and A. Chowdhury. 2015. “Development of compaction monitoring system for asphalt pavements.” Constr. Build. Mater. 96 (Oct): 334–345. https://doi.org/10.1016/j.conbuildmat.2015.07.041.
Kato, H., D. Beker, M. Morariu, T. Ando, T. Matsuoka, W. Kehl, and A. Gaidon. 2020. “Differentiable rendering: A survey.” Preprint, submitted June 22, 2020. https://arxiv.org/abs/2006.12057.
Kim, D., S. Lee, and V. R. Kamat. 2020. “Proximity prediction of mobile objects to prevent contact-driven accidents in co-robotic construction.” J. Comput. Civ. Eng. 34 (4): 04020022. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000899.
Kim, D., M. Liu, S. Lee, and V. R. Kamat. 2019a. “Remote proximity monitoring between mobile construction resources using camera-mounted UAVs.” Autom. Constr. 99 (Mar): 168–182. https://doi.org/10.1016/j.autcon.2018.12.014.
Kim, H., C. R. Ahn, D. Engelhaupt, and S. Lee. 2018a. “Application of dynamic time warping to the recognition of mixed equipment activities in cycle time measurement.” Autom. Constr. 87 (45): 225–234. https://doi.org/10.1016/j.autcon.2017.12.014.
Kim, H., H. Kim, Y. W. Hong, and H. Byun. 2018b. “Detecting construction equipment using a region-based fully convolutional network and transfer learning.” J. Comput. Civ. Eng. 32 (2): 04017082. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000731.
Kim, J., S. Chi, and M. Choi. 2019b. “Sequential pattern learning of visual features and operation cycles for vision-based action recognition of earthmoving excavators.” Comput. Civ. Eng. 2019 (1): 298–304. https://doi.org/10.1061/(Ahttps://doi.org/10.1061/9780784482438.038.
Lee, D.-S., L. F. Gonzalez, J. Periaux, and K. Srinivas. 2008. “Robust design optimisation using multi-objective evolutionary algorithms.” Comput. Fluids 37 (5): 565–583. https://doi.org/10.1016/j.compfluid.2007.07.011.
Lee, H.-S., K.-P. Lee, M. Park, Y. Baek, and S. Lee. 2012. “RFID-based real-time locating system for construction safety management.” J. Comput. Civ. Eng. 26 (3): 366–377. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000144.
Liang, C.-J., K. M. Lundeen, W. McGee, C. C. Menassa, S. Lee, and V. R. Kamat. 2019. “A vision-based marker-less pose estimation system for articulated construction robots.” Autom. Constr. 104 (Apr): 80–94. https://doi.org/10.1016/j.autcon.2019.04.004.
Lim, R., H. La, and W. Sheng. 2014. “A robotic crack inspection and mapping system for bridge deck maintenance.” Autom. Sci. Eng. 11 (12): 367–378. https://doi.org/10.1109/TASE.2013.2294687.
Liu, D., J. Chen, and S. Li. 2019. “Collaborative operation and real-time control of roller fleet for asphalt pavement compaction.” Autom. Constr. 98 (Nov): 16–29. https://doi.org/10.1016/j.autcon.2018.11.005.
Liu, D., B. Cui, Y. Liu, and D. Zhong. 2013. “Automatic control and real-time monitoring system for earth–rock dam material truck watering.” Autom. Constr. 30 (Nov): 70–80. https://doi.org/10.1016/j.autcon.2012.11.007.
Lu, W. 2018. Smart construction objects (SCOs): An alternative way to smart construction. Reston, VA: ASCE.
Lu, W., J. Chen, and F. Xue. 2022. “Using computer vision to recognize composition of construction waste mixtures: A semantic segmentation approach.” Resour. Conserv. Recycl. 178 (21): 106022. https://doi.org/10.1016/j.resconrec.2021.106022.
Luo, X., H. Li, D. Cao, F. Dai, J. Seo, and S. Lee. 2018. “Recognizing diverse construction activities in site images via relevance networks of construction-related objects detected by convolutional neural networks.” J. Comput. Civ. Eng. 32 (3): 04018012. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000756.
Luo, X., H. Li, H. Wang, Z. Wu, F. Dai, and D. Cao. 2019. “Vision-based detection and visualization of dynamic workspaces.” Autom. Constr. 104 (4): 1–13. https://doi.org/10.1016/j.autcon.2019.04.001.
Mertan, A., D. J. Duff, and G. Unal. 2022. “Single image depth estimation: An overview.” Digital Signal Process. 123 (Apr): 103441. https://doi.org/10.1016/j.dsp.2022.103441.
Nicosia, G., and G. Stracquadanio. 2008. “Generalized pattern search algorithm for peptide structure prediction.” Biophys. J. 95 (10): 4988–4999. https://doi.org/10.1529/biophysj.107.124016.
Niu, Y., W. Lu, K. Chen, G. G. Huang, and C. Anumba. 2016. “Smart construction objects.” J. Comput. Civ. Eng. 30 (4): 04015070. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000550.
OSHA (Occupational Safety and Health Administration). 2013. “Census of fatal occupational injuries summary, 2012.” Accessed November 15, 2021. https://www.bls.gov/news.release/cfoi.nr0.htm.
Park, M.-W., and I. Brilakis. 2012. “Enhancement of construction equipment detection in video frames by combining with tracking.” Comput. Civ. Eng. 2012 (1): 421–428. https://doi.org/10.1061/9780784412343.0053.
Park, M.-W., C. Koch, and I. Brilakis. 2012. “Three-dimensional tracking of construction resources using an on-site camera system.” J. Comput. Civ. Eng. 26 (4): 541–549. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000168.
Péntek, Z., T. Hiller, T. Liewald, B. Kuhlmann, and A. Czmerk. 2017. “IMU-based mounting parameter estimation on construction vehicles.” In Proc., 2017 DGON Inertial Sensors and Systems (ISS), 1–14. New York: IEEE.
Pradhan, A., and B. Akinci. 2012. “Planning-based approach for fusing data from multiple sources for construction productivity monitoring.” J. Comput. Civ. Eng. 26 (4): 530–540. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000155.
Ronneberger, O., P. Fischer, and T. Brox. 2015. “U-net: Convolutional networks for biomedical image segmentation.” In Proc., Int. Conf. on Medical Image Computing and Computer-Assisted Intervention, 234–241. Berlin: Springer.
Sherafat, B., C. R. Ahn, R. Akhavian, A. H. Behzadan, M. Golparvar-Fard, H. Kim, Y.-C. Lee, A. Rashidi, and E. R. Azar. 2020. “Automated methods for activity recognition of construction workers and equipment: State-of-the-art review.” J. Constr. Eng. Manage. 146 (6): 03120002. https://doi.org/10.1061/(ASCE)CO.1943-7862.0001843.
Soltani, M. M., Z. Zhu, and A. Hammad. 2018. “Framework for location data fusion and pose estimation of excavators using stereo vision.” J. Comput. Civ. Eng. 32 (6): 04018045. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000783.
Song, H. A. 2021. “OpenGL transformation.” Accessed December 2, 2021. http://www.songho.ca/opengl/gl_transform.html#modelview.
Straw, A. 2011. “Augmented reality—Computing the OpenGL projection matrix from intrinsic camera parameters.” Accessed December 2, 2021. https://strawlab.org/2011/11/05/augmented-reality-with-OpenGL/.
Trimble. 2021. “Grade control for compact machines.” Accessed November 15, 2021. https://heavyindustry.trimble.com/en/products/grade-control-compact-machines.
Vahdatikhaki, F., A. Hammad, and H. Siddiqui. 2015. “Optimization-based excavator pose estimation using real-time location systems.” Autom. Constr. 56 (Mar): 76–92. https://doi.org/10.1016/j.autcon.2015.03.006.
Wang, C., and Y. K. Cho. 2015. “Smart scanning and near real-time 3D surface modeling of dynamic construction equipment from a point cloud.” Autom. Constr. 49 (6): 239–249. https://doi.org/10.1016/j.autcon.2014.06.003.
Xiao, B., and S.-C. Kang. 2021. “Vision-based method integrating deep learning detection for tracking multiple construction machines.” J. Comput. Civ. Eng. 35 (2): 04020071. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000957.
Xue, F., W. Lu, and K. Chen. 2018. “Automatic generation of semantically rich as-built building information models using 2D images: A derivative-free optimization approach.” Comput.-Aided Civ. Infrastruct. Eng. 33 (11): 926–942. https://doi.org/10.1111/mice.12378.
You, K., L. Ding, C. Zhou, Q. Dou, X. Wang, and B. Hu. 2021. “5G-based earthwork monitoring system for an unmanned bulldozer.” Autom. Constr. 131 (21): 103891. https://doi.org/10.1016/j.autcon.2021.103891.
Yu, G., Z. Mao, M. Hu, Z. Li, and V. Sugumaran. 2019. “BIM+ topology diagram-driven multiutility tunnel emergency response method.” J. Comput. Civ. Eng. 33 (6): 04019038. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000851.
Yu, G., Y. Wang, M. Hu, L. Shi, Z. Mao, and V. Sugumaran. 2021a. “RIOMS: An intelligent system for operation and maintenance of urban roads using spatio-temporal data in smart cities.” Future Gener. Comput. Syst. 115 (Sep): 583–609. https://doi.org/10.1016/j.future.2020.09.010.
Yu, G., Y. Wang, Z. Mao, M. Hu, V. Sugumaran, and Y. K. Wang. 2021b. “A digital twin-based decision analysis framework for operation and maintenance of tunnels.” Tunnelling Underground Space Technol. 116 (21): 104125. https://doi.org/10.1016/j.tust.2021.104125.
Yu, Y., H. Li, J. Cao, and X. Luo. 2021c. “Three-dimensional working pose estimation in industrial scenarios with monocular camera.” IEEE Internet Things J. 8 (3): 1740–1748. https://doi.org/10.1109/JIOT.2020.3014930.
Yuan, C., S. Li, and H. Cai. 2017. “Vision-based excavator detection and tracking using hybrid kinematic shapes and key nodes.” J. Comput. Civ. Eng. 31 (1): 04016038. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000602.
Zaheer, A., M. Rashid, M. A. Riaz, and S. Khan. 2018. “Single-View Reconstruction using orthogonal line-pairs.” Comput. Vision Image Understanding 172 (11): 107–123. https://doi.org/10.1016/j.cviu.2017.11.014.
Zhang, C., A. Hammad, and S. Rodriguez. 2012. “Crane pose estimation using UWB real-time location system.” J. Comput. Civ. Eng. 26 (5): 625–637. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000172.
Zhang, Q., T. Liu, Z. Zhang, Z. Huangfu, Q. Li, and Z. An. 2019. “Unmanned rolling compaction system for rockfill materials.” Autom. Constr. 100 (Apr): 103–117. https://doi.org/10.1016/j.autcon.2019.01.004.
Zhang, Z. 2000. “A flexible new technique for camera calibration.” IEEE Trans. Pattern Anal. Mach. Intell. 22 (11): 1330–1334. https://doi.org/10.1109/34.888718.
Information & Authors
Information
Published In
Journal of Computing in Civil Engineering
Volume 36 • Issue 5 • September 2022
Copyright
© 2022 American Society of Civil Engineers.
History
Received: Dec 23, 2021
Accepted: May 8, 2022
Published online: Jul 13, 2022
Published in print: Sep 1, 2022
Discussion open until: Dec 13, 2022
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
Cited by
- Frank Ato Ghansah, Weisheng Lu, Major opportunities of digital twins for smart buildings: a scientometric and content analysis, Smart and Sustainable Built Environment, 10.1108/SASBE-09-2022-0192, (2023).