Training a Visual Scene Understanding Model Only with Synthetic Construction Images
Publication: Computing in Civil Engineering 2021
ABSTRACT
While the use of deep neural networks (DNN) for computer vision is increasing in the construction domain, the shortage of training data sets prevents such models from achieving their maximum potential. To address this issue, we investigate the potential of using synthetic data for vision model development. Specifically, we synthesize construction images and train a DNN model only with the synthetic data. We then evaluate the performance of the synthetic data-trained model on a worker detection task, and the results demonstrate the great potential of synthetic images: 97.3% of mean average precision. Given the benefits of synthetic data—it is possible to automatically create an unlimited number of images without manual labeling—this finding is promising. Moreover, this approach can be readily applied to other computer vision tasks, without requiring the manual labeling. This finding will enable the creation of more accurate and scalable DNN models for construction applications.
Get full access to this chapter
View all available purchase options and get full access to this chapter.
REFERENCES
Bogo, F., Kanazawa, A., Lassner, C., Gehler, P., Romero, J., and Black, M. J. (2016). “Keep it SMPL: Automatic estimation of 3D human pose and shape from a single image.” Lecture Notes in Computer Science, 9909 LNCS, 561–578.
Braun, A., and Borrmann, A. (2019). “Combining inverse photogrammetry and BIM for automated labeling of construction site images for machine learning.” Automation in Construction, 106, 102879.
CMU Graphics Lab. (2008). “CMU Graphics Lab Motion Capture Database.” <http://mocap.cs.cmu.edu/>(Apr. 7, 2021).
Fang, Q., Li, H., Luo, X., Ding, L., Luo, H., Rose, T. M., and An, W. (2018a). “Detecting non-hardhat-use by a deep learning method from far-field surveillance videos.” Automation in Construction, 85, 1–9.
Fang, W., Ding, L., Luo, H., and Love, P. E. D. (2018b). “Falls from heights: A computer vision-based approach for safety harness detection.” Automation in Construction, 91, 53–61.
Fang, W., Ding, L., Zhong, B., Love, P. E. D., and Luo, H. (2018c). “Automated detection of workers and heavy equipment on construction sites: A convolutional neural network approach.” Advanced Engineering Informatics, 37, 139–149.
Han, S., and Lee, S. (2013). “A vision-based motion capture and recognition framework for behavior-based safety management.” Automation in Construction, 35, 131–141.
Han, S., Lee, S., and Peña-Mora, F. (2013). “Vision-based detection of unsafe actions of a construction worker: Case study of ladder climbing.” J. of Comp. in CE, 27(6), 635–644.
Kim, D., Lee, S., and Kamat, V. R. (2020a). “Proximity prediction of mobile objects to prevent contact-driven accidents in co-robotic construction.” Jl of Comp. in CE, 34(4), 04020022.
Kim, D., Liu, M., Lee, S., and Kamat, V. R. (2019). “Remote proximity monitoring between mobile construction resources using camera-mounted UAVs.” Automation in Construction, 99, 168–182.
Kim, H., Kim, H., Hong, Y. W., and Byun, H. (2018). “Detecting construction equipment using a region-based fully convolutional network and transfer learning.” Journal of Computing in Civil Engineering, 32(2), 04017082.
Kim, J., and Chi, S. (2019). “Action recognition of earthmoving excavators based on sequential pattern analysis of visual features and operation cycles.” Automat. in Const., 104, 255–264.
Kim, J., and Chi, S. (2020). “Multi-camera vision-based productivity monitoring of earthmoving operations.” Automation in Construction, 112, 103121.
Kim, J., Hwang, J., Chi, S., and Seo, J. (2020b). “Towards database-free vision-based monitoring on construction sites: A deep active learning approach.” Automat. in Constr., 120, 103376.
Lin, T. Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L. (2014). “Microsoft COCO: Common objects in context.” Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 8693 LNCS(PART 5), 740–755.
Liu, M., Han, S., and Lee, S. (2016). “Tracking-based 3D human skeleton extraction from stereo video camera toward an on-site safety and ergonomic analysis.” Construction Innovation, 16(3), 348–367.
Loper, M., Mahmood, N., Romero, J., Pons-Moll, G., and Black, M. J. (2015). “SMPL: A skinned multi-person linear model.” ACM Transactions on Graphics, 34(6), 1–16.
Luo, X., Li, H., Cao, D., Dai, F., Seo, J., and Lee, S. (2018). “Recognizing diverse construction activities in site images via relevance networks of construction-related objects detected by convolutional neural networks.” Journal of Computing in Civil Engineering, 32(3), 04018012.
Nath, N. D., Behzadan, A. H., and Paal, S. G. (2020). “Deep learning for site safety: Real-time detection of personal protective equipment.” Automation in Construction, 112, 103085.
Pavlakos, G., Zhu, L., Zhou, X., and Daniilidis, K. (2018). “Learning to estimate 3D human pose and shape from a single color image.” Proceedings of 2018 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, IEEE, Salt Lake City, U.S., 459–468.
Pi, Y., Nath, N. D., and Behzadan, A. H. (2020). “Convolutional neural networks for object detection in aerial imagery for disaster response and recovery.” Advanced Engineering Informatics, 43, 101009.
Roberts, D., and Golparvar-Fard, M. (2019). “End-to-end vision-based detection, tracking and activity analysis of earthmoving equipment filmed at ground level.” Automation in Construction, 105, 102811.
Seo, J., Han, S., Lee, S., and Kim, H. (2015a). “Computer vision techniques for construction safety and health monitoring.” Advanced Engineering Informatics, 29(2), 239–251.
Seo, J., Starbuck, R., Han, S., Lee, S., and Armstrong, T. J. (2015b). “Motion data-driven biomechanical analysis during construction tasks on sites.” Journal of Computing in Civil Engineering, 29(4), B4014005.
Soltani, M. M., Zhu, Z., and Hammad, A. (2016). “Automated annotation for visual recognition of construction resources using synthetic images.” Automation in Construction, 62, 14–23.
Son, H., Choi, H., Seong, H., and Kim, C. (2019). “Detection of construction workers under varying poses and changing background in image sequences via very deep residual networks.” Automation in Construction, 99, 27–38.
Torres Calderon, W., Roberts, D., and Golparvar-Fard, M. (2021). “Synthesizing pose sequences from 3D assets for vision-based activity analysis.” J. of Com. in CE, 35(1), 04020052.
Xuehui, A., Li, Z., Zuguang, L., Chengzhi, W., Pengfei, L., and Zhiwei, L. (2021). “Dataset and benchmark for detecting moving objects in construction sites” Auto. in Constr.,122, 103482.
Information & Authors
Information
Published In
Computing in Civil Engineering 2021
Pages: 221 - 229
History
Published online: May 24, 2022
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.