Synthetic Simulated Data for Construction Automation: A Review
Publication: Construction Research Congress 2024
ABSTRACT
The integration of deep learning (DL) technologies into construction offers great potential for promoting the level of automation in construction. However, the implementation of the DL model requires the acquisition of substantial data, which is error-prone and time-consuming. Additionally, due to safety and privacy concerns, not all real-world data can be retrieved. To address these issues, synthetic simulated data have emerged as promising alternatives, and various methods have been developed to generate such data. However, currently there is neither a summary of synthetic simulated data generation methods nor unified metrics to evaluate these methods. In this paper, a comprehensive review of 129 scholarly articles from Web of Science is conducted. Based on the source of data assets and the techniques employed for their combination, we categorize synthetic simulated data generation methods into seven distinct categories. Furthermore, we summarize seven metrics for evaluating these methods and consolidate the evaluation results in a table. The provided table serves as a reference for practitioners in identifying and selecting suitable synthetic simulated data generation methods for their applications.
Get full access to this article
View all available purchase options and get full access to this chapter.
REFERENCES
Abioye, S. O., Oyedele, L. O., Akanbi, L., Ajayi, A., Delgado, J. M. D., Bilal, M., Akinade, O. O., and Ahmed, A. (2021). Artificial intelligence in the construction industry: A review of present status, opportunities and future challenges. J. Build. Eng., 44, 103299.
Abu Alhaija, H., Mustikovela, S. K., Mescheder, L., Geiger, A., and Rother, C. (2018). Augmented reality meets computer vision: Efficient data generation for urban driving scenes. Int. J. Comput. Vision, 126(9), 961–972.
Abufadda, M., and Mansour, K. (2021). A Survey of Synthetic Data Generation for Machine Learning. In 2021 22nd International Arab Conference on Information Technology (ACIT), Muscat, Oman: IEEE, 1–7.
Adadi, A. (2021). A survey on data-efficient algorithms in big data era. J. Big Data, 8(1), 1–54.
Ahmad, S., Samarawickrama, K., Rahtu, E., and Pieters, R. (2021). Automatic Dataset Generation From CAD for Vision-Based Grasping. In 2021 20th International Conference on Advanced Robotics (ICAR), Ljubljana, Slovenia: IEEE, 715–721.
Barbosa, F., Woetzel, J., and Mischke, J. (2017). Reinventing construction: A route of higher productivity. McKinsey Global Institute.
Beery, S., Liu, Y., Morris, D., Piavis, J., Kapoor, A., Joshi, N., Meister, M., and Perona, P. (2020). Synthetic examples improve generalization for rare classes. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, Snowmass, CO: IEEE, 863–873.
Cauli, N., and Reforgiato Recupero, D. (2022). Survey on Videos Data Augmentation for Deep Learning Models. Future Internet, 14(3), 93.
Chen, Y., Rong, F., Duggal, S., Wang, S., Yan, X., Manivasagam, S., Xue, S., Yumer, E., and Urtasun, R. (2021). Geosim: Realistic video simulation via geometry-aware composition for self-driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN: IEEE, 7230–7240.
Darko, A., Chan, A. P., Adabre, M. A., Edwards, D. J., Hosseini, M. R., and Ameyaw, E. E. (2020). Artificial intelligence in the AEC industry: Scientometric analysis and visualization of research activities. Autom. Constr., 112, 103081.
de Melo, C. M., Torralba, A., Guibas, L., DiCarlo, J., Chellappa, R., and Hodgins, J. (2021). Next-generation deep learning based on simulators and synthetic data. Trends Cognit. Sci.
Deng, J., Dong, W., Socher, R., Li, L. J., Li, K., and Fei-Fei, L. (2009). Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, Miami, FL: IEEE, 248–255.
Ekanayake, B., Wong, J. K. W., Fini, A. A. F., and Smith, P. (2021). Computer vision-based interior construction progress monitoring: A literature review and future research directions. Autom. Constr., 127, 103705.
Gupta, A., Vedaldi, A., and Zisserman, A. (2016). Synthetic Data for Text Localisation in Natural Images. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV: IEEE, 2315–2324.
Hämäläinen, A., Arndt, K., Ghadirzadeh, A., and Kyrki, V. (2019). Affordance learning for end-to-end visuomotor robot control. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Macau, China: IEEE, 1781–1788.
Huang, L., Zhu, Z., and Zou, Z. (2023). To imitate or not to imitate: Boosting reinforcement learning-based construction robotic control for long-horizon tasks using virtual demonstrations. Autom. Constr., 146, 104691.
Jeon, H. G., Im, S., Lee, B. U., Choi, D. G., Hebert, M., and Kweon, I. S. (2019). Disc: A large-scale virtual dataset for simulating disaster scenarios. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Macau, China: IEEE, 187–194.
Kaur, D. P., Singh, N. P., and Banerjee, B. (2022). A review of platforms for simulating embodied agents in 3D virtual environments. Artif. Intell. Rev., 56(4), 3711–3753.
Lee, S., Jeong, M., Cho, C. S., Park, J., and Kwon, S. (2022). Deep Learning-Based PC Member Crack Detection and Quality Inspection Support Technology for the Precise Construction of OSC Projects. Applied Sciences, 12(19), 9810.
Li, W., et al. (2019). AADS: Augmented autonomous driving simulation using data-driven algorithms. Sci. Rob., 4(28), eaaw0863.
Li, Y. (2017). Deep reinforcement learning: An overview.
Lin, T. Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L. (2014). Microsoft coco: Common objects in context. In European conference on computer vision, Zurich, Switzerland: Springer, 740–755.
Lu, J., Richter, F., and Yip, M. C. (2022). Pose Estimation for Robot Manipulators via Keypoint Optimization and Sim-to-Real Transfer. IEEE Rob. Autom. Lett., 7(2), 4622–4629.
Neuhausen, M., Herbers, P., and König, M. (2020). Using synthetic data to improve and evaluate the tracking performance of construction workers on site. Applied Sciences, 10(14), 4948.
Nikolenko, S. I. (2021). Synthetic data for deep learning (Vol. 174). New York: Springer.
Pan, Y., and Zhang, L. (2021). Roles of artificial intelligence in construction engineering and management: A critical review and future trends. Autom. Constr., 122, 103517.
Papon, J., and Schoeler, M. (2015). Semantic Pose Using Deep Networks Trained on Synthetic RGB-D. In 2015 IEEE International Conference on Computer Vision (ICCV), Santiago, Chile: IEEE, 774–782.
Peres, R. S., Guedes, M., Miranda, F., and Barata, J. (2021). Simulation-based data augmentation for the quality inspection of structural adhesive with deep learning. IEEE Access, 9, 76532–76541.
Pivoňka, T., and Přeučil, L. (2020). Stereo Camera Simulation in Blender. In International Conference on Modelling and Simulation for Autonomous Systems, Prague, Czech Republic: Springer, 206–216.
Plunkett, G., Dixon, S., Deneke, W., and Harley, R. (2019). Designing a Simulation Platform for Generation of Synthetic Videos for Human Activity Recognition. In 2019 International Conference on Computational Science and Computational Intelligence (CSCI), Las Vegas, NV: IEEE, 635–640.
Rao, R., de Melo, C., and Krim, H. (2021). Synthetic environments for artificial intelligence (AI) and machine learning (ML) in multi-domain operations. In DEVCOM Army Research Laboratory.
Rudorfer, M., Neumann, L., and Krüger, J. (2019). Towards Learning 3d Object Detection and 6d Pose Estimation from Synthetic Data. In 2019 24th IEEE International Conference on Emerging Technologies and Factory Automation (ETFA), Zaragoza, Spain: IEEE, 1540–1543.
Talwar, D., Guruswamy, S., Ravipati, N., and Eirinaki, M. (2020). Evaluating Validity of Synthetic Data in Perception Tasks for Autonomous Vehicles. In 2020 IEEE International Conference On Artificial Intelligence Testing (AITest), Oxford, UK: IEEE, 73–80.
Teixeira, L., Oswald, M. R., Pollefeys, M., and Chli, M. (2020). Aerial single-view depth completion with image-guided uncertainty estimation. IEEE Rob. Autom. Lett., 5(2), 1055–1062.
Tobin, J., Fong, R., Ray, A., Schneider, J., Zaremba, W., and Abbeel, P. (2017). Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS), Vancouver, BC, Canada: IEEE, 23–30.
Tremblay, J., Prakash, A., Acuna, D., Brophy, M., Jampani, V., Anil, C., To, T., Cameracci, E., Boochoon, S., and Birchfield, S. (2018). Training deep networks with synthetic data: Bridging the reality gap by domain randomization. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, Salt Lake City, UT: IEEE, 969–977.
Xu, J., Yuan, C., Gu, J., Liu, J., An, J., and Kong, Q. (2023). Innovative synthetic data augmentation for dam crack detection, segmentation, and quantification. Struct. Health Monit., 22(4), 2402–2426.
Young, D., Panthi, K., and Noor, O. (2021). Challenges involved in adopting BIM on the construction jobsite. EPiC Series in Built Environment, 2, 302–310.
Zakour, M., Mellouli, A., and Chaudhari, R. (2021). HOIsim: Synthesizing Realistic 3D Human-Object Interaction Data for Human Activity Recognition. In 2021 30th IEEE International Conference on Robot & Human Interactive Communication (RO-MAN), Vancouver, BC, Canada: IEEE, 1124–1131.
Ziegeldorf, J. H., Morchon, O. G., and Wehrle, K. (2014). Privacy in the Internet of Things: threats and challenges. Secur. Commun. Netw., 7(12), 2728–2742.
Information & Authors
Information
Published In
Construction Research Congress 2024
Pages: 527 - 536
History
Published online: Mar 18, 2024
ASCE Technical Topics:
- Artificial intelligence and machine learning
- Automation and robotics
- Business management
- Computer programming
- Computing in civil engineering
- Construction engineering
- Construction management
- Data collection
- Engineering fundamentals
- Errors (statistics)
- Mathematics
- Measurement (by type)
- Methodology (by type)
- Metric systems
- Neural networks
- Practice and Profession
- Public administration
- Public health and safety
- Research methods (by type)
- Safety
- Statistics
- Systems engineering
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.