Learning Multi-Granularity Task Primitives from Construction Videos for Human-Robot Collaboration
Publication: Computing in Civil Engineering 2023
ABSTRACT
Human-robot collaboration (HRC) is an emerging solution for the construction industry’s productivity and safety challenges. The seamless HRC requires robots to understand the structure of construction tasks. Nevertheless, the implicit, dynamic construction task flow poses a non-trivial challenge. To address this challenge, a vision-based multi-granularity task’s primitive learning method is proposed. This study seeks to enhance the mutual understanding between workers and robots by determining which granularity level is best for the tasks’ understanding. Results show that the intermediate level has the best compromise between classification performance and embedded task knowledge. The outcomes will improve the smoothness of a HRC team.
Get full access to this article
View all available purchase options and get full access to this chapter.
REFERENCES
Bock, T., and T. Linner, eds. 2016. “Single-Task Construction Robots by Category.” Constr. Robot., 14–290. Cambridge: Cambridge University Press.
Carreira, J., and A. Zisserman. 2017. “Quo Vadis, Action Recognition? {A} New Model and the Kinetics Dataset.”.
Cheng, Y., L. Sun, C. Liu, and M. Tomizuka. 2020. “Towards Efficient Human-Robot Collaboration With Robust Plan Recognition and Trajectory Prediction.” IEEE Robot. Autom. Lett., 5 (2): 2602–2609. https://doi.org/10.1109/LRA.2020.2972874.
Gong, J., and C. H. Caldas. 2010. “Computer Vision-Based Video Interpretation Model for Automated Productivity Analysis of Construction Operations.” J. Comput. Civ. Eng., 24 (3): 252–263. American Society of Civil Engineers. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000027.
Gong, J., C. H. Caldas, and C. Gordon. 2011. “Learning and classifying actions of construction workers and equipment using Bag-of-Video-Feature-Words and Bayesian network models.” Adv. Eng. Informatics, 25 (4): 771–782. https://doi.org/10.1016/j.aei.2011.06.002.
Grigore, E. C., A. Roncone, O. Mangin, and B. Scassellati. 2018. “Preference-Based Assistance Prediction for Human-Robot Collaboration Tasks.” 2018 IEEE/RSJ Int. Conf. Intell. Robot. Syst., 4441–4448.
Hayes, B., and B. Scassellati. 2016. “Autonomously constructing hierarchical task networks for planning and human-robot collaboration.” 2016 IEEE Int. Conf. Robot. Autom., 5469–5476.
Lin, W., K. Hasenstab, G. Moura Cunha, and A. Schwartzman. 2020. “Comparison of handcrafted features and convolutional neural networks for liver MR image adequacy assessment.” Sci. Rep., 10 (1): 20336. https://doi.org/10.1038/s41598-020-77264-y.
Liu, Y., and H. Jebelli. 2022. “Intention Estimation in Physical Human-Robot Interaction in Construction.” Constr. Res. Congr. 2022 Comput. Appl. Autom. Data Anal. CRC 2022. American Society of Civil Engineers (ASCE).
Luo, X., H. Li, D. Cao, F. Dai, J. Seo, and S. Lee. 2018. “Recognizing Diverse Construction Activities in Site Images via Relevance Networks of Construction-Related Objects Detected by Convolutional Neural Networks.” J. Comput. Civ. Eng., 32 (3): 4018012. American Society of Civil Engineers. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000756.
Luo, X., H. Li, H. Wang, Z. Wu, F. Dai, and D. Cao. 2019. “Vision-based detection and visualization of dynamic workspaces.” Autom. Constr., 104: 1–13. https://doi.org/10.1016/j.autcon.2019.04.001.
Luo, X., H. Li, Y. Yu, C. Zhou, and D. Cao. 2020. “Combining deep features and activity context to improve recognition of activities of workers in groups.” Comput. Civ. Infrastruct. Eng., 35 (9): 965–978. John Wiley & Sons, Ltd. https://doi.org/10.1111/mice.12538.
Martinez, P., B. Barkokebas, F. Hamzeh, M. Al-Hussein, and R. Ahmad. 2021. “A vision-based approach for automatic progress tracking of floor paneling in offsite construction facilities.” Autom. Constr., 125: 103620. https://doi.org/10.1016/j.autcon.2021.103620.
Ruder, S. 2017. “An Overview of Multi-Task Learning in Deep Neural Networks.”.
Wu, H., H. Li, X. Fang, and X. Luo. 2022. “A survey on teaching workplace skills to construction robots.” Expert Syst. Appl., 205: 117658. https://doi.org/10.1016/j.eswa.2022.117658.
Zhang, J., P. Wang, and R. X. Gao. 2021. “Hybrid machine learning for human action recognition and prediction in assembly.” Robot. Comput. Integr. Manuf., 72: 102184. https://doi.org/10.1016/j.rcim.2021.102184.
Zhang, Y., and Q. Yang. 2022. “A Survey on Multi-Task Learning.” IEEE Trans. Knowl. Data Eng., 34 (12): 5586–5609. https://doi.org/10.1109/TKDE.2021.3070203.
Zheng, P., S. Li, L. Xia, L. Wang, and A. Nassehi. 2022. “A visual reasoning-based approach for mutual-cognitive human-robot collaboration.” CIRP Ann., 71 (1): 377–380. https://doi.org/10.1016/j.cirp.2022.04.016.
Information & Authors
Information
Published In
Computing in Civil Engineering 2023
Pages: 674 - 681
History
Published online: Jan 25, 2024
ASCE Technical Topics:
- Automation and robotics
- Business management
- Computer vision and image processing
- Construction engineering
- Construction industry
- Construction management
- Engineering fundamentals
- Flow (fluid dynamics)
- Fluid dynamics
- Fluid mechanics
- Hydrologic engineering
- Methodology (by type)
- Occupational safety
- Personnel management
- Practice and Profession
- Productivity
- Public administration
- Public health and safety
- Safety
- Systems engineering
- Water and water resources
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.