Automatic and Real-Time Joint Tracking and Three-Dimensional Scanning for a Construction Welding Robot
Publication: Journal of Construction Engineering and Management
Volume 150, Issue 3
Abstract
Although welding is one of the essential steel fabrication processes, the American Welding Society expects that the labor shortage in the United States will reach a deficit of 360,000 welders by 2027. Developing an automatic robotic welding system could potentially alleviate the labor shortage and provide better welding quality. As a first step, this paper designs a system pipeline that can automatically detect different welding joints and plan and track the joints’ trajectory with the initial point alignment in real time. There are rare studies that could achieve automatic initial point alignment in real time because the laser stripe’s deformation is not obvious at the narrow weld. In this study, the target joint’s endpoints were detected once the joint was detected on live video. Then, the joint trajectory was planned, and the robotic arm automatically aligned with the initial point and tracked the planned trajectory while scanning. The results demonstrate the accuracy and effectiveness of the proposed method.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
Some or all data, models, or code that support the findings of this study are available from the corresponding author upon reasonable request.
Acknowledgments
This work was partially based upon work supported by the National Science Foundation under Grant No. 2105555. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.
References
American Welding Society. 2021. “Highlighting the welding workforce demand in the US.” Accessed April 19, 2023. https://weldingworkforcedata.com/.
Balali, V., M. Noghabaei, A. Heydarian, and K. Han. 2018. “Improved stakeholder communication and visualizations: Real-time interaction and cost estimation within immersive virtual environments.” In Proc., Construction Research Congress 2018, 522–530. Reston, VA: ASCE.
Chen, X., and A. Yuille. 2014. “Articulated pose estimation by a graphical model with image dependent pairwise relations.” Adv. Neural Inf. Process Syst. 2 (Jan): 1736–1744.
Chen, X. Z., and S. B. Chen. 2010. “The autonomous detection and guiding of start welding position for arc welding robot.” Ind. Robot 37 (1): 70–78. https://doi.org/10.1108/01439911011009975.
Cordts, M., M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele. 2016. “The cityscapes dataset for semantic urban scene understanding.” In Proc., IEEE Conf. on Computer Vision And Pattern Recognition, 3213–3223. New York: IEEE.
Dinham, M., and G. Fang. 2013. “Autonomous weld seam identification and localisation using eye-in-hand stereo vision for robotic arc welding.” Robot Comput. Integr. Manuf. 29 (5): 288–301. https://doi.org/10.1016/j.rcim.2013.01.004.
Dyk, D. A. V., and X. L. Meng. 2001. “The art of data augmentation.” J. Comput. Graphical Stat. 10 (1): 1–50. https://doi.org/10.1198/10618600152418584.
EventHelix. 2023. “Issues in real-time system design.” Accessed April 27, 2023. https://www.eventhelix.com/embedded/issues-in-real-time-system-design/.
Everingham, M., S. M. A. Eslami, L. van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. 2015. “The Pascal visual object classes challenge: A retrospective.” Int. J. Comput. Vis. 111 (1): 98–136. https://doi.org/10.1007/s11263-014-0733-5.
Fan, J., S. Deng, F. Jing, C. Zhou, L. Yang, T. Long, and M. Tan. 2020. “An initial point alignment and seam-tracking system for narrow weld.” IEEE Trans. Ind. Inf. 16 (2): 877–886. https://doi.org/10.1109/TII.2019.2919658.
Fan, J., F. Jing, Z. Fang, and M. Tan. 2017. “Automatic recognition system of welding seam type based on SVM method.” Int. J. Adv. Manuf. Technol. 92 (1–4): 989–999. https://doi.org/10.1007/s00170-017-0202-8.
Fisher, R., S. Perkins, A. Walker, and E. Wolfart. 2003. “Hit-and-miss transform.” Hypermedia Image Processing Reference. Accessed October 26, 2021. https://homepages.inf.ed.ac.uk/rbf/HIPR2/hitmiss.htm.
Han, K., J. Degol, and M. Golparvar-Fard. 2018. “Geometry- and appearance-based reasoning of construction progress monitoring.” J. Constr. Eng. Manage. 144 (2): 04017110. https://doi.org/10.1061/(ASCE)CO.1943-7862.0001428.
Han, K., and M. Golparvar-Fard. 2017. “Potential of big visual data and building information modeling for construction performance analytics: An exploratory study.” Autom. Constr. 73 (Jan): 184–198. https://doi.org/10.1016/j.autcon.2016.11.004.
Idris, J., H. Kevin, and A. Alex. 2020. “Development of virtual reality and stereo-panoramic environments for construction safety training.” Eng. Constr. Archit. Manage. 27 (8): 1853–1876.
Intel. 2021. “At John Deere, ‘hard iron meets artificial intelligence.’” Accessed October 19, 2021. https://www.intel.com/content/www/us/en/newsroom/news/john-deere-hard-iron-meets-artificial-intelligence.html#gs.e68c3t.
Jubin Dipakkumar Kothari. 2018. “Detecting welding defects in steel plates using computer vision algorithms.” Int. J. Adv. Res. Electr. Electron. Instrum. Eng. 7 (9): 5. https://doi.org/10.15662/IJAREEIE.2018.0709013.
Kim, J.-S., G. Lee, and Y. S. Kim. 2022. “2D-LiDAR-sensor-based retaining wall displacement measurement system.” Appl. Sci. 12 (22): 11335. https://doi.org/10.3390/app122211335.
Lee, D. 2022a. “Automatic alignment to the welding joint using UR10e and Logitech Brio.” Accessed June 17, 2022. https://youtu.be/GzYtcJzFq6g.
Lee, D. 2022b. “Automatic welding joint scanning with Gocator 2350D and UR10e using Python.” Accessed June 17, 2022. https://youtu.be/bFKs24UZj8I.
Lee, D., and K. Han. 2020. “Vision-based quality assessment of prefabricated components using images and camera poses.” In Construction research congress 2020, 268–277. Reston, VA: ASCE.
Lee, D., and G. Nie. 2022. “WJ1000 dataset.” Accessed May 1, 2022. https://drive.google.com/drive/folders/1cw4jwXMwWbPrSJhyiIPxz0Qx_5pnPXiE?usp=sharing.
Lee, D., G. Y. Nie, and K. Han. 2023. “Vision-based inspection of prefabricated components using camera poses: Addressing inherent limitations of image-based 3D reconstruction.” J. Build. Eng. 64 (Apr): 105710. https://doi.org/10.1016/j.jobe.2022.105710.
Lee, D., G.-Y. Nie, and K. Han. 2022. “Real-time and automatic detection of welding joints using deep learning.” In Construction research congress 2022: Project management and controls, materials, and contracts, 601609. Reston, VA: ASCE.
Lei, T., W. Wang, Y. Rong, P. Xiong, and Y. Huang. 2020. “Cross-lines laser aided machine vision in tube-to-tubesheet welding for welding height control.” Opt. Laser Technol. 121 (Sep): 105796. https://doi.org/10.1016/j.optlastec.2019.105796.
Lin, J., K. Han, Y. Fukuchi, M. Eda, and M. Golparvar-Fard. 2015. “Model-based monitoring of work-in-progress via images taken by camera-equipped UAV and BIM.” In Proc., 2nd Int. Conf. on Civil and Building Engineering Informatics. Tokyo: Asian Group for Civil Engineering Informatics.
Lin, L., K. Wang, W. Zuo, M. Wang, J. Luo, and L. Zhang. 2016. “A deep structured model with radius–margin bound for 3D human activity recognition.” Int. J. Comput. Vis. 118 (2): 256–273. https://doi.org/10.1007/s11263-015-0876-z.
Martin, J. 1965. Programming real-time computer systems. Englewood Cliffs, NJ: Prentice-Hall.
MATLAB & Simulink. 2022. “Semantic segmentation.” Accessed November 7, 2022. https://www.mathworks.com/solutions/image-video-processing/semantic-segmentation.html.
Mohd Shah, H. N., M. Sulaiman, and A. Z. Shukor. 2017. “Autonomous detection and identification of weld seam path shape position.” Int. J. Adv. Manuf. Technol. 92 (9–12): 3739–3747. https://doi.org/10.1007/s00170-017-0380-4.
Nelson, J. 2021. “What is TensorRT? roboflow.” Accessed May 1, 2022. https://blog.roboflow.com/what-is-tensorrt/.
Nie, G., Y. Guo, Y. Liu, and Y. Wang. 2018. “Real-time salient object detection based on fully convolutional networks.” Commun. Comput. Inf. Sci. 757 (Nov): 189–198. https://doi.org/10.1007/978-981-10-7389-2_19.
Noghabaei, M., and K. Han. 2020. “Hazard recognition in an immersive virtual environment: Framework for the simultaneous analysis of visual search and EEG patterns.” In Proc., Construction Research Congress 2020, 934–943. Reston, VA: ASCE.
Noghabaei, M., and K. Han. 2021. “Object manipulation in immersive virtual environments: Hand motion tracking technology and snap-to-fit function.” Autom. Constr. 124 (Dec): 103594. https://doi.org/10.1016/j.autcon.2021.103594.
Noh, H., S. Hong, and B. Han. 2015. “Learning deconvolution network for semantic segmentation.” In Proc., IEEE Int. Conf. on Computer Vision, 1520–1528. New York: IEEE. https://doi.org/10.1109/ICCV.2015.178.
NVIDIA (Northern California Video and Audio). 2016. “Tensor RT.” Accessed December 22, 2020. https://developer.nvidia.com/tensorrt.
Preferred Networks. 2019. “CuPy: NumPy & SciPy for GPU.” Accessed May 1, 2022. https://cupy.dev/.
Rahman, M. M., Y. Tan, J. Xue, and K. Lu. 2020. “Recent advances in 3D object detection in the era of deep neural networks: A survey.” IEEE Trans. Image Process. 29 (10): 2947–2962. https://doi.org/10.1109/TIP.2019.2955239.
Rashmi. 2021. “Robotic welding market 2021 report: Top manufacturers research with top countries data, size, trends, share, and forecasts to 2030.” Accessed October 19, 2021. https://themarketwriteup.com/robotic-welding-market-2021-report-top-manufacturers-research-with-top-countries-data-size-trends-share-and-forecasts-to-2030/.
Seo, J., S. Han, S. Lee, and H. Kim. 2015. “Computer vision techniques for construction safety and health monitoring.” Adv. Eng. Inf. 29 (2): 239–251. https://doi.org/10.1016/j.aei.2015.02.001.
Shafeek, H. I., E. S. Gadelmawla, A. A. Abdel-Shafy, and I. M. Elewa. 2004. “Assessment of welding defects for gas pipeline radiographs using computer vision.” NDT & E Int. 37 (4): 291–299. https://doi.org/10.1016/j.ndteint.2003.10.003.
Shao, W. J., Y. Huang, and Y. Zhang. 2018. “A novel weld seam detection method for space weld seam of narrow butt joint in laser welding.” Opt. Laser Technol. 99 (Feb): 39–51. https://doi.org/10.1016/j.optlastec.2017.09.037.
Shorten, C., and T. M. Khoshgoftaar. 2019. “A survey on image data augmentation for deep learning.” J. Big Data 6 (1): 1–48. https://doi.org/10.1186/s40537-019-0197-0.
Tuckey. 2017. “Most common metal fabrication processes and applications.” Accessed September 28, 2021. https://www.tuckey.com/blog/common-metal-fabrication-processes-applications/.
VIATECHNIK. 2020. “Global steel consumption and the largest steel structures in the world.” Accessed September 28, 2021. https://www.viatechnik.com/global-steel-consumption-an-steel-structures-in-the-world/.
Voulodimos, A., N. Doulamis, A. Doulamis, and E. Protopapadakis. 2018. “Deep learning for computer vision: A brief review.” Comput. Intell. Neurosci. 2018 (Feb): 1–13. https://doi.org/10.1155/2018/7068349.
Wei, S., H. Ma, T. Lin, and S. Chen. 2010. “Autonomous guidance of initial welding position with ‘single camera and double positions’ method.” Sens. Rev. 30 (1): 62–68. https://doi.org/10.1108/02602281011010808.
Xiong, J., Y. Pi, and H. Chen. 2019. “Deposition height detection and feature point extraction in robotic GTA-based additive manufacturing using passive vision sensing.” Robot Comput. Integr. Manuf. 59 (Dec): 326–334. https://doi.org/10.1016/j.rcim.2019.05.006.
Xu, Y., N. Lv, G. Fang, S. Du, W. Zhao, Z. Ye, and S. Chen. 2017. “Welding seam tracking in robotic gas metal arc welding.” J. Mater. Process. Technol. 248 (Jan): 18–30. https://doi.org/10.1016/j.jmatprotec.2017.04.025.
Xue, B., B. Chang, G. Peng, Y. Gao, Z. Tian, D. Du, and G. Wang. 2019. “A vision based detection method for narrow butt joints and a robotic seam tracking system.” Sensors 19 (5): 1144. https://doi.org/10.3390/s19051144.
Yang, L., J. Fan, Y. Liu, E. Li, J. Peng, and Z. Liang. 2021. “Automatic detection and location of weld beads with deep convolutional neural networks.” IEEE Trans. Instrum. Meas. 70 (Sep): 1–12. https://doi.org/10.1109/TIM.2020.3026514.
Yen, S. B., and S. Francisco. 1990. “Automatic welding seam tracking and identification.” Microbiology 28 (9): 1877–1880. https://doi.org/10.1109/TIE.2017.2694399.
Yoo, H.-S., J.-H. L. Lee, Y.-S. Kim, and J.-R. Kim. 2004. “The development of a machine vision algorithm for automation of pavement crack sealing.” J. Constr. Eng. Manage. 5 (2): 90–105.
Yu, C., C. Gao, J. Wang, G. Yu, C. Shen, and N. Sang. 2018. “BiSeNet V2: Bilateral network with guided aggregation for real-time semantic segmentation.” In Proc., European Conf. on Computer Vision, 334–349. Berlin: Springer.
Zarenski, E. 2018. “Steel statistics and steel cost increase affect on construction?” Accessed September 28, 2021. https://edzarenski.com/2016/09/18/steel-statistics-and-steel-cost-increase-affect-on-construction-02-19/.
Zhang, T. Y., and C. Y. Suen. 1984. “A fast parallel algorithm for thinning digital patterns.” Commun. ACM 27 (3): 236–239. https://doi.org/10.1145/357994.358023.
Zhong, Z., L. Zheng, G. Kang, S. Li, and Y. Yang. 2020. “Random erasing data augmentation.” In Vol. 37 of Proc., AAAI Conf. on Artificial Intelligence, 13001–13008. Washington, DC: Association for the Advancement of Artificial Intelligence.
Information & Authors
Information
Published In
Journal of Construction Engineering and Management
Volume 150 • Issue 3 • March 2024
Copyright
© 2023 American Society of Civil Engineers.
History
Received: Jun 12, 2023
Accepted: Oct 11, 2023
Published online: Dec 21, 2023
Published in print: Mar 1, 2024
Discussion open until: May 21, 2024
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.