Multiimaging Sensor Data Fusion-Based Enhancement for 3D Workspace Representation for Remote Machine Operation
Publication: Journal of Construction Engineering and Management
Volume 139, Issue 4
Abstract
In incompletely characterized environments such as construction sites, remote machine operation is the preferred—and sometimes the only—safe and efficient solution for the operation of construction machines. When it comes to the operation of remote-controlled construction machines, a human—machine interface is needed so that even in the case of an unstructured environment (such as a construction site), the operator can interact with the machine in a safe and efficient manner. The human—machine interface needs to have the capability of realistically representing a three-dimensional (3D) workspace that provides information feedback to the remote operator. Workspace representation methods that are currently in use have certain limitations—they are time consuming and labor intensive and require high-performance computers. A major objective of this study is the development of an efficient means of representing a workspace in 3D that has the capacity to provide interactive visual feedback to the operator of remote-controlled construction machines. To achieve this objective, the ability is required to acquire dense, accurate, and visually realistic 3D data that can be converted into high-quality models. This allows creation of a realistic 3D workspace representation of terrain, including objects that might be in proximity to the machine. For this purpose, this study proposes a multiimaging sensor data fusion-based system employing joint bilateral upsampling, which enhances the quality and availability of the information acquired in such an environment. The field experiment results show that combining data acquired with a complementary multiimaging sensor setup allows enhancement of the quality of the information available to the remote operator. The resulting task-specific 3D workspace representation can be successfully incorporated into the development of remote-controlled construction machines that require interactive visual feedback. This provides the opportunity to make human—machine interaction more efficient and to improve the remote operation capability by assisting the operator. Moreover, the proposed multiimaging sensor data fusion-based 3D representation approach has the potential for effective use in a broad class of applications within the construction industry.
Get full access to this article
View all available purchase options and get full access to this article.
Acknowledgments
This research was supported by Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education, Science and Technology (2010-0023229).
References
Amenta, N., and Bern, M. (1999). “Surface reconstruction by Voronoi filtering.” Discrete Comput. Geom., 22(4), 481–504.
Asai, T., Kanbara, M., and Yokoya, N. (2004). “3D modeling of wide area outdoor environments by integrating omnidirectional range and color images.” Proc., 3rd IEEE/ACM Int. Symp. on Mixed and Augmented Reality, IEEE Computer Society, Washington, DC, 264–265.
Attali, D. (1998). “R-regular shape reconstruction from unorganized points.” Comput. Geom., 10(4), 239–247.
Bukchin, J., Luquer, R., and Shtub, A. (2002). “Learning in tele-operations.” IIE Trans., 34(3), 245–252.
Chan, D., Buisman, H., Theobalt, C., and Thrun, S. (2008). “A noise-aware filter for real-time depth upsampling.” Proc., Workshop on Multi-Camera and Multi-Modal Sensor Fusion Algorithms and Applications, Mitsubishi Electric Research Laboratories, Cambridge, MA, 1–12.
Choi, J., Min, D., and Sohn, K. (2010). “2D-plus-depth based resolution and frame-rate up-conversion technique for depth video.” IEEE Trans. Consum. Electron., 56(4), 2489–2497.
Crabb, R., Tracey, C., Purank, A., and Davis, J. (2008). “Real-time foreground segmentation via range and color imaging.” Proc., IEEE Computer Society Conf. on Computer Vision and Pattern Recognition Workshop, IEEE, New York, 1–5.
Diebel, J., and Thrun, S. (2005). “An application of Markov random fields to range sensing.” Proc., 19th Annual Conf. on Advances in Neural Information Processing Systems, Neural Information Proceeding Systems Foundation, La Jolla, CA, 291–298.
Fong T. (2001). “Collaborative control: A robot-centric model for vehicle teleoperation.” Ph.D. dissertation, Dept. of Mechanical Engineering, Carnegie Mellon Univ., Pittsburgh.
Fruh, C., and Zakhor, A. (2005). “Data processing algorithms for generating textured 3D building facade meshes from laser scans and camera images.” Int. J. Comput. Vision, 61(2), 159–184.
Garcia, F., Mirbach, B., Ottersten, B., Grandidier, F., and Cuesta, A. (2010). “Pixel weighted average strategy for depth sensor data fusion.” Proc., 17th IEEE Int. Conf. on Image Processing, IEEE, New York, 2805–2808.
Gloud, S., Baumstarck, P., Quigley, M., Ng, A. Y., and Daphne, K. (2008). “Integrating visual and range data for robotic object detection.” Proc., Workshop on Multi-Camera and Multi-Modal Sensor Fusion Algorithms and Applications, Mitsubishi Electric Research Laboratories, Cambridge, MA, 1–12.
Gong, J., and Caldas, C. H. (2008). “Data processing for real-time construction site spatial modeling.” Autom. Constr., 17(5), 526–535.
Hahne U., and Alexa M. (2009). “Depth imaging by combining time-of-flight and on-demand stereo.” Dynamic 3D imaging, lecture notes in computer science, Springer-Verlag, Berlin, 70–83.
Heikkila, J., and Silven, O. (1997). “A four-step camera calibration procedure with implicit image correction.” Proc., IEEE Computer Society Conf. on Computer Vision and Pattern Recognition, IEEE, New York, 1106–1112.
Huber, D., Herman, H., Kelly, A., Rander, P., and Ziglar, J. (2009). “Real-time photo-realistic visualization of 3D environments for enhanced tele-operation of vehicles.” Proc., IEEE 12th Int. Conf. on 3-D Digital Imaging and Modeling, IEEE, New York, 1518–1525.
Huhle, B., Schairer, T., Jenke, P., and Strasser, W. (2010). “Fusion of range and color images for denoising and resolution enhancement with a non-local filter.” Comput. Vision Image Understanding, 114(12), 1336–1345.
Kim, C., Haas, C. T., and Liapi, K. A. (2005). “Rapid, on-site spatial information acquisition and its use for infrastructure operation and maintenance.” Autom. Constr., 14(5), 666–684.
Kim, C., Haas, C. T., Liapi, K. A., and Caldas, C. H. (2006). “Human-assisted obstacle avoidance system using 3D workspace modeling for construction equipment operation.” J. Comput. Civil Eng., 20(3), 177–186.
Kopf, J., Cohen, M. F., Lischinski, D., and Uyttendaele, M. (2007). “Joint bilateral upsampling.” ACM Trans. Graphics, 26(3), 839–846.
Meier, R., Fong, T., Thorpe, C., and Baur, C. (1999). “A sensor fusion based user interface for vehicle teleoperation.” Proc., Int. Conf. on Field and Service Robots, Pittsburgh, PA.
Oloufa, A. A., Ikeda, M., and Oda, H. (2003). “Situational awareness of construction equipment using GPS, wireless and web technologies.” Autom. Constr., 12(6), 737–748.
Schiller I., and Koch R. (2009). “Data structures for capturing dynamic scenes with a time-of-flight camera.” Dynamic 3D imaging, lecture notes in computer science, Springer-Verlag Berlin Heidelberg, New York, NY, 42–57.
Schuon, S., Theobalt, C., Davis, J., and Thrun, S. (2008). “High-quality scanning using time-of-flight depth superresolution.” Proc., IEEE Computer Vision Society Conf. on Computer Vision and Pattern Recognition Workshop, IEEE, New York, 1–7.
Seo, J., Lee, S., Kim, J., and Kim, S. (2011). “Task planner design for an automated excavation system.” Autom. Constr., 20(7), 954–966.
Son, H., Kim, C., and Choi, K. (2010). “Rapid 3D object detection and modeling using range data from 3D range imaging camera for heavy equipment operation.” Autom. Constr., 19(7), 898–906.
Stentz, A., Bares, J., Singh, S., and Rowe, P. (1999). “A robotic excavator for autonomous truck loading.” Autonom. Rob., 7(2), 175–186.
Teizer, J., Caldas, C. H., and Haas, C. T. (2007). “Real-time three-dimensional occupancy grid modeling for the detection and tracking of construction resources.” J. Constr. Eng. Manage., 133(11), 880–888.
Tichon, J., and Diver, P. (2010). “Plant operator simulation: Benefits and drawbacks for a construction training organization.” Cognit. Technol. Work, 12(3), 219–229.
Tomasi, C., and Manduchi, R. (1998). “Bilateral filtering for gray and color images.” Proc., 6th Int. Conf. on Computer Vision, IEEE Computer Society, Washington, DC, 839–846.
Wang, Q., Li, Q., Chen, Z., Sun, J., and Yao, R. (2009). “Range image noise suppression in laser imaging system.” Opt. Laser Technol., 41(2), 140–147.
Xiang, X., Li, G., Tong, J., and Pan, Z. (2010). “Fast and simple super resolution for range data.” Proc., Int. Conf. on Cyberworlds, IEEE Computer Society, Washington, DC, 319–324.
Yang, Q., Tan, K., Culbertson, B., and Apostolopoulos, J. (2010). “Fusion of active and passive sensors for fast 3D capture.” Proc., IEEE Int. Workshop on Multimedia Signal Processing, IEEE, New York, 69–74.
Yang, Q. X., Yang, R. G., Davis, J., and Nister, D. (2007). “Spatial-depth super resolution for range images.” Proc., IEEE Computer Vision and Pattern Recognition, IEEE, New York, 1–8.
Zaatri, A., and Oussalah, M. (2003). “Integration and design of multi-modal interfaces for supervisory control systems.” Inf. Fusion, 4(2), 135–150.
Zhang, Z. (2000). “A flexible new technique for camera calibration.” IEEE Trans. Pattern Anal. Mach. Intell., 22(11), 1330–1334.
Zhu, J., Wang, L., Gao, J., and Yang, R. (2010). “Spatial-temporal fusion for high accuracy depth maps using dynamic MRFs.” IEEE Trans. Pattern Anal. Mach. Intell., 32(5), 899–909.
Information & Authors
Information
Published In
Journal of Construction Engineering and Management
Volume 139 • Issue 4 • April 2013
Pages: 434 - 444
Copyright
© 2013 American Society of Civil Engineers.
History
Received: Oct 31, 2011
Accepted: Jun 13, 2012
Published online: Jul 25, 2012
Published in print: Apr 1, 2013
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.