Abstract

Ensuring the resiliency of critical infrastructures is essential in modern society, but much of the deployed infrastructure has yet to fully leverage modern technical developments. This paper intersects two unique fields—deep learning and critical infrastructure protection—and illustrates how deep learning can improve resiliency within the electricity sector. Machine vision is the combination of machine intelligence, or computer systems automatically learning patterns from exemplar data, and image analysis, or objects of interest being automatically segmented and identified from video image data. This technology has the potential to automate threat assessments in the context of securing critical infrastructures. Rather than traditional reactionary approaches, we present here a method of leveraging deep learning for the detection of threats to critical infrastructures before failures occur. This paper discusses the state-of-the-art in deep learning for creating machine vision systems, and the concepts are applied to increase the resiliency of critical infrastructures. The intersection between machine vision and critical infrastructures is discussed, as are key benefits and challenges of invoking such an approach, and examples within several fields of critical infrastructures are presented. Automated inspection of the power infrastructure using vehicle-mounted video acquisition equipment is explored, and a proof-of-concept implementation of a deep convolutional neural network is developed, achieving 95.5% accuracy in distinguishing power-related infrastructures within images largely typical of rural settings. These preliminary results show promise in the application of deep learning and machine vision to protecting critical infrastructures through preventative maintenance.

Get full access to this article

View all available purchase options and get full access to this article.

References

Abadi, M., et al. 2016. “TensorFlow: Large-scale machine learning on heterogeneous distributed systems.” Preprint, submitted March 14, 2016. https://arxiv.org/abs/1603.04467.
Allison, R., J. Johnston, G. Craig, and S. Jennings. 2016. “Airborne optical and thermal remote sensing for wildfire detection and monitoring.” Sensors 16 (8): 1310. https://doi.org/10.3390/s16081310.
America’s Electric Cooperatives. 2017. “Can the infrastructure be fixed?” Accessed March 20, 2017. http://www.electric.coop/.
Anjomshoaa, A., F. Duarte, D. Rennings, T. Matarazzo, P. de Souza, and C. Ratti. 2018. “City scanner: Building and scheduling a mobile sensing platform for smart city services.” IEEE Internet Things J. 5 (6): 4567–4579. https://doi.org/10.1109/JIOT.2018.2839058.
Blasco, J., N. Aleixos, J. M. Roger, G. Rabatel, and E. Moltó. 2002. “AE—Automation and emerging technologies: Robotic weed control using machine vision.” Biosyst. Eng. 83 (2): 149–157. https://doi.org/10.1006/bioe.2002.0109.
Caetano, E., S. Silva, and J. Bateira. 2011. “A vision system for vibration monitoring of civil engineering structures.” Exp. Tech. 35 (4): 74–82. https://doi.org/10.1111/j.1747-1567.2010.00653.x.
Cai, Y., X. Sun, H. Wang, L. Chen, and H. Jiang. 2016. “Night-time vehicle detection algorithm based on visual saliency and deep learning.” J. Sensors 2016: 1–7. https://doi.org/10.1155/2016/8046529.
Casola, V., M. Esposito, F. Flammini, N. Mazzocca, and C. Pragliola. 2013. “Performance evaluation of video analytics for surveillance on-board trains.” In Proc., Int. Conf. on Advanced Concepts for Intelligent Vision Systems, 414–425. Cham, Switzerland: Springer.
Cha, Y.-J., W. Choi, and O. Büyüköztürk. 2017. “Deep learning-based crack damage detection using convolutional neural networks.” Comput.-Aided Civ. Infrastruct. Eng. 32 (5): 361–378. https://doi.org/10.1111/mice.12263.
Chen, Y.-R., K. Chao, and M. S. Kim. 2002. “Machine vision technology for agricultural applications.” Comput. Electron. Agric. 36 (2–3): 173–191. https://doi.org/10.1016/S0168-1699(02)00100-X.
Community and Regional Resilience Institute. 2013. Definitions of community resilience: An analysis. Washington, DC: Community and Regional Resilience Institute.
Conti, F., D. Rossi, A. Pullini, I. Loi, and L. Benini. 2016. “PULP: A ultra-low power parallel accelerator for energy-efficient and flexible embedded vision.” J. Signal Process. Syst. 84 (3): 339–354. https://doi.org/10.1007/s11265-015-1070-9.
Dick, K., F. Charih, and J. Green. 2018a. “High quality annotations of power infrastructure in Rural Ontario.” In Proc., Scholars Portal Dataverse. Toronto: Scholars Portal Dataverse.
Dick, K., F. Charih, Y. Souley Dosso, L. Russell, and J. R. Green. 2018b. “Systematic street view sampling.” In Proc., 15th Conf. on Computer and Robot Vision. Piscataway, NJ: IEEE.
Dodge, S., and L. Karam. 2016. “Understanding how image quality affects deep neural networks.” In Proc., 8th Int. Conf. on Quality of Multimedia Experience, QoMEX 2016. Lisbon, Portugal: IEEE Signal Processing Society.
Dutta, T., H. Sharma, A. Vellaiappan, and P. Balamuralidhar. 2015. “Image analysis-based automatic detection of transmission towers using aerial imagery.” In Proc., Iberian Conf. on Pattern Recognition and Image Analysis, 641–651. Cham, Switzerland: Springer.
Flammini, F., R. Naddei, C. Pragliola, and G. Smarra. 2016. “Towards automated drone surveillance in railways: State-of-the-art and future directions.” In Proc., Int. Conf. on Advanced Concepts for Intelligent Vision Systems, 336–348. Lecce, Italy: Springer.
German, S., J.-S. Jeon, Z. Zhu, C. Bearman, I. Brilakis, R. DesRoches, and L. Lowes. 2013. “Machine vision-enhanced postearthquake inspection.” J. Comput. Civ. Eng. 27 (6): 622–634. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000333.
Gibert, X., V. M. Patel, and R. Chellappa. 2017. “Deep multitask learning for railway track inspection.” IEEE Trans. Intell. Transp. Syst. 18 (1): 153–164. https://doi.org/10.1109/TITS.2016.2568758.
Government of Canada. 2017. “Critical infrastructure.” Accessed June 6, 2017. https://www.publicsafety.gc.ca/cnt/ntnl-scrt/crtcl-nfrstrctr/index-en.aspx.
Guikema, S. D., R. A. Davidson, and H. Liu. 2006. “Statistical models of the effects of tree trimming on power system outages.” IEEE Trans. Power Delivery 21 (3): 1549–1557. https://doi.org/10.1109/TPWRD.2005.860238.
Hochreiter, S., and J. Schmidhuber. 1997. “Long short-term memory” Neural Comput. 9 (8): 1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735.
Hubel, D. H., and T. N. Wiesel. 1962. “Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex.” J. Physiol. 160 (1): 106–154. https://doi.org/10.1113/jphysiol.1962.sp006837.
Krizhevsky, A. 2009. “CIFAR-10 and CIFAR-100 datasets.” Accessed February 1, 2018. https://www.cs.toronto.edu/~kriz/cifar.html.
Krizhevsky, A., I. Sutskever, and G. E. Hinton. 2012. “ImageNet classification with deep convolutional neural networks.” In Proc., Advances in Neural Information Processing Systems 25, 1097–1105. Red Hook, NY: Curran Associates, Inc.
Kubat, M., R. C. Holte, and S. Matwin. 1998. “Machine learning for the detection of oil spills in satellite radar images.” Mach. Learn. 30 (2/3): 195–215. https://doi.org/10.1023/A:1007452223027.
LeCun, Y., C. Cortes, and C. Burges. 1998 “MNIST handwritten digit database.” Accessed March 21, 2017. http://yann.lecun.com/exdb/mnist/.
Lee, S., J. Ha, M. Zokhirova, H. Moon, and J. Lee. 2018. “Background information of deep learning for structural engineering.” Arch. Comput. Methods Eng. 25 (1): 121–129. https://doi.org/10.1007/s11831-017-9237-0.
Li, S., Y. Dou, X. Niu, Q. Lv, and Q. Wang. 2016. “A fast and memory saved GPU acceleration algorithm of convolutional neural networks for target detection.” Neurocomputing 230: 48–59. https://doi.org/10.1016/j.neucom.2016.11.046.
Liu, Y., J. Yong, L. Liu, J. Zhao, and Z. Li. 2016. “The method of insulator recognition based on deep learning.” In Proc., 4th Int. Conf. on Applied Robotics for the Power Industry (CARPI), 1–5. Piscataway, NJ: IEEE.
Marvin, H. J. P., E. M. Janssen, Y. Bouzembrak, P. J. M. Hendriksen, and M. Staats. 2017. “Big data in food safety: An overview.” Crit. Rev. Food Sci. Nutr. 57 (11): 2286–2295. https://doi.org/10.1080/10408398.2016.1257481.
Mohanty, S. P., D. P. Hughes, and M. Salathé. 2016. “Using deep learning for image-based plant disease detection.” Front. Plant Sci. 7: 1419. https://doi.org/10.3389/fpls.2016.01419.
Oh, J.-K., G. Jang, S. Oh, J. H. Lee, B.-J. Yi, Y. S. Moon, J. S. Lee, and Y. Choi. 2009. “Bridge inspection robot system with machine vision.” Autom. Constr. 18 (7): 929–941. https://doi.org/10.1016/j.autcon.2009.04.003.
Park, S., I. Hong, J. Park, and H.-J. Yoo. 2016. “An energy-efficient embedded deep neural network processor for high speed visual attention in mobile vision recognition SoC.” IEEE J. Solid-State Circuits 51 (10): 2380–2388.
Public Safety Canada. 2015. “Enhancing critical infrastructure resiliency.” Accessed June 6, 2017. https://www.publicsafety.gc.ca/cnt/ntnl-scrt/crtcl-nfrstrctr/nhncng-rslnc-en.aspx.
Raina, R., A. Madhavan, and A. Y. Ng. 2009. “Large-scale deep unsupervised learning using graphics processors.” In Proc., 26th Annual Int. Conf. on Machine Learning—ICML ’09, 1–8. New York: Association for Computing Machinery.
Redzuwan, R. M., N. M. Din, M. Z. Baharuddin, I. S. Mustafa, and R. C. Omar. 2013. “Transmission tower environment monitoring using UAV.” In Vol. 16 of Proc., IOP Conf. Series: Earth and Environmental Science, 012105. Bristol, UK: IOP Publishing.
Russell, B. C., A. Torralba, K. P. Murphy, and W. T. Freeman. 2008. “LabelMe: A database and web-based tool for image annotation.” Int. J. Comput. Vision 77 (1–3): 157–173. https://doi.org/10.1007/s11263-007-0090-8.
Sampedro, C., C. Martinez, A. Chauhan, and P. Campoy. 2014. “A supervised approach to electric tower detection and classification for power line inspection.” In Proc., Int. Joint Conf. on Neural Networks, 1970–1977. Piscataway, NJ: IEEE.
Sathish, D., S. Kamath, K. V. Rajagopal, and K. Prasad. 2016. “Medical imaging techniques and computer aided diagnostic approaches for the detection of breast cancer with an emphasis on thermography: A review.” Int. J. Med. Eng. Inf. 8 (3): 275.
Simonyan, K., and A. Zisserman. 2014. “Very deep convolutional networks for large-scale image recognition.” Preprint, submitted September 4, 2014. https://arxiv.org/abs/1409.1556.
Sun, H., Z. Wang, J. Wang, Z. Huang, N. Carrington, and J. Liao. 2016. “Data-driven power outage detection by social sensors.” IEEE Trans. Smart Grid 7 (5): 2516–2524. https://doi.org/10.1109/TSG.2016.2546181.
Szegedy, C., W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. 2015. “Going deeper with convolutions.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition, 1–9. Piscataway, NJ: IEEE.
Taubenböck, H., T. Esch, M. Wurm, A. Roth, and S. Dech. 2010. “Object-based feature extraction using high spatial resolution satellite data of urban areas.” J. Spatial Sci. 55 (1): 117–132. https://doi.org/10.1080/14498596.2010.487854.
TensorFlow. n.d. “How to retrain inception’s final layer for new categories.” Accessed March 22, 2017. https://www.tensorflow.org/tutorials/image_retraining.
Thornton, J., J. Baran-Gale, and A. Yahr. 2009. “An assessment of the video analytics technology gap for transportation facilities.” In Proc., 2009 IEEE Conf. on Technologies for Homeland Security, 135–142. Piscataway, NJ: IEEE.
UAV America. n.d. “Electrical grid inspection.” Accessed March 30, 2017. http://uavamerica.com/electrical-grid-inspection/.
Varghese, A., J. Gubbi, H. Sharma, and P. Balamuralidhar. 2017. “Power infrastructure monitoring and damage detection using drone captured images.” In Proc., 2017 Int. Joint Conf. on Neural Networks (IJCNN), 1681–1687. Piscataway, NJ: IEEE.
Vazifeh, M. M., P. Santi, G. Resta, S. H. Strogatz, and C. Ratti. 2018. “Addressing the minimum fleet problem in on-demand urban mobility.” Nature 557 (7706): 534–538. https://doi.org/10.1038/s41586-018-0095-1.
Verma, B., L. Zhang, and D. Stockwell. 2017. “Deep learning techniques for roadside video data analysis.” In Roadside video data analysis, 119–157. Singapore: Springer.
Victor, D. G., and K. Yanosek. 2017. “The next energy revolution: The promise and peril of high-tech innovation.” Foreign Aff. 96: 124.
Wang, H., and Q. Chen. 2012. “An energy consumption model for GPU computing at instruction level.” 4 (2): 192–200.
Wu, G., H. Xiao, X. Xiao, Z. Huang, and Y. Li. 2010. “Transmission line inspection robot and deicing robot: Key technologies, prototypes and applications.” In Proc., 1st Int. Conf. on Applied Robotics for the Power Industry, CARPI 2010, 1–6. Piscataway, NJ: IEEE.
Wu, G., X. Xu, H. Xiao, J. Dai, X. Xiao, Z. Huang, and L. Ruan. 2008. “A novel self-navigated inspection robot along high-voltage power transmission line and its application.” In Proc., Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 1145–1154. Berlin, Heidelberg: Springer.
Yu, S., Y. Wu, W. Li, Z. Song, and W. Zeng. 2017. “A model for fine-grained vehicle classification based on deep learning.” Neurocomputing 257: 97–103. https://doi.org/10.1016/j.neucom.2016.09.116.
Zahodiakin, P. 2016. “Three-year research project looks at pole-top components, vegetation management, and more.” Accessed March 20, 2017. http://eprijournal.com/making-distribution-grids-stronger-more-resilient/.
Zhu, Z., and I. Brilakis. 2010. “Machine vision-based concrete surface quality assessment.” J. Constr. Eng. Manage. 136 (2): 210–218. https://doi.org/10.1061/(ASCE)CO.1943-7862.0000126.

Information & Authors

Information

Published In

Go to Journal of Infrastructure Systems
Journal of Infrastructure Systems
Volume 25Issue 2June 2019

History

Received: Mar 21, 2018
Accepted: Sep 13, 2018
Published online: Jan 31, 2019
Published in print: Jun 1, 2019
Discussion open until: Jun 30, 2019

Permissions

Request permissions for this article.

Authors

Affiliations

Ph.D. Candidate, Dept. of Systems and Computer Engineering, Carleton Univ., Ottawa, ON, Canada K1S 5B6. Email: [email protected]
Luke Russell [email protected]
Researcher, Dept. of Systems and Computer Engineering, Carleton Univ., Ottawa, ON, Canada K1S 5B6. Email: [email protected]
Yasmina Souley Dosso [email protected]
Ph.D. Candidate, Dept. of Systems and Computer Engineering, Carleton Univ., Ottawa, ON, Canada K1S 5B6. Email: [email protected]
Felix Kwamena, Ph.D. [email protected]
Adjunct Professor, Dept. of Systems and Computer Engineering, Carleton Univ., Ottawa, ON, Canada K1S 5B6. Email: [email protected]
Associate Professor, Dept. of Systems and Computer Engineering, Carleton Univ., Ottawa, ON, Canada K1S 5B6 (corresponding author). ORCID: https://orcid.org/0000-0002-6039-2355. Email: [email protected]

Metrics & Citations

Metrics

Citations

Download citation

If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.

Cited by

View Options

Get Access

Access content

Please select your options to get access

Log in/Register Log in via your institution (Shibboleth)
ASCE Members: Please log in to see member pricing

Purchase

Save for later Information on ASCE Library Cards
ASCE Library Cards let you download journal articles, proceedings papers, and available book chapters across the entire ASCE Library platform. ASCE Library Cards remain active for 24 months or until all downloads are used. Note: This content will be debited as one download at time of checkout.

Terms of Use: ASCE Library Cards are for individual, personal use only. Reselling, republishing, or forwarding the materials to libraries or reading rooms is prohibited.
ASCE Library Card (5 downloads)
$105.00
Add to cart
ASCE Library Card (20 downloads)
$280.00
Add to cart
Buy Single Article
$35.00
Add to cart

Get Access

Access content

Please select your options to get access

Log in/Register Log in via your institution (Shibboleth)
ASCE Members: Please log in to see member pricing

Purchase

Save for later Information on ASCE Library Cards
ASCE Library Cards let you download journal articles, proceedings papers, and available book chapters across the entire ASCE Library platform. ASCE Library Cards remain active for 24 months or until all downloads are used. Note: This content will be debited as one download at time of checkout.

Terms of Use: ASCE Library Cards are for individual, personal use only. Reselling, republishing, or forwarding the materials to libraries or reading rooms is prohibited.
ASCE Library Card (5 downloads)
$105.00
Add to cart
ASCE Library Card (20 downloads)
$280.00
Add to cart
Buy Single Article
$35.00
Add to cart

Media

Figures

Other

Tables

Share

Share

Copy the content Link

Share with email

Email a colleague

Share