Automated Human Use Mapping of Social Infrastructure by Deep Learning Methods Applied to Smart City Camera Systems
Publication: Journal of Computing in Civil Engineering
Volume 36, Issue 4
Abstract
With the emergence of the smart city, there is a growing need for scalable methods that sense how humans interact and use infrastructure in order to model social behaviors relevant to designing sustainable and resilient built environments. Cyber-physical system (CPS) frameworks used to monitor and automate infrastructure systems in smart cities can be extended to sense people to better understand how they use infrastructure systems including social infrastructure (e.g., parks, markets). This paper adopts convolutional neural network (CNN) architectures to automate the detection and spatiotemporal mapping of people using camera data to form a cyber-physical-social system (CPSS) for smart cities. The Mask region based convolutional neural network (R-CNN) detector was adopted and tailored to identify and segment human subjects in real time using camera images with an average speed of 7 frames per second. The Mask R-CNN framework was trained end to end using the Objects in Public Open Spaces (OPOS) image data set that includes classified segmentations of people in public spaces. A two-dimensional/three-dimensional (2D-3D) lifting algorithm based on a monocular camera calibration model was also employed to accurately position detected people in space. Finally, a Hungarian assignment algorithm based on association metrics extracted from detected people was used to assign people to spatiotemporal trajectories. To demonstrate the proposed framework, this study used the Detroit riverfront parks to study how people utilize community parks, which are a form of social infrastructure. The Mask R-CNN detector is proven precise in detecting and classifying the behavior of people in parks with mean average precision well above 85% for all class types defined in the OPOS library. The framework is also shown to be effective in spatially mapping the various uses of park furnishings, leading to better management of parks.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
All data, models, or code generated or used during the study are proprietary or confidential in nature and may only be provided with restrictions.
Acknowledgments
Thanks to the Detroit Riverfront Conservancy for the permission to use the video data, to Hao Zhou and Hsing-Yi Song for the help of camera calibration, to Curt Wolf for the coordination, and to Zohaib Hassan, Kevin Chase, and Kelsey Reindahl for data set annotation. This work is supported by the National Science Foundation (NSF) under Grant No. 1831347 and the Office of Naval Research (ONR) under Grant No. N00014-21-1-2033. Additional support was provided by the Michigan Institute for Data Science (MIDAS).
References
Adriaens, P. 2019. “‘Moneyball’ approach to closing the $2 trillion infrastructure finance gap.” Accessed January 7, 2019. https://thehill.com/opinion/energy-environment/451111-moneyball-approach-to-closing-the-2-trillion-infrastructure.
Bartos, M., B. Wong, and B. Kerkez. 2018. “Open storm: A complete framework for sensing and control of urban watersheds.” Environ. Sci.: Water Res. Technol. 4 (3): 346–358.
Bedimo-Rung, A. L., A. J. Mowen, and D. A. Cohen. 2005. “The significance of parks to physical activity and public health: A conceptual model.” Am. J. Preventive Med. 28 (2): 159–168. https://doi.org/10.1016/j.amepre.2004.10.024.
Bernardin, K., and R. Stiefelhagen. 2008. “Evaluating multiple object tracking performance: The clear mot metrics.” J. Image Video Process. 2008 (1): 1. https://doi.org/10.1155/2008/246309.
Bewley, A., Z. Ge, L. Ott, F. Ramos, and B. Upcroft. 2016. “Simple online and realtime tracking.” In Proc., 2016 IEEE Int. Conf. on Image Processing (ICIP), 3464–3468. New York: IEEE.
Crompton, J. L. 2005. “The impact of parks on property values: Empirical evidence from the past two decades in the united states.” Managing Leisure 10 (4): 203–218. https://doi.org/10.1080/13606710500348060.
Dalal, N., and B. Triggs. 2005. “Histograms of oriented gradients for human detection.” In Proc., 2005 IEEE Computer Society Conf. on Computer Vision and Pattern Recognition (CVPR’05), 886–893. New York: IEEE.
Deng, J., W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. 2009. “Imagenet: A large-scale hierarchical image database.” In Proc., 2009 IEEE Conf. on Computer Vision and Pattern Recognition, 248–255. New York: IEEE.
Elefteriadou, L. 2020. “Cyber physical systems in transportation: Traffic management with connected and autonomous vehicles.” In Cyber-physical systems in the built environment, edited by C. J. Anumba and N. Roofigari-Esfahan, 237–254. New York: Springer.
Everingham, M., L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman. 2010. “The PASCAL visual object classes (VOC) challenge.” Int. J. Comput. Vision 88 (2): 303–338. https://doi.org/10.1007/s11263-009-0275-4.
Fryar, C. D., Q. Gu, C. L. Ogden, and K. M. Flegal. 2016. “Anthropometric reference data for children and adults.” In Vital and health statistics. Series 3, Analytical studies, 1–46.
Gehl, J. 2013. Cities for people. Washington, DC: Island Press.
Girshick, R. 2015. “Fast R-CNN.” In Proc., IEEE Int. Conf. on Computer Vision, 1440–1448. New York: IEEE.
Girshick, R., J. Donahue, T. Darrell, and J. Malik. 2014. “Rich feature hierarchies for accurate object detection and semantic segmentation.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition, 580–587. New York: IEEE.
He, K., G. Gkioxari, P. Dollár, and R. Girshick. 2017. “Mask R-CNN.” In Proc., IEEE Int. Conf. on Computer vision, 2961–2969. New York: IEEE.
He, K., X. Zhang, S. Ren, and J. Sun. 2016. “Deep residual learning for image recognition.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition, 770–778. New York: IEEE.
Hou, Y.-L., Y. Song, X. Hao, Y. Shen, M. Qian, and H. Chen. 2018. “Multispectral pedestrian detection based on deep convolutional neural networks.” Infrared Phys. Technol. 94 (Nov): 69–77. https://doi.org/10.1016/j.infrared.2018.08.029.
Jacobs, J. 2020. The death and life of great American cities. London: Bodley Head.
Jeong, S., and K. Law. 2018. “An IoT platform for civil infrastructure monitoring.” In Proc., 2018 IEEE 42nd Annual Computer Software and Applications Conf. (COMPSAC), 746–754. New York: IEEE.
Kalman, R. E. 1960. “A new approach to linear filtering and prediction problems.” J. Basic Eng. 82 (1): 35–45. https://doi.org/10.1115/1.3662552.
Kaye, K. 2021. “Privacy concerns still loom over Detroit’s project green light.” Accessed January 2, 2021. https://www.smartcitiesdive.com/news/privacy-concerns-still-loom-over-detroits-project-green-light/594230/.
Krizhevsky, A., I. Sutskever, and G. E. Hinton. 2012. “Imagenet classification with deep convolutional neural networks.” Adv. Neural Inf. Process. Syst. 25 (Jan): 1097–1105.
Kuhn, H. W. 1955. “The Hungarian method for the assignment problem.” Nav. Res. Logist. Q. 2 (1–2): 83–97. https://doi.org/10.1002/nav.3800020109.
Lee, E. A. 2008. “Cyber physical systems: Design challenges.” In Proc., 2008 11th IEEE Int. Symp. on Object and Component-Oriented Real-Time Distributed Computing (ISORC), 363–369. New York: IEEE.
Lin, T.-Y., P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie. 2017. “Feature pyramid networks for object detection.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition, 2117–2125. New York: IEEE.
Lin, T.-Y., M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. 2014. “Microsoft COCO: Common objects in context.” In Proc., European Conf. on Computer Vision, 740–755. Berlin: Springer.
Liu, L., and M. B. Jensen. 2018. “Green infrastructure for sustainable urban water management: Practices of five forerunner cities.” Cities 74 (Apr): 126–133. https://doi.org/10.1016/j.cities.2017.11.013.
Liu, W., D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C. Berg. 2016. “SSD: Single shot multibox detector.” In Proc., European Conf. on Computer Vision, 21–37. Berlin: Springer.
Liu, Z., D.-S. Yang, D. Wen, W.-M. Zhang, and W. Mao. 2011. “Cyber-physical-social systems for command and control.” IEEE Intell. Syst. 26 (4): 92–96. https://doi.org/10.1109/MIS.2011.69.
Lowe, D. G. 1999. “Object recognition from local scale-invariant features.” In Proc., 7th IEEE Int. Conf. on Computer Vision, 1150–1157. New York: IEEE.
Luo, X., H. Li, D. Cao, F. Dai, J. Seo, and S. Lee. 2018. “Recognizing diverse construction activities in site images via relevance networks of construction-related objects detected by convolutional neural networks.” J. Comput. Civ. Eng. 32 (3): 04018012. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000756.
Luo, X., H. Li, H. Wang, Z. Wu, F. Dai, and D. Cao. 2019. “Vision-based detection and visualization of dynamic workspaces.” Autom. Constr. 104 (Aug): 1–13. https://doi.org/10.1016/j.autcon.2019.04.001.
Ma, C., C. Wan, Y. W. Chau, S. M. Kang, and D. Selviah. 2017. “Subway station real-time indoor positioning system for cell phones.” In Proc., 2017 Int. Conf. on Indoor Positioning and Indoor Navigation (IPIN), 1–7. New York: IEEE.
Ma, Y., S. Soatto, J. Kosecka, and S. S. Sastry. 2012. An invitation to 3-D vision: From images to geometric models. Berlin: Springer.
Milan, A., L. Leal-Taixé, I. Reid, S. Roth, and K. Schindler. 2016. “MOT16: A benchmark for multi-object tracking.” Preprint, submitted March 2, 2016. http://arxiv.org/abs/1603.00831.
Pan, S., N. Wang, Y. Qian, I. Velibeyoglu, H. Y. Noh, and P. Zhang. 2015. “Indoor person identification through footstep induced structural vibration.” In Proc., 16th Int. Workshop on Mobile Computing Systems and Applications. New York: Association for Computing Machinery.
Redmon, J., S. Divvala, R. Girshick, and A. Farhadi. 2016. “You only look once: Unified, real-time object detection.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition, 779–788. New York: IEEE.
Ren, S., K. He, R. Girshick, and J. Sun. 2015. “Faster R-CNN: Towards real-time object detection with region proposal networks.” Adv. Neural Inf. Process. Syst. 2015 (28): 91–99.
Sargano, A. B., P. Angelov, and Z. Habib. 2017. “A comprehensive review on handcrafted and learning-based action representation approaches for human activity recognition.” Appl. Sci. 7 (1): 110. https://doi.org/10.3390/app7010110.
Segen, J., and S. Pingali. 1996. “A camera-based system for tracking people in real time.” In. Vol. 3 of Proc., 13th Int. Conf. on Pattern Recognition, 63–67. New York: IEEE.
Shiomi, M., and N. Hagita. 2015. “Finding a person with a wearable acceleration sensor using a 3D position tracking system in daily environments.” Adv. Rob. 29 (23): 1563–1574. https://doi.org/10.1080/01691864.2015.1095651.
Silverman, B. W. 2018. Density estimation for statistics and data analysis. Boca Raton, FL: Routledge.
Simonyan, K., and A. Zisserman. 2014. “Very deep convolutional networks for large-scale image recognition.” Preprint, submitted September 4, 2014. http://arxiv.org/abs/1409.1556.
Sun, P., R. Hou, and J. Lynch. 2019. “Measuring the utilization of public open spaces by deep learning: A benchmark study at the Detroit riverfront.” In Proc., IEEE Winter Conf. on Applications of Computer. New York: IEEE.
Temple, J. A., and A. J. Reynolds. 2007. “Benefits and costs of investments in preschool education: Evidence from the child–parent centers and related programs.” Econ. Educ. Rev. 26 (1): 126–144. https://doi.org/10.1016/j.econedurev.2005.11.004.
Whitzman, C. 2001. “Social infrastructure in tall buildings: A tale of two towers.” In Tall buildings and urban habitat, 81–96. London: CRC Press.
Whyte, W. H. 1980. The social life of small urban spaces. Washington, DC: Conservation Foundation.
Wojke, N., and A. Bewley. 2018. “Deep cosine metric learning for person re-identification.” In Proc., 2018 IEEE Winter Conf. on Applications of Computer Vision (WACV), 748–756. New York: IEEE.
Wojke, N., A. Bewley, and D. Paulus. 2017. “Simple online and realtime tracking with a deep association metric.” In Proc., 2017 IEEE Int. Conf. on Image Processing (ICIP), 3645–3649. New York: IEEE.
Wu, D., N. Sharma, and M. Blumenstein. 2017. “Recent advances in video-based human action recognition using deep learning: A review.” In Proc., 2017 Int. Joint Conf. on Neural Networks (IJCNN), 2865–2872. New York: IEEE.
Yu, X., and Y. Xue. 2016. “Smart grids: A cyber-physical systems perspective.” Proc. IEEE 104 (5): 1058–1070. https://doi.org/10.1109/JPROC.2015.2503119.
Zagoruyko, S., and N. Komodakis. 2016. “Wide residual networks.” Preprint, submitted May 23, 2016. http://arxiv.org/abs/1605.07146.
Zhang, Z. 2000. “A flexible new technique for camera calibration.” IEEE Trans. Pattern Anal. Mach. Intell. 22 (11): 1330–1334. https://doi.org/10.1109/34.888718.
Zheng, L., Z. Bie, Y. Sun, J. Wang, C. Su, S. Wang, and Q. Tian. 2016. “Mars: A video benchmark for large-scale person re-identification.” In Proc., European Conf. on Computer Vision, 868–884. Berlin: Springer.
Information & Authors
Information
Published In
Journal of Computing in Civil Engineering
Volume 36 • Issue 4 • July 2022
Copyright
© 2022 American Society of Civil Engineers.
History
Received: Jun 5, 2021
Accepted: Dec 18, 2021
Published online: Apr 4, 2022
Published in print: Jul 1, 2022
Discussion open until: Sep 4, 2022
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.