Multi-Objective Reinforcement Learning for Autonomous Drone Navigation in Urban Area
Publication: Construction Research Congress 2024
ABSTRACT
Unmanned aerial vehicles (drones) are widely used for reality capture, data collection, and aerial logistics in multiple industries. Existing research in autonomous drones focuses much on path planning and automated obstacle avoidance. However, in complex environments such as urban areas, a safe and efficient drone navigation depends on many more dynamic constraints, such as layout of buildings, local aerodynamics (e.g., side winds), signal coverage, and malicious attacks. And many of these factors are interdependent, such as the wind field being affected by the geometric features of nearby buildings. This paper proposes a multi-objective reinforcement learning algorithm for the drone’s path planning while satisfying a variety of interdependent constraint conditions. The drone will be able to sense multiple environmental and physical conditions of the near field, and develop dynamic policies for prioritizing the navigational decision to optimize the path while minimizing the negative environmental impact. A policy network will be trained to set priority weights for specific environment and physical factors at a given point. A simulation study is performed to test the proposed algorithm in an urban aerial logistics case.
Get full access to this article
View all available purchase options and get full access to this chapter.
REFERENCES
Albert, A., Leira, F. S., and Imsland, L. S. (2017). UAV path planning using MILP with experiments.
Andersson, O., Wzorek, M., Rudol, P., and Doherty, P. “Model-predictive control with stochastic collision avoidance using bayesian policy optimization.” Proc., 2016 IEEE International Conference on Robotics and Automation (ICRA), IEEE, 4597–4604.
Baca, T., Hert, D., Loianno, G., Saska, M., and Kumar, V. “Model predictive trajectory tracking and collision avoidance for reliable outdoor deployment of unmanned aerial vehicles.” Proc., 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, 6753–6760.
Biao, L., Cunyan, J., Lu, W., Weihua, C., and Jing, L. (2019). “A parametric study of the effect of building layout on wind flow over an urban area.” Building and Environment, 160, 106160.
Blocken, B., Stathopoulos, T., and Carmeliet, J. (2008). “Wind environmental conditions in passages between two long narrow perpendicular buildings.” Journal of Aerospace Engineering, 21(4), 280–287.
Bolognini, M., and Fagiano, L. (2020). “Lidar-based navigation of tethered drone formations in an unknown environment.” IFAC-PapersOnLine, 53(2), 9426–9431.
Cadena, C., Carlone, L., Carrillo, H., Latif, Y., Scaramuzza, D., Neira, J., Reid, I., and Leonard, J. J. (2016). “Past, present, and future of simultaneous localization and mapping: Toward the robust-perception age.” IEEE Transactions on robotics, 32(6), 1309–1332.
Capolupo, A., Pindozzi, S., Okello, C., Fiorentino, N., and Boccia, L. (2015). “Photogrammetry for environmental monitoring: The use of drones and hydrological models for detection of soil contaminated by copper.” Science of the Total Environment, 514, 298–306.
Chakravarty, P., Kelchtermans, K., Roussel, T., Wellens, S., Tuytelaars, T., and Van Eycken, L. “CNN-based single image obstacle avoidance on a quadrotor.” Proc., 2017 IEEE international conference on robotics and automation (ICRA), IEEE, 6369–6374.
El-Sheimy, N., and Li, Y. (2021). “Indoor navigation: State of the art and future trends.” Satellite Navigation, 2(1), 1–23.
Entrop, A., and Vasenev, A. (2017). “Infrared drones in the construction industry: designing a protocol for building thermography procedures.” Energy procedia, 132, 63–68.
Esposito, M., Crimaldi, M., Cirillo, V., Sarghini, F., and Maggio, A. (2021). “Drone and sensor technology for sustainable weed management: A review.” Chemical and Biological Technologies in Agriculture, 8(1), 1–11.
Fan, Y., Chu, S., Zhang, W., Song, R., and Li, Y. “Learn by observation: Imitation learning for drone patrolling from videos of a human navigator.” Proc., 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, 5209–5216.
Floreano, D., and Wood, R. J. (2015). “Science, technology and the future of small autonomous drones.” nature, 521(7553), 460–466.
García Carrillo, L. R., Dzul López, A. E., Lozano, R., and Pégard, C. (2012). “Combining stereo vision and inertial navigation system for a quad-rotor UAV.” Journal of intelligent & robotic systems, 65(1-4), 373–387.
Hayes, J., Dwivedi, S., Karthikeyan, R., Abujelala, M., Kang, J., Ye, Y., Du, E., and Mehta, R. K. “Identifying Early Predictors of Learning in VR-based Drone Training.” Proc., Proceedings of the Human Factors and Ergonomics Society Annual Meeting, SAGE Publications Sage CA: Los Angeles, CA, 1872–1876.
Hochreiter, S., and Schmidhuber, J. (1997). “Long short-term memory.” Neural computation, 9(8), 1735–1780.
Huang, C., Zhou, X., Ran, X., Wang, J., Chen, H., and Deng, W. (2023). “Adaptive cylinder vector particle swarm optimization with differential evolution for UAV path planning.” Engineering Applications of Artificial Intelligence, 121, 105942.
Imrane, M. L., Melingui, A., Mvogo Ahanda, J. J. B., Biya Motto, F., and Merzouki, R. (2021). “Artificial potential field neuro-fuzzy controller for autonomous navigation of mobile robots.” Proceedings of the Institution of Mechanical Engineers, Part I: Journal of Systems and Control Engineering, 235(7), 1179–1192.
Jasak, H. (2009). “OpenFOAM: open source CFD in research and industry.” International Journal of Naval Architecture and Ocean Engineering, 1(2), 89–94.
Jung, S., Hwang, S., Shin, H., and Shim, D. H. (2018). “Perception, guidance, and navigation for indoor autonomous drone racing using deep learning.” IEEE Robotics and Automation Letters, 3(3), 2539–2544.
Khan, M. T. R., Muhammad Saad, M., Ru, Y., Seo, J., and Kim, D. (2021). “Aspects of unmanned aerial vehicles path planning: Overview and applications.” International Journal of Communication Systems, 34(10), e4827.
Liu, Y., Wang, Q., Hu, H., and He, Y. (2018). “A novel real-time moving target tracking and path planning system for a quadrotor UAV in unknown unstructured outdoor scenes.” IEEE Transactions on Systems, Man, and Cybernetics: Systems, 49(11), 2362–2372.
Minoda, K., Schilling, F., Wüest, V., Floreano, D., and Yairi, T. (2021). “Viode: A simulated dataset to address the challenges of visual-inertial odometry in dynamic environments.” IEEE Robotics and Automation Letters, 6(2), 1343–1350.
Mishra, B., Garg, D., Narang, P., and Mishra, V. (2020). “Drone-surveillance for search and rescue in natural disaster.” Computer Communications, 156, 1–10.
Mousavian, A., Anguelov, D., Flynn, J., and Kosecka, J. “3d bounding box estimation using deep learning and geometry.” Proc., Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 7074–7082.
Ragi, S., and Chong, E. K. (2013). “UAV path planning in a dynamic environment via partially observable Markov decision process.” IEEE Transactions on Aerospace and Electronic Systems, 49(4), 2397–2412.
Rejeb, A., Rejeb, K., Simske, S. J., and Treiblmaier, H. (2021). “Drones for supply chain management and logistics: a review and research agenda.” International Journal of Logistics Research and Applications, 1–24.
Sani, M. F., and Karimian, G. “Automatic navigation and landing of an indoor AR. drone quadrotor using ArUco marker and inertial sensors.” Proc., 2017 international conference on computer and drone applications (IConDA), IEEE, 102–107.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). “Proximal policy optimization algorithms.”.
Shavarani, S. M., Nejad, M. G., Rismanchian, F., and Izbirak, G. (2018). “Application of hierarchical facility location problem for optimization of a drone delivery system: a case study of Amazon prime air in the city of San Francisco.” The International Journal of Advanced Manufacturing Technology, 95, 3141–3153.
Wen, N., Su, X., Ma, P., Zhao, L., and Zhang, Y. (2017). “Online UAV path planning in uncertain and hostile environments.” International journal of machine learning and cybernetics, 8, 469–487.
Ye, Y., Shi, Y., Srinivasan, D., and Du, J. (2022). “Sensation transfer for immersive exoskeleton motor training: Implications of haptics and viewpoints.” Automation in Construction, 141, 104411.
Ye, Y., Zhou, T., and Du, J. (2023). “Robot-Assisted Immersive Kinematic Experience Transfer for Welding Training.” Journal of Computing in Civil Engineering, 37(2), 04023002.
Information & Authors
Information
Published In
Construction Research Congress 2024
Pages: 707 - 716
History
Published online: Mar 18, 2024
ASCE Technical Topics:
- Automation and robotics
- Buildings
- Engineering fundamentals
- Equipment and machinery
- Freight transportation
- Geomatics
- Infrastructure
- Logistics
- Navigation (geomatic)
- Structural engineering
- Structures (by type)
- Systems engineering
- Transportation engineering
- Unmanned vehicles
- Urban and regional development
- Urban areas
- Wind engineering
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.