Deep Recurrent Q-Learning Method for Single Intersection Signal Control

In recent years, reinforcement learning is applied into traffic control as an emerging technique, which obtain more and more research attention. In this paper, a deep recurrent Q-learning agent was implemented in the context of traffic signal control in order to improve the efficiency of highway transportation while maintaining a significant degree of reality. The reinforcement learning agent was designed with a state representation that identifies the position of vehicles in the environment, an action set defined by traffic light configurations with a fixed duration, and a reward function that capture in different magnitudes the difference of vehicles waiting times between actions. In particular, the elements of the agent are designed to make sense for possible real-world devices. The learning approach applied for the agent’s training is the deep Q-Network combined with a recurrent neural network. The Q-learning is used for the update of the action values as the experience of the agent increases and the neural network is employed for the Q-values prediction and, therefore, the approximation of the state-action function. SUMO was used to replicate a 4-way intersection with multiple lanes, and to reproduce various traffic scenarios with different traffic distributions. The reward was calculated based on the simulated waiting time of vehicles, making the agent aware of the consequence of actions in different situations. Results indicate that the proposed agent can adapt to several traffic situations and is able to outperform the static traffic light system in situations of low, medium, and high densities, increasing the overall efficiency of more than 50%.