A Comprehensive Analysis of Reward Function for Adaptive Traffic Signal Control

Abu Rafe Md Jamil; Naushin Nower

doi:10.17977/um018v4i22021p85-96

A Comprehensive Analysis of Reward Function for Adaptive Traffic Signal Control

Abu Rafe Md Jamil, Naushin Nower

Abstract

Adaptive traffic control systems (ATCS) can play an important role to reduce traffic congestion in urban areas. The main challenge for ATSC is to determine the proper signal timing. Recently, Deep Reinforcement learning (DRL) is used to determine proper signal timing. However, the success of the DRL algorithm depends on the appropriate reward function design. There exist various reward functions for ATSC in the existing research. In this research, a comprehensive analysis of the widely used reward function is presented. The pros and cons of various reward algorithms are discussed and experimental analysis shows that multi-objective reward function enhances the performance of ATSC.

Full Text:

PDF

References

Scorecard, U.M.: ‘The texas a&m transportation institute and inrix’, Inc, USA, 2015, 9, (2015), pp. 10.

T. Economist, “The cost of traffic jams.” https://www.economist.com/ the-economist explains/2014/11/03/the-cost-of-traffic-jams, 2014. Accessed: 2020-02-17.

M. Alam, J. Ferreira, J. and J. Fonseca, “Introduction to intelligent transportation systems”, In Intelligent transportation systems pp. 1-17 2016, Springer, Cham.

J. Gao, Y. Shen, J. Liu, M. Ito, and N. Shiratori, “Adaptive traffic signal control: Deep reinforcement learning algorithm with experience replay and target network,” arXiv preprint arXiv:1705.02755, 2017.

L. Li, Y. Lv, and F.Y Wang, “Traffic signal timing via deep reinforcement learning”. IEEE/CAA Journal of Automatica Sinica, 3(3), pp.247-254 2016.

H. Wei, Hua, et al., "Intellilight: A reinforcement learning approach for intelligent traffic light control." Proceedings of the 24th ACM SIGINTELLILIGHT International Conference on Knowledge Discovery & Data Mining, 2018.

W. Genders, and R. Saiedeh, "Using a deep reinforcement learning agent for traffic signal control." arXiv preprint arXiv:1611.01142, 2016.

A.R.M. Jamil, K.K. Ganguly, and N. Nower, “Adaptive traffic signal control system using composite reward architecture based deep reinforcement learning”, IET Intelligent Transport Systems, 14(14), pp.2030-2041 2021.

S. Lange and M. Riedmiller, “Deep auto-encoder neural networks in reinforcement learning,” in The 2010 International Joint Conference on Neural Networks (IJCNN), pp. 1–8, IEEE, 2010.

X. Liang, X. Du, G. Wang, and Z. Han, “Deep reinforcement learning for traffic light control in vehicular networks,” arXiv preprint arXiv:1803.11115, 2018.

C. Chen et al. "Toward a thousand lights: Decentralized deep reinforcement learning for large-scale traffic signal control", Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 34, No. 04, 2020.

A.R.M. Jamil, K.K Ganguly, N. Nower, “An experimental analysis of reward functions for adaptive traffic signal control system”, Proceedings of the International conference on distributed sensing and intelligent system (ICDSIS), Springer, 2020.

P. Mannion, J. Duggan, E. Howley, “An experimental review of reinforcement learning algorithms for adaptive traffic signal control”, Proceedings of the Autonomic road transport support systems. Springer, pp. 47–66, 2016.

H. Van Seijen et al., "Hybrid reward architecture for reinforcement learning "Proceedings of the Advances in Neural Information Processing Systems” pp. 5392–5402, 2017.

S. B. Kotsiantis, I. Zaharakis, and P. Pintelas, “Supervised machine learning: A review of classification techniques,” Emerging artificial intelligence applications in computer engineering, vol. 160, pp. 3–24, 2007.

H. B. Barlow, “Unsupervised learning,” Neural computation, vol. 1, no. 3, pp. 295–311, 1989.

S. Lange and M. Riedmiller, “Deep auto-encoder neural networks in reinforcement learning,” in The 2010 International Joint Conference on Neural Networks (IJCNN), pp. 1–8, IEEE, 2010.

T. Schaul, J. Quan, I. Antonoglou, and D. Silver, “Prioritized experience replay,” arXiv preprint arXiv:1511.05952, 2015.

E. Van der Pol and F. A. Oliehoek, “Coordinated deep reinforcement learners for traffic light control,” Proceedings of Learning, Inference and Control of Multi-Agent Systems (NIPS), 2016.

J. van Dijk, “Recurrent neural networks for reinforcement learning: an investigation of relevant design choices,” 2017.

M. Coskun, A. Baggag, and S. Chawla, “Deep reinforcement learning for traffic light optimization,” in 2018 IEEE International Conference on Data Mining Workshops (ICDMW), pp. 564–571, IEEE, 2018.

M. A. Khamis, W. Gomaa, and H. El-Shishiny, “Multi-objective traffic light control system based on bayesian probability interpretation,” in 2012 15th International IEEE Conference on Intelligent Transportation Systems, pp. 995– 1000, IEEE, 2012.

M. A. Khamis and W. Gomaa, “Adaptive multi-objective reinforcement learning with hybrid exploration for traffic signal control based on cooperative multi-agent framework,” Engineering Applications of Artificial Intelligence, vol. 29, pp. 134–151, 2014.

A. Vidali, L. Crociani, G. Vizzari, and S. Bandini, “A deep reinforcement learning approach to adaptive traffic lights management,” in Proceedings of the 20th Workshop” From Objects to Agents”, Parma, Italy, 2019.

X. Zang et al. "Metalight: Value-based meta-reinforcement learning for traffic signal control", Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34. No. 01. 2020.

DOI: http://dx.doi.org/10.17977/um018v4i22021p85-96