Multiagent Stochastic Dynamic Game for Smart Generation Control
Publication: Journal of Energy Engineering
Volume 142, Issue 1
Abstract
This paper proposes a multiagent (MA) smart generation control (SGC) scheme for the coordination of automatic generation control (AGC) in power grids with system uncertainties. Under the control performance standards, SGC will undergo a non-Markov random process, of which the optimal solution can be resolved online by the reinforcement learning. Therefore, an MA decentralized correlated equilibrium Q()-learning algorithm, and an MA stochastic dynamic game-based SGC simulation platform (SGC-SP) have been proposed for its implementation, which can achieve AGC coordination in a highly uncertain environment resulting from the increasing penetration of renewable energy. Single-agent Q-learning, Q()-learning, R()-learning, and proportional integral control are implemented and embedded in SGC-SP for the control performance analysis. Two case studies on both a two-area power system and the China Southern Power Grid model have been done, which verify its effectiveness and scalability.
Get full access to this article
View all available purchase options and get full access to this article.
Acknowledgments
This work was partially supported by The National Basic Research Program (973 Program) (Grant No. 2013CB228205), Guangdong Key Laboratory of Clean Energy Technology (2008A060301002), The National Natural Science Foundation of China (Grant No. 51177051), and the Theme-based Research Scheme of the Research Grants Council of the Hong Kong Special Administrative Region, China (Grant No. T23-407/13-N).
References
Bellifemine, F., Bergenti, F., Caire, G., and Poggi, A. (2005). “Jade—A Java agent development framework.” Multi-agent programming: Languages, platforms and applications, R. H. Bordini, M. Dastani, J. Dix, and A. El Fallah-Seghrouchni, eds., Vol. 15, Springer, London, 125–147.
Bellifemine, F., Poggi, A., and Rimassa, G. (2001). “Developing multi-agent systems with JADE.” Proc., 7th Int. Workshop on Intelligent Agents VII. Agent Theories Architectures and Languages, Springer, London, 89–103.
Bevrani, H., Daneshfar, F., and Hiyama, T. (2012). “A new intelligent agent-based AGC design with real-time application.” IEEE Trans. Syst. Man Cybern. C Appl. Rev., 42(6), 994–1002.
Daneshfar, F., and Bevrani, H. (2010). “Load-frequency control: A GA-based multi-agent reinforcement learning.” IET Gener. Transm. Distrib., 4(1), 13–26.
de Luján Latorre, M., and Granville, S. (2003). “The stackelberg equilibrium applied to AC power systems—A non-interior point algorithm.” IEEE Trans. Power Syst., 18(2), 611–617.
Dimeas, A. L., and Hatziargyriou, N. D. (2010). “Multi-agent reinforcement learning for microgrids.” IEEE Conf. on Power and Energy Society General Meeting, IEEE, Minneapolis, 1–8.
Elgerd, O. I. (1983). Electric energy system theory—An introduction, McGraw-Hill, New York.
Elmitwally, A., Elsaid, M., and Elgamal, M. (2014). “Novel multiagent-based scheme for voltage control coordination in smart grids.” J. Energy Eng., 04014025.
Ernst, D., Glavic, M., and Wehenkel, L. (2004). “Power systems stability control: Reinforcement learning framework.” IEEE Trans. Power Syst., 19(1), 427–435.
Etingov, P. V., et al. (2010). “Possible improvements of the ACE diversity interchange methodology.” Proc., 2010 IEEE Power and Energy Society General Meeting, Minneapolis, 1–8.
Gao, Z. H., Teng, X. L., and Tu, L. Q. (2004). “Hierarchical AGC mode and CPS control strategy for interconnected power systems.” Autom. Electr. Power Syst., 28(1), 78–81 (in Chinese).
Greenwald, A., and Hall, K. (2003). “Correlated-Q learning.” Proc., 20th Int. Conf. on Machine Learning (ICML-03), Morgan Kaufmann, San Francisco, 242–249.
Hu, J., and Wellman, M. P. (1998). “Multiagent reinforcement learning: Theoretical framework and an algorithm.” Proc., 15th Int. Conf. on Machine Learning, Morgan Kaufmann, San Francisco, 242–250.
Hu, J., and Wellman, M. P. (2003). “Nash Q-learning for general-sum stochastic games.” J. Mach. Learn. Res., 4, 1039–1069.
Jaleeli, N., and VanSlyck, L. S. (1999). “NERC’s new control performance standards.” IEEE Trans. Power Syst., 14(3), 1092–1099.
Jing, P., and Williams, R. J. (1996). “Incremental multi-step Q-learning.” Mach. Learn., 22(1–3), 283–290.
Keiding, H., and Peleg, B. (2000). “Correlated equilibrium of games with many players.” Int. J. Game Theory, 29(3), 375–389.
Keyhani, A., and Chatterjee, A. (2012). “Automatic generation control structure for smart power grids.” IEEE Trans. Smart Grid, 3(3), 1310–1316.
Könönen, V. (2006). “Dynamic pricing based on asymmetric multiagent reinforcement learning.” Int. J. Intell. Syst., 21(1), 73–98.
Littman, M. L. (1994). “Markov games as a framework for multi-agent reinforcement learning.” Proc., 11th Int. Conf. on Machine Learning, Morgan Kaufmann, San Francisco, 157–163.
Littman, M. L. (1996). “A generalized reinforcement-learning model: Convergence and applications.” Proc., 13th Int. Conf. on Machine Learning, Morgan Kaufmann, San Francisco, 310–318.
Littman, M. L. (2001). “Friend or foe Q-learning in general-sum Markov games.” Proc., 18th Int. Conf. on Machine Learning, Morgan Kaufmann, San Francisco, 322–328.
Marco, G. D., and Morgan, J. (2008). “Slightly altruistic equilibria.” J. Optim. Theory Appl., 137(2), 347–362.
MATLAB [Computer software]. Natick, MA, MathWorks.
Moslehi, K., and Kumar, R. (2010). “Smart grid—A reliability perspective.” IEEE Conf. on Innovative Smart Grid Technologies, IEEE, Gaithersburg, MD, 1–8.
Nanda, J., Mishra, S., and Saikia, L. C. (2009). “Maiden application of bacterial foraging-based optimization technique in multiarea automatic generation control.” IEEE Trans. Power Syst., 24(2), 602–609.
Nash, J. (1951). “Non-cooperative games.” Ann. Math., 54(2), 286–295.
NERC (North American Electric Reliability Council). (2005). “Standard BAL-001-0—Real power balancing control performance.” 〈http://stan-dard.nerc.net/〉 (Apr. 1, 2005).
Oneal, A. R. (1995). “A simple method for improving control area performance: Area control error (ACE) diversity interchange.” IEEE Trans. Power Syst., 10(2), 1071–1076.
Sutton, R. S. (1988). “Learning to predict by the methods of temporal differences.” Mach. Learn., 3(1), 9–44.
Sutton, R. S., and Barto, A. G. (1998). Reinforcement learning: An introduction, MIT Press, Cambridge, MA.
Watkins, C. J. H., and Dayan, P. (1992). “Q-learning.” Mach. Learn., 8(3–4), 279–292.
Yao, M., Shoults, R. R., and Kelm, R. (2000). “AGC logic based on NERC’s new control performance standard and disturbance control standard.” IEEE Trans. Power Syst., 15(2), 852–857.
Yu, T., Zhou, B., Chan, K. W., Chen, L., and Yang, B. (2011). “Stochastic optimal relaxed automatic generation control in non-Markov environment based on multi-step Q() learning.” IEEE Trans. Power Syst., 26(3), 1272–1282.
Yu, T., Zhou, B., Chan, K. W., Yuan, Y., and Yang, B. (2012). “R() imitation learning for automatic generation control of interconnected power grids.” Automatica, 48(9), 2130–2136.
Zhang, P., Li, F. X., and Bhatt, N. (2010). “Next-generation monitoring, analysis, and control for the future smart control center.” IEEE Trans. Smart Grid, 1(2), 186–192.
Information & Authors
Information
Published In
Journal of Energy Engineering
Volume 142 • Issue 1 • March 2016
Copyright
© 2015 American Society of Civil Engineers.
History
Received: Sep 25, 2014
Accepted: Feb 6, 2015
Published online: Apr 22, 2015
Discussion open until: Sep 22, 2015
Published in print: Mar 1, 2016
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.