مدیریت مصرف انرژی خانگی با استفاده از یادگیری تقویتی چندعاملی

افزایش مصرف انرژی الکتریکی، مسئله‌ای است که همواره به‌عنوان یکی از چالش‌های تامین‌کنندگان برق مطرح بوده است. به‌دنبال افزایش مصرف، برنامه‌های پاسخ‌گویی بار که سعی در مدیریت مصرف انرژی با اهدافی نظیرکاهش هزینه‌ها و افزایش قابلیت اطمینان دارند، بیش از پیش مورد توجه قرار گرفته‌اند. از طرفی هوشمندسازی مصرف‌کنندگان، امکان بهره‌گیری هرچه بیشتر از هوش مصنوعی برای مدیریت انرژی را میسر ساخته است. این مقاله روشی برای مدیریت مصرف انرژی خانگی با هدف کمینه کردن قبض برق و نارضایتی مشترک ارائه می‌دهد. با تفکیک بارهای خانه به سه دسته بار‌های غیرقابل کنترل، قابل جابه‌جایی و قابل کنترل، یادگیری تقویتی چندعاملی با الگوریتم qlearning راهکاری است که در این مقاله برای اتخاذ تصمیمات بهینه دربارۀ هریک از وسایل خانه در نظر گرفته شده است. به‌دلیل ماهیت الگوریتم qlearning، روش پیشنهادی در این مقاله برخلاف روش‌های برنامه‌ریزی عدد صحیح امکان افزودن وسایل بیشتری از خانه و حل مسئله‌های پیچیده‌تری را داراست. پیاده‌سازی روش پیشنهادی این مقاله در بخش مطالعۀ عددی منجر به کاهش قبض برق مشترک تا 24.8% گردید. همچنین، نتایج حاصل از اعمال روش ارائه‌شده حاکی از صحت عملکرد آن است.

Home Energy Management Using Multi-Agent Reinforcement Learning

Extended AbstractIntroduction: In developed societies, residential customers use highlevel appliances. The progress in smart grids and the internet of things have eased the way for home energy management to schedule controllable appliances. Revently, looking to demand increment and demand response strategies aiming at energy management for demand reduction and the improvement of reliability have attreacted attention. A deep review of the existing literature shows that notable efforts have been put into optimizing the problem of home energy management through classic and metaheuristic optimization algorithms such as game theory, genetic algorithm, and PSO. However, it is worth saying that these algorithms are not pragmatic due to the inherent nature of the home energy management problem. To be more precise, these algorithms, as the environment of the problem changes continuously, fail to solve the problem. Hence, some essential assumptions such as considering fixed scenarios are presumed in the previous works to enable the conventional algorithm to solve the problem. This is while machine learning addresses this issue by extracting the main features from input data and constructing a general description of the environment. Implementation of machine learningbased algorithms to a home energy management problem requires smart appliances. Hence, in the case of having a smart home, taking the advantage of artificial intelligence for energy management would be feasible and useful. It should be noted that electricity cost reduction can make the demand response program inviting, where customers rsquo; satisfaction is taken into consideration. Accordingly, customers rsquo; satisfaction should be considered in the formulation of the problem. Regarding the mentioned issues, lately, with a remarkable progress in machine learning, novel algorithms have evolved for solving optimal decisionmaking problems such as demand response. Machine learning can be categorized into three main categories, namely, supervised learning, unsupervised learning, and reinforcement learning (RL). Among these, reinforcement learning has shown notable performance in decisionmaking problems. QLearning is a modelfree RL algorithm that solves nonlinear problems through estimating and maximizing the cumulative reward, triggered by decided actions. The fundamental idea of this algorithm is to identify the best action in each situation. This paper aims to provide a dayahead demand response program for a smart home. It is done by specifying the quantity of the energy consumption of each appliance, aiming to reduce the electricity cost and user dissatisfaction. In this respect, it is presumed that the smart home is equipped with smart appliances. Moreover, smart meters are installed on appliances to monitor the stati and receive command signals from devices at each hour. These appliances can be divided into three categories: nonresponsive, timeshiftable, and controllable loads. Dishwashers and washing machines, as timeshiftable loads; EV, air conditioners, and lighting systems, as controllable loads; and TVs and refrigerators, as nonresponsive loads, are taken into account. All in all, we recommend an advanced home energy management system proposing the following contributions: i) proposing a dayahead multiagent QLearning method to minimize electricity cost; ii) proposing a satisfactionbased framework, which employs a precise model of the customer dissatisfaction functions (i.e., thermal comfort, battery degradation, and desirable operation period). Compared to integer programming approaches, the proposed method in this paper is capable of modeling more appliances and of solving complex problems due to the innate nature of the QLearning algorithm. Implementing the proposed method in the numerical study section led to a 24.8% electricity bill reduction. The numerical results proved the effectiveness of the proposed approach.Materials and Methods: In this paper, a multiagent QLearning approach is used to solve home energy management for a smart home. Qlearning is a popular modelfree algorithm among reinforcement learning algorithms not only because of the fact that its convergence is proven but also because of its feasiblity for implementatio. In order to deploy QLearning on a home energy management system, first of all a smart home should be formed as a Markov decision process. A Markov decision process consists of four fundamental parameters, namely, state, action, reward, and transition probability matrix. Afterward, an agent is trained through experiencing a specific state, taking an action, transiting to a new state, and calculating the cumulative reward. By doing so, it will learn, after visiting a considerable number of states and taking diverse decisions, gradually to select the optimum action whatever the state is.Another fundamental aspect of this paper is that the proposed approach takes customers rsquo; satisfaction into account. In this paper, a nonlinear thermal comfort model, a nonlinear desirable operation period model, and a linear battery degradation model are deployed to consider the customer rsquo;s dissatisfaction precisely. It should be noted that all simulations have been implemented by python 3.6 programming language without making use of any commercial solver.Results: Various case studies have been designed to verify the effectiveness of the proposed method. Scenario 1 is designed to simulate the behavior of a smart home associated with a random manner of energy usage. Scenario 2 is designed to verify the effectiveness of the proposed home energy management system, where QLearning is conducted. In this case, battery degradation is overlooked. Scenario 3 is similar to the previous one, where battery degradation is also taken into consideration. Comparing the obtained results indicates that the proposed algorithm has successfully reduced the electricity bill by 31.3% and 24.8% in scenarios 2 and 3 respectively. It is worth saying that customers rsquo; satisfaction is not violated in mentioned scenarios. Furthermore, in order to evaluate the effect of thermal comfort on the electricity bill, another case study is deployed, where the thermal comfort coefficient is decreased to smaller magnitudes. As expected, the less thermal comfort coefficient becomes, the less the electricity bill will be. The reason behind this lies in the fact that having a lower thermal comfort coefficient leads to less importance of temperature control compared to the electricity bill.Conclusion: This paper proposed a method for home energy management, regarding minimizing the electricity bill and users rsquo; discomfort. In this paper, a multiagent reinforcement learning via QLearning is used to make optimal decisions for home appliances, which are categorized into nonshiftable loads, timeshiftable loads, and controllable loads. Comparing to classic optimization methods, the proposed approach in this paper is capable of modeling more appliances and solving complex problems due to the inherent nature of the QLearning algorithm. Implementing the proposed method in the numerical study section led to a 24.8% electricity bill reduction. The numerical results proved the effectiveness of the proposed approach.