Exploring LSTM-based Attention Mechanisms with PSO and Grid Search under Different Normalization Techniques for Energy demands Time Series Forecasting

Andri Pranolo; Xiaofeng Zhou; Yingchi Mao; Bambang Widi Pratolo; Aji Prasetya Wibawa; Agung Bella Putra Utama; Abdoul Fatakhou Ba; Abdullahi Uwaisu Muhammad

doi:10.17977/um018v7i12024p1-12

Exploring LSTM-based Attention Mechanisms with PSO and Grid Search under Different Normalization Techniques for Energy demands Time Series Forecasting

Andri Pranolo, Xiaofeng Zhou, Yingchi Mao, Bambang Widi Pratolo, Aji Prasetya Wibawa, Agung Bella Putra Utama, Abdoul Fatakhou Ba, Abdullahi Uwaisu Muhammad

Abstract

Advanced analytical approaches are required to accurately forecast the energy sector's rising complexity and volume of time series data. This research aims to forecast the energy demand utilizing sophisticated Long Short-Term Memory (LSTM) configurations with Attention mechanisms (Att), Grid search, and Particle Swarm Optimization (PSO). In addition, the study also examines the influence of Min-Max and Z-Score normalization approaches in the preprocessing stage on the accuracy performances of the baselines and the proposed models. PSO and Grid Search techniques are used to select the best hyperparameters for LSTM models, while the attention mechanism selects the important input for the LSTM. The research compares the performance of baselines (LSTM, Grid-search-LSTM, and PSO-LSTM) and proposes models (Att-LSTM, Att-Grid-search-LSTM, and Att-PSO-LSTM) based on MAPE, RMSE, and R2 metrics into two scenarios normalization: Min-Max, and Z-Score. The results show that all models with Min-Max normalization have better MAPE, RMSE, and R2 than those with Z-Score. The best model performance is shown in Att-PSO-LSTM MAPE 3.1135, RMSE 0.0551, and R2 0.9233, followed by Att-Grid-search-LSTM, Att-LSTM, PSO-LSTM, Grid-search-LSTM, and LSTM. These findings emphasize the effectiveness of attention mechanisms in improving model predictions and the influence of normalization methods on model performance. This study's novel approach provides valuable insights into time series forecasting in energy demands.

Full Text:

PDF

References

T. González Grandón, J. Schwenzer, T. Steens, and J. Breuing, “Electricity demand forecasting with hybrid classical statistical and machine learning algorithms: Case study of Ukraine,” Appl. Energy, vol. 355, p. 122249, 2024.

E. Yukseltan, A. Yucekaya, and A. H. Bilge, “Hourly electricity demand forecasting using Fourier analysis with feedback,” Energy Strateg. Rev., vol. 31, p. 100524, 2020.

M. Aldarraji, B. Vega-Márquez, B. Pontes, B. Mahmood, and J. C. Riquelme, “Addressing energy challenges in Iraq: Forecasting power supply and demand using artificial intelligence models,” Heliyon, vol. 10, no. 4, p. e25821, 2024.

S. O. Effiom, P. C. O. Effiom, R. Akwagiobe, and P. O. Odu, “Technical and economic appraisal for harnessing a proposed hybrid energy system nexus for power generation and CO2 mitigation in Cross River State, Nigeria,” Appl. Eng. Technol., vol. 2, no. 2, pp. 153–175, 2023.

H. Jin, J. Guo, L. Tang, and P. Du, “Long-term electricity demand forecasting under low-carbon energy transition: Based on the bidirectional feedback between power demand and generation mix,” Energy, vol. 286, p. 129435, 2024.

S. Eggimann, W. Usher, N. Eyre, and J. W. Hall, “How weather affects energy demand variability in the transition towards sustainable heating,” Energy, vol. 195, p. 116947, 2020.

A. Dietrich and C. Melville, “Energy demand characteristics and the potential for energy efficiency in sports statium and arenas,” Duke Univ., 2011.

A. Nikseresht and H. Amindavar, “Energy demand forecasting using adaptive ARFIMA based on a novel dynamic structural break detection framework,” Appl. Energy, vol. 353, p. 122069, Jan. 2024.

S. Chaturvedi, E. Rajasekar, S. Natarajan, and N. McCullen, “A comparative assessment of SARIMA, LSTM RNN and Fb Prophet models to forecast total and peak monthly energy demand for India,” Energy Policy, vol. 168, p. 113097, 2022.

P. Liu, F. Quan, Y. Gao, B. Alotaibi, T. R. Alsenani, and M. Abuhussain, “Green energy forecasting using multiheaded convolutional LSTM model for sustainable life,” Sustain. Energy Technol. Assessments, vol. 63, p. 103609, 2024.

Q. Wang, R. Suo, and Q. Han, “A study on natural gas consumption forecasting in China using the LMDI-PSO-LSTM model: Factor decomposition and scenario analysis,” Energy, vol. 292, p. 130435, 2024.

M. Bilgili and E. Pinar, “Gross electricity consumption forecasting using LSTM and SARIMA approaches: A case study of Türkiye,” Energy, vol. 284, p. 128575, 2023.

N. Mounir, H. Ouadi, and I. Jrhilifa, “Short-term electric load forecasting using an EMD-BI-LSTM approach for smart grid energy management system,” Energy Build., vol. 288, p. 113022, 2023.

L. Wang et al., “Optimal allocation of customer energy storage based on power big data and improved LSTM load forecasting,” Energy Reports, vol. 11, pp. 3902–3913, 2024.

J. Liu and S. Chen, “Non-stationary Multivariate Time Series Prediction with Selective Recurrent Neural Networks,” in Lecture Notes in Computer Science, 2019, pp. 636–649.

A. W. Saputra, A. P. Wibawa, U. Pujianto, A. B. P. Utama, and A. Nafalski, “LSTM-based Multivariate Time-Series Analysis : A Case of Journal Visitors Forecasting,” Ilk. J. Ilm., vol. 14, no. 1, pp. 57–62, 2022.

B. Li, X. Lv, and J. Chen, “Demand and supply gap analysis of Chinese new energy vehicle charging infrastructure: Based on CNN-LSTM prediction model,” Renew. Energy, vol. 220, p. 119618, 2024.

H. Chen, M. Zhu, X. Hu, J. Wang, Y. Sun, and J. Yang, “Research on short-term load forecasting of new-type power system based on GCN-LSTM considering multiple influencing factors,” Energy Reports, vol. 9, pp. 1022–1031, 2023.

Z. Niu, G. Zhong, and H. Yu, “A review on the attention mechanism of deep learning,” Neurocomputing, vol. 452, pp. 48–62, Sep. 2021.

H. Abbasimehr and R. Paki, “Improving time series forecasting using LSTM and attention models,” J. Ambient Intell. Humaniz. Comput., Jan. 2021.

A. Pranolo, Y. Mao, A. P. Wibawa, A. B. P. Utama, and F. A. Dwiyanto, “Robust LSTM With Tuned-PSO and Bifold-Attention Mechanism for Analyzing Multivariate Time-Series,” IEEE Access, vol. 10, pp. 78423–78434, 2022.

A. P. Wibawa et al., “Bidirectional Long Short-Term Memory (Bi-LSTM) Hourly Energy Forecasting,” in E3S Web of Conferences, 2024, vol. 501, p. 1023.

S. Huber, H. Wiemer, D. Schneider, and S. Ihlenfeldt, “DMME: Data mining methodology for engineering applications – a holistic extension to the CRISP-DM model,” Procedia CIRP, vol. 79, pp. 403–408, 2019.

A. Mirzaei, S. R. Carter, A. E. Patanwala, and C. R. Schneider, “Missing data in surveys: Key concepts, approaches, and applications,” Res. Soc. Adm. Pharm., vol. 18, no. 2, pp. 2308–2316, Feb. 2022.

A. Abayomi-Alli, M. O. Odusami, O. Abayomi-Alli, S. Misra, and G. F. Ibeh, “Long Short-Term Memory Model for Time Series Prediction and Forecast of Solar Radiation and other Weather Parameters,” in 2019 19th International Conference on Computational Science and Its Applications (ICCSA), Jul. 2019, pp. 82–92.

N. Passalis, J. Kanniainen, M. Gabbouj, A. Iosifidis, and A. Tefas, “Forecasting Financial Time Series Using Robust Deep Adaptive Input Normalization,” J. Signal Process. Syst., vol. 93, no. 10, pp. 1235–1251, Oct. 2021.

M. Ehteram, M. Afshari Nia, F. Panahi, and A. Farrokhi, “Read-First LSTM model: A new variant of long short term memory neural network for predicting solar radiation data,” Energy Convers. Manag., vol. 305, p. 118267, Apr. 2024.

B. H. Shekar and G. Dagnew, “Grid Search-Based Hyperparameter Tuning and Classification of Microarray Cancer Data,” in 2019 Second International Conference on Advanced Computational and Communication Paradigms (ICACCP), Feb. 2019, pp. 1–8.

Q. Kang, E. J. Chen, Z.-C. Li, H.-B. Luo, and Y. Liu, “Attention-based LSTM predictive model for the attitude and position of shield machine in tunneling,” Undergr. Sp., vol. 13, pp. 335–350, 2023.

E. Vivas, H. Allende-Cid, and R. Salas, “A Systematic Review of Statistical and Machine Learning Methods for Electrical Power Forecasting with Reported MAPE Score,” Entropy, vol. 22, no. 12, p. 1412, Dec. 2020.

F. Baig, L. Ali, M. A. Faiz, H. Chen, and M. Sherif, “How accurate are the machine learning models in improving monthly rainfall prediction in hyper arid environment?,” J. Hydrol., vol. 633, p. 131040, Apr. 2024.

W. Sun and C. Huang, “A novel carbon price prediction model combines the secondary decomposition algorithm and the long short-term memory network,” Energy, vol. 207, p. 118294, Sep. 2020.

DOI: http://dx.doi.org/10.17977/um018v7i12024p1-12