This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
Classification of Livin' by Mandiri Customer Satisfaction Using MLP with BM25 and TF-IDF Feature Weighting
Corresponding Author(s) : Dewi Fatmarani Surianto
Kinetik: Game Technology, Information System, Computer Network, Computing, Electronics, and Control,
Vol. 10, No. 3, August 2025
Abstract
The increasing use of mobile banking applications such as Livin' by Mandiri requires analyzing customer satisfaction based on user reviews. This study classifies customer satisfaction level using Multi-Layer Perceptron (MLP) algorithm with two feature extraction methods, namely BM25 and TF-IDF. Data totaling 1,143 reviews were collected from Google Play Store and App Store. Three test scenarios were applied: (1) comparison of feature extraction methods, (2) application of Synthetic Minority Over-Sampling Technique (SMOTE), and (3) application of Synonym Replacement-based Easy Data Augmentation (EDA) technique. The evaluation results show that the combination of BM25 and data augmentation produces the highest performance with 97% accuracy and 98% precision, recall, and F1-score respectively. BM25 proved to be more effective in understanding the context of reviews, while data augmentation improved the quality of representation, especially on minority classes such as neutral sentiment. These findings make a real contribution to the improvement of Livin' by Mandiri digital services and serve as a reference for the development of review-based satisfaction classification systems in the digital banking sector.
Keywords
Download Citation
Endnote/Zotero/Mendeley (RIS)BibTeX
- S. I. Murpratiwi, S. E. Anjarwani, I. G. P. S. Wijaya, and A. Aranta, "Socialization of Healthy Internet and Training on the Use of Internet as Learning Support Media at SD Negeri Anggaraksa," J. Begawe Teknol. Inf., vol. 4, no. 2, pp. 256-263, 2023, doi: 10.29303/jbegati.v4i2.1111.
- D. D. Audiansyah, D. E. Ratnawati, and B. T. Hanggara, "Sentiment Analysis of MyXL App using Support Vector Machine Method based on User Reviews in Google Play Store," vol. 6, no. 8, pp. 3987-3994, 2022.
- A. Ahmad, W. Gata, and S. Panggabean, "Sentiment Analysis with Long Short-Term Memory and Synthetic Minority Over Sampling Technic in Digital Banking Applications," J. Technol. Inf. and Commun., vol. 8, no. 4, pp. 973-984, 2024.
- T. Wahudi and Z. Hutabarat, "Factors Affecting the Intention to Use Digital Banking: Livin' By Mandiri," J. Tech. Inform. and Sist. Inf., vol. 10, no. 1, pp. 509-525, 2023.
- M. Management, F. Economics, and U. P. Harapan, "Factors Affecting Intention to Use Digital Banking: Livin' By Mandiri," vol. 10, no. 1, pp. 509-525, 2023.
- S. H. Alviyanti, A. Purwandira, I. Febiyanti, E. Daniati, and A. Ristyawan, "Sentiment Classification of Livin By Mandiri Application Users on Playstore Using Naive Bayes Algorithm," August, vol. 8, pp. 2549-7952, 2024.
- S. L. Ranataru and N. Trianasari, "Social Media Sentiment Analysis of Banking Applications to Determine Application User Satisfaction: Case Study on Livin by Mandiri and BCA Mobile," Al-Kharaj J. Ekon. , Finance. Sharia Business, vol. 6, pp. 6818-6838, 2024, doi: 10.47467/alkharaj.v6i9.3805.
- I. D. Onantya and P. P. Adikara, "Sentiment Analysis on BCA Mobile Application Reviews Using BM25 and Improved K-Nearest Neighbor," J-Ptiik.Ub.Ac.Id, vol. 3, no. 3, pp. 2575-2580, 2019.
- M. Z. Hariansyah and Siswanto, "Implementation of Multinomial Naive Bayes Method on Sentiment Analysis of Livin by Mandiri Application Services," Semin. Nas. Mhs. Fak. Technol. Inf., vol. 1, no. 1, pp. 517-524, 2022.
- N. Nurfadila, M. Ariyanti, and N. Trianasari, "Service Quality Analysis of New Livin' By Mandiri Mobile Banking Using Sentiment Analysis," JIBR J. Indones. Bus. Res., vol. 1, no. 1, pp. 77-82, 2023.
- C. A. Qurniaty and K. Kusnawi, "Emotion Expression Based on Voice Using Multi Layer Perceptron Algortima and Support Vector Machine," Indonesia. J. Comput. Sci., vol. 12, no. 6, pp. 4014-4025, 2023, doi: 10.33022/ijcs.v12i6.3567.
- G. G. Warow and H. Pandia, "Sentiment Analysis of Dana Application Using Naïve Bayes Classifier and Support Vector Machine," Jutisi J. Ilm. Tech. Inform. and Sist. Inf., vol. 13, no. 1, p. 609, 2024, doi: 10.35889/jutisi.v13i1.1893.
- F. Riza and B. Kurniawan, "Sentiment analysis of BSI mobile application users due to ransomware using support vector machine algorithm," INFOTECH J. Inform. Technol., vol. 5, no. 1, pp. 42-51, 2024, doi: 10.37373/infotech.v5i1.1102.
- R. A. Husen, R. Astuti, L. Marlia, R. Rahmaddeni, and L. Efrizoni, "Sentiment Analysis of Public Opinion on Twitter towards BSI Bank Using Machine Learning Algorithm," MALCOM Indones. J. Mach. Learn. Comput. Sci., vol. 3, no. 2, pp. 211-218, 2023, doi: 10.57152/malcom.v3i2.901.
- F. H. Pasaribu, N. Khairina, D. Noviandri, S. Susilawati, and R. Syah, "Analysis of The Multilayer Perceptron Algorithm on Twitter User's Sentiment Towards The COVID-19 Vaccine," J. Informatics Telecommun. Eng., vol. 7, no. 1, pp. 155-163, 2023, doi: 10.31289/jite.v7i1.9664.
- I. Daniel, A. Fahmi Limas Ptr, and A. Ichsan, "Risk Classification of Heart Attack Disease with Multi-Layer Perceptron," Data Sci. Indones., vol. 14, no. 1, pp. 57-64, 2024.
- A. Purnamawati, M. N. Winarto, and M. Mailasari, "Sentiment Analysis of TikTok Application using BM25 Method and Improved K-NN Chi-Square Feature," J. Komtika (Computing and Inform., vol. 7, no. 1, pp. 97-105, 2023, doi: 10.31603/komtika.v7i1.8938.
- R. Maulana, A. Voutama, and T. Ridwan, "Sentiment Analysis of MyPertamina App Reviews on Google Play Store using NBC Algorithm," J. Teknol. Integrated, vol. 9, no. 1, pp. 42-48, 2023, doi: 10.54914/jtt.v9i1.609.
- D. Duei Putri, G. F. Nama, and W. E. Sulistiono, "Sentiment Analysis of the Performance of the House of Representatives (DPR) on Twitter Using the Naive Bayes Classifier Method," J. Inform. and Tech. Electro Applied, vol. 10, no. 1, pp. 34-40, 2022, doi: 10.23960/jitet.v10i1.2262.
- N. Agustina, D. H. Citra, W. Purnama, C. Nisa, and A. R. Kurnia, "Implementation of Naive Bayes Algorithm for Sentiment Analysis of Shopee Reviews on Google Play Store," MALCOM Indones. J. Mach. Learn. Comput. Sci., vol. 2, no. 1, pp. 47-54, 2022, doi: 10.57152/malcom.v2i1.195.
- R. A. Nurdian, Mujib Ridwan, and Ahmad Yusuf, "Comparison of SMOTE and ADASYN Methods in Improving the Performance of New Student Herregistration Classification," J. Tech. Inform. and Sist. Inf., vol. 8, no. 1, pp. 24-32, 2022, doi: 10.28932/jutisi.v8i1.4004.
- S. Sasmita, R. N. Jariah S. Intam, D. F. Surianto, and M. F. B, "Sentiment Analysis of the Constitutional Court Decision Controversy Regarding the Age of Candidates Using Multi-Layer Perceptron with SMOTE Technique," Fakt. Exacta, vol. 17, no. 2, p. 188, 2024, doi: 10.30998/faktorexacta.v17i2.22442.
- R. N. Harahap and K. Muslim, "Improving Accuracy in Twitter User Mbti Personality Prediction Using Data Augmentation," J. Technol. Inf. and Comput. Science, vol. 7, no. 4, p. 815, 2020, doi: 10.25126/jtiik.2020743622.
- A. Nur Azizah, M. Falach Asy'ari, I. Wisma Dwi Prastya, and D. Purwitasari, "Easy Data Augmentation for Imbalance Data in Online Health Consultation," J. Technol. Inf. and Comput. Science, vol. 10, no. 5, pp. 1095-1104, 2023, doi: 10.25126/jtiik.20231057082.
- I. Athiyyah Rahma and L. Hulliyyatus Suadaa, "Application of Text Augmentation to Overcome Unbalanced Data in Indonesian Text Classification Case Study: Detection of Clickbait Titles and Hate Speech Comments on Online News," J. Technol. Inf. and Comput. Science, vol. 10, no. 6, pp. 1329-1340, 2023, doi: 10.25126/jtiik.2023107325.
- A. I. Tanggraeni and M. N. N. Sitokdana, "Sentiment Analysis of E-Government Applications on Google Play Using Naïve Bayes Algorithm," JATISI (Journal of Tech. Inform. and Information Systems), vol. 9, no. 2, pp. 785-795, 2022, doi: 10.35957/jatisi.v9i2.1835.
- W. Wahyudi, R. Kurniawan, and Y. Arie Wijaya, "User Sentiment Analysis of Blu Bca Application in Playstore Using Naïve Bayes Algorithm," JATI (Journal of Mhs. Tech. Inform., vol. 8, no. 3, pp. 2511-2517, 2024, doi: 10.36040/jati.v8i3.9216.
References
S. I. Murpratiwi, S. E. Anjarwani, I. G. P. S. Wijaya, and A. Aranta, "Socialization of Healthy Internet and Training on the Use of Internet as Learning Support Media at SD Negeri Anggaraksa," J. Begawe Teknol. Inf., vol. 4, no. 2, pp. 256-263, 2023, doi: 10.29303/jbegati.v4i2.1111.
D. D. Audiansyah, D. E. Ratnawati, and B. T. Hanggara, "Sentiment Analysis of MyXL App using Support Vector Machine Method based on User Reviews in Google Play Store," vol. 6, no. 8, pp. 3987-3994, 2022.
A. Ahmad, W. Gata, and S. Panggabean, "Sentiment Analysis with Long Short-Term Memory and Synthetic Minority Over Sampling Technic in Digital Banking Applications," J. Technol. Inf. and Commun., vol. 8, no. 4, pp. 973-984, 2024.
T. Wahudi and Z. Hutabarat, "Factors Affecting the Intention to Use Digital Banking: Livin' By Mandiri," J. Tech. Inform. and Sist. Inf., vol. 10, no. 1, pp. 509-525, 2023.
M. Management, F. Economics, and U. P. Harapan, "Factors Affecting Intention to Use Digital Banking: Livin' By Mandiri," vol. 10, no. 1, pp. 509-525, 2023.
S. H. Alviyanti, A. Purwandira, I. Febiyanti, E. Daniati, and A. Ristyawan, "Sentiment Classification of Livin By Mandiri Application Users on Playstore Using Naive Bayes Algorithm," August, vol. 8, pp. 2549-7952, 2024.
S. L. Ranataru and N. Trianasari, "Social Media Sentiment Analysis of Banking Applications to Determine Application User Satisfaction: Case Study on Livin by Mandiri and BCA Mobile," Al-Kharaj J. Ekon. , Finance. Sharia Business, vol. 6, pp. 6818-6838, 2024, doi: 10.47467/alkharaj.v6i9.3805.
I. D. Onantya and P. P. Adikara, "Sentiment Analysis on BCA Mobile Application Reviews Using BM25 and Improved K-Nearest Neighbor," J-Ptiik.Ub.Ac.Id, vol. 3, no. 3, pp. 2575-2580, 2019.
M. Z. Hariansyah and Siswanto, "Implementation of Multinomial Naive Bayes Method on Sentiment Analysis of Livin by Mandiri Application Services," Semin. Nas. Mhs. Fak. Technol. Inf., vol. 1, no. 1, pp. 517-524, 2022.
N. Nurfadila, M. Ariyanti, and N. Trianasari, "Service Quality Analysis of New Livin' By Mandiri Mobile Banking Using Sentiment Analysis," JIBR J. Indones. Bus. Res., vol. 1, no. 1, pp. 77-82, 2023.
C. A. Qurniaty and K. Kusnawi, "Emotion Expression Based on Voice Using Multi Layer Perceptron Algortima and Support Vector Machine," Indonesia. J. Comput. Sci., vol. 12, no. 6, pp. 4014-4025, 2023, doi: 10.33022/ijcs.v12i6.3567.
G. G. Warow and H. Pandia, "Sentiment Analysis of Dana Application Using Naïve Bayes Classifier and Support Vector Machine," Jutisi J. Ilm. Tech. Inform. and Sist. Inf., vol. 13, no. 1, p. 609, 2024, doi: 10.35889/jutisi.v13i1.1893.
F. Riza and B. Kurniawan, "Sentiment analysis of BSI mobile application users due to ransomware using support vector machine algorithm," INFOTECH J. Inform. Technol., vol. 5, no. 1, pp. 42-51, 2024, doi: 10.37373/infotech.v5i1.1102.
R. A. Husen, R. Astuti, L. Marlia, R. Rahmaddeni, and L. Efrizoni, "Sentiment Analysis of Public Opinion on Twitter towards BSI Bank Using Machine Learning Algorithm," MALCOM Indones. J. Mach. Learn. Comput. Sci., vol. 3, no. 2, pp. 211-218, 2023, doi: 10.57152/malcom.v3i2.901.
F. H. Pasaribu, N. Khairina, D. Noviandri, S. Susilawati, and R. Syah, "Analysis of The Multilayer Perceptron Algorithm on Twitter User's Sentiment Towards The COVID-19 Vaccine," J. Informatics Telecommun. Eng., vol. 7, no. 1, pp. 155-163, 2023, doi: 10.31289/jite.v7i1.9664.
I. Daniel, A. Fahmi Limas Ptr, and A. Ichsan, "Risk Classification of Heart Attack Disease with Multi-Layer Perceptron," Data Sci. Indones., vol. 14, no. 1, pp. 57-64, 2024.
A. Purnamawati, M. N. Winarto, and M. Mailasari, "Sentiment Analysis of TikTok Application using BM25 Method and Improved K-NN Chi-Square Feature," J. Komtika (Computing and Inform., vol. 7, no. 1, pp. 97-105, 2023, doi: 10.31603/komtika.v7i1.8938.
R. Maulana, A. Voutama, and T. Ridwan, "Sentiment Analysis of MyPertamina App Reviews on Google Play Store using NBC Algorithm," J. Teknol. Integrated, vol. 9, no. 1, pp. 42-48, 2023, doi: 10.54914/jtt.v9i1.609.
D. Duei Putri, G. F. Nama, and W. E. Sulistiono, "Sentiment Analysis of the Performance of the House of Representatives (DPR) on Twitter Using the Naive Bayes Classifier Method," J. Inform. and Tech. Electro Applied, vol. 10, no. 1, pp. 34-40, 2022, doi: 10.23960/jitet.v10i1.2262.
N. Agustina, D. H. Citra, W. Purnama, C. Nisa, and A. R. Kurnia, "Implementation of Naive Bayes Algorithm for Sentiment Analysis of Shopee Reviews on Google Play Store," MALCOM Indones. J. Mach. Learn. Comput. Sci., vol. 2, no. 1, pp. 47-54, 2022, doi: 10.57152/malcom.v2i1.195.
R. A. Nurdian, Mujib Ridwan, and Ahmad Yusuf, "Comparison of SMOTE and ADASYN Methods in Improving the Performance of New Student Herregistration Classification," J. Tech. Inform. and Sist. Inf., vol. 8, no. 1, pp. 24-32, 2022, doi: 10.28932/jutisi.v8i1.4004.
S. Sasmita, R. N. Jariah S. Intam, D. F. Surianto, and M. F. B, "Sentiment Analysis of the Constitutional Court Decision Controversy Regarding the Age of Candidates Using Multi-Layer Perceptron with SMOTE Technique," Fakt. Exacta, vol. 17, no. 2, p. 188, 2024, doi: 10.30998/faktorexacta.v17i2.22442.
R. N. Harahap and K. Muslim, "Improving Accuracy in Twitter User Mbti Personality Prediction Using Data Augmentation," J. Technol. Inf. and Comput. Science, vol. 7, no. 4, p. 815, 2020, doi: 10.25126/jtiik.2020743622.
A. Nur Azizah, M. Falach Asy'ari, I. Wisma Dwi Prastya, and D. Purwitasari, "Easy Data Augmentation for Imbalance Data in Online Health Consultation," J. Technol. Inf. and Comput. Science, vol. 10, no. 5, pp. 1095-1104, 2023, doi: 10.25126/jtiik.20231057082.
I. Athiyyah Rahma and L. Hulliyyatus Suadaa, "Application of Text Augmentation to Overcome Unbalanced Data in Indonesian Text Classification Case Study: Detection of Clickbait Titles and Hate Speech Comments on Online News," J. Technol. Inf. and Comput. Science, vol. 10, no. 6, pp. 1329-1340, 2023, doi: 10.25126/jtiik.2023107325.
A. I. Tanggraeni and M. N. N. Sitokdana, "Sentiment Analysis of E-Government Applications on Google Play Using Naïve Bayes Algorithm," JATISI (Journal of Tech. Inform. and Information Systems), vol. 9, no. 2, pp. 785-795, 2022, doi: 10.35957/jatisi.v9i2.1835.
W. Wahyudi, R. Kurniawan, and Y. Arie Wijaya, "User Sentiment Analysis of Blu Bca Application in Playstore Using Naïve Bayes Algorithm," JATI (Journal of Mhs. Tech. Inform., vol. 8, no. 3, pp. 2511-2517, 2024, doi: 10.36040/jati.v8i3.9216.