User-Centered Experimental Evaluation of Recommendation Strategies in Online Education

Abstract

Users of online education platforms often struggle to select courses that align with their needs and interests. This challenge arises because there are a large number of online courses. Recommendation systems can help address this challenge by directing users toward courses that are more relevant to their preferences. This paper presents a pilot experimental evaluation of recommendation strategies in online educational systems to assess their impact on user decision-making. A web-based platform was developed as an experimental framework to compare the effects of different recommendation strategies on user behavior. Three recommendation strategies were implemented in this platform: a standard interface without recommendations, a popularity-based recommendation strategy, and a content-based personalization strategy. This experimental platform was distributed to 51 Computer Science students at the University of Vlora in January 2026.     

During the periment, the system recorded student interactions and course selections. The collected data was then  evaluated using several behavioral metrics: the average number of selections per user, the total number of selected courses, the selection rate, and diversity.  To compare the three experimental groups, Repeated-Measures ANOVA was used as a statistical test.  The results show that the standard system reflects a high level of exploration, but with lower efficiency. The popularity-based method reduces the number of choices and was linked to increased user focus. While the content-based approach appears to be more effective, it achieves the highest selection rate and a better balance between exploration and orientation. 

CONFLICT OF INTEREST

The authors declare no conflict of interest.

REFERENCES

Adomavicius, G., & Tuzhilin, A. (2005). Toward the next generation of recommender systems: A survey of the state-of-the-art and possible extensions. IEEE Transactions on Knowledge and Data Engineering, 17(6), 734–749. Amer, A. B., & Jamal, A. (2016). An automated recommender system for course selection. International Journal of Advanced Computer Science and Applications, 7(3), 166-175. Anvitha, K., Durjay, T., Sathvika, K., Gnanendra, G., & Natarajan, S. K. (2025). EduBot: A compact AIdriven study assistant for contextual knowledge retrieval. In the 2025 Global Conference in Emerging Technology (GINOTECH) (pp. 1–7). IEEE. DOI: 10.1109/GINOTECH63460.2025.1107709 7. Bekarystankyzy, A., Mereilim, I., Kassenkhan, A., Khan, V., & Tolegenov, A. (2026). Adaptive Educational Recommendation Systems For Personalized Learning: A Review Of User Modeling And Machine Learning Approaches. INTERNATIONAL JOURNAL OF ADVANCES IN SIGNAL AND IMAGE SCIENCES, 589-609. Blythe, S. (2001). Designing online courses: Usercentered practices. Computers and Composition, 18(4), 329-346. Chaudhari, A., Seddig, A. H., Sarlan, A., & Raut, R. (2024). A hybrid recommendation system: A review. IEEE Access, 12, 157107–157126. Chen, L., Chen, G., & Wang, F. (2021). Recommender systems based on deep learning: A review. IEEE Access, 9, 689–702. Drachsler, H., Verbert, K., Santos, O. C., & Manouselis, N. (2015). Panorama of recommender systems in education: An overview of different approaches and their applications. International Journal of Technology Enhanced Learning, 7(1), 1–23. Elahi, M., Kholgh, D. K., Kiarostami, M. S., Saghari, S., Rad, S. P., & Tkalčič, M. (2021). Investigating the impact of recommender systems on userbased and item-based popularity bias. Information Processing & Management, 58(5), 102655. Field, A. (2024). Discovering statistics using IBM SPSS statistics. Sage Publications Limited. Hui, B., Zhang, L., Zhou, X., Wen, X., & Nian, Y. (2022). Personalized recommendation system based on knowledge embedding and historical behavior. Applied Intelligence, 52(1), 954–966. Islam, F., Arman, M. S., Jahan, N., Sammak, M. H., Tasnim, N., & Mahmud, I. (2022). Model and popularity-based recommendation system: A collaborative filtering approach. In 2022, the 13th International Conference on Computing Communication and Networking Technologies (ICCCNT) (pp. 1–5). IEEE. DOI: 10.1109/ICCCNT54827.2022.9984348 Ko, H., Lee, S., Park, Y., & Choi, A. (2022). A survey of recommendation systems: Recommendation models, techniques, and application fields. Electronics, 11(1), 141. https://doi.org/10.3390/electronics11010141 Kundu, S. S., Sarkar, D., Jana, P., & Kole, D. K. (2020). Personalization in education using a recommendation system: an overview. Computational Intelligence in Digital Pedagogy, 85-111. Lahiassi, J., Aammou, S., & Warraki, O. E. (2023). Enhancing personalized learning with a recommendation system in private online courses. Conhecimento & Diversidade, 15(39), 176–189. Lenth, R. (2023). emmeans: Estimated Marginal Means, aka Least-Squares Means. [R package]. Retrieved from https://cran.r-project.org/package=emmeans. Li, J., & Sun, Z. (2025, January). Development of a Knowledge Graph-Based Framework for Music Education in Higher Education: A Technical and Experimental Approach. In Proceedings of the 2025 2nd International Conference on Informatics Education and Computer Technology Applications (pp. 341-345). DOI: 10.1109/ACCESS.2021.3069646. Li, J., & Ye, Z. (2020). Course recommendations in online education based on collaborative filtering recommendation algorithm. Complexity, 2020(1), 6619249. Lynn, N. D., & Emanuel, A. W. R. (2021, March). A review of Recommender Systems for course selection in higher education. In IOP conference series: Materials science and engineering (Vol. 1098, No. 3, p. 032039). IOP Publishing. M. R. M., V., Rodriguez, C., Navarro Depaz, C., Concha, U. R., Pandey, B., S. Kharat, R., & Marappan, R. (2023). Machine learning-based recommendation system for web-search learning. Telecom, 4(1), 118–134. https://doi.org/10.3390/telecom4010008 Ma, F. (2025). Learning behavior analysis and personalized recommendation system of an online education platform based on machine learning. Computers and Education: Artificial Intelligence, 8, 100408. Maphosa, M., Doorsamy, W., & Paul, B. (2020). A review of recommender systems for choosing elective courses. International Journal of Advanced Computer Science and Applications, 11(9), 287-295. Resnick, P., & Varian, H. R. (1997). Recommender systems. Communications of the ACM, 40(3), 5658. Ricci, F., Rokach, L., & Shapira, B. (2021). Recommender systems: Techniques, applications, and challenges. Recommender Systems Handbook, 135. Ricci, F., Rokach, L., & Shapira, B. (2022). Recommender systems handbook (3rd ed.). Springer. Shani, G., & Gunawardana, A. (2010). Evaluating recommendation systems. In Recommender Systems Handbook (pp. 257-297). Boston, MA: Springer US. Son, J., & Kim, S. B. (2017). Content-based filtering for recommendation systems using multiattribute networks. Expert Systems with Applications, 89, 404-412. The jamovi project. (2024). jamovi (Version 2.6) [Computer software]. https://www.jamovi.org Tatineni, S. (2020). Recommendation systems for personalized learning: A data-driven approach in education. Journal of Computer Engineering and Technology (JCET), 4(2), 18-31. Yin, S., Yang, K., & Wang, H. (2020, May). A MOOC course recommendation system based on learning behaviours. In Proceedings of the ACM Turing Celebration Conference-China (pp. 133-137). DOI:10.1145/3393527.3393550 Zhang, S., Yao, L., Sun, A., & Tay, Y. (2019). Deep learning-based recommender system: A survey and new perspectives. ACM Computing Surveys, 52(1), 1–38. https://doi.org/10.1145/3285029

Citing Literature