User-Centered Experimental Evaluation of Recommendation Strategies in Online Education
1University “Ismail Qemali” Vlore, Albania
DOI: https://doi.org/10.63871/unvl.jsuv2.1.40
Keywords
- user behavior
- recommender systems
- online education
- course recommendation
- ANOVA
Abstract
Users of online education platforms often struggle to select courses that align with their needs and interests. This challenge arises because there are a large number of online courses. Recommendation systems can help address this challenge by directing users toward courses that are more relevant to their preferences. This paper presents a pilot experimental evaluation of recommendation strategies in online educational systems to assess their impact on user decision-making. A web-based platform was developed as an experimental framework to compare the effects of different recommendation strategies on user behavior. Three recommendation strategies were implemented in this platform: a standard interface without recommendations, a popularity-based recommendation strategy, and a content-based personalization strategy. This experimental platform was distributed to 51 Computer Science students at the University of Vlora in January 2026.
During the periment, the system recorded student interactions and course selections. The collected data was then evaluated using several behavioral metrics: the average number of selections per user, the total number of selected courses, the selection rate, and diversity. To compare the three experimental groups, Repeated-Measures ANOVA was used as a statistical test. The results show that the standard system reflects a high level of exploration, but with lower efficiency. The popularity-based method reduces the number of choices and was linked to increased user focus. While the content-based approach appears to be more effective, it achieves the highest selection rate and a better balance between exploration and orientation.
Declarations
Conflict of interest: The authors declare no conflict of interest.
Acknowledged: NA
AI use: NA
References
Adomavicius, G., & Tuzhilin, A. (2005). Toward the next generation of recommender systems: A survey of the state-of-the-art and possible extensions. IEEE Transactions on Knowledge and Data Engineering, 17(6), 734–749.
Amer, A. B., & Jamal, A. (2016). An automated recommender system for course selection. International Journal of Advanced Computer Science and Applications, 7(3), 166–175.
Anvitha, K., Durjay, T., Sathvika, K., Gnanendra, G., & Natarajan, S. K. (2025). EduBot: A compact AI-driven study assistant for contextual knowledge retrieval. In 2025 Global Conference in Emerging Technology (GINOTECH) (pp. 1–7). IEEE. DOI: 10.1109/GINOTECH63460.2025.1107709
Bekarystankyzy, A., Mereilim, I., Kassenkhan, A., Khan, V., & Tolegenov, A. (2026). Adaptive educational recommendation systems for personalized learning: A review of user modeling and machine learning approaches. International Journal of Advances in Signal and Image Sciences, 589–609.
Blythe, S. (2001). Designing online courses: User-centered practices. Computers and Composition, 18(4), 329–346.
Chaudhari, A., Seddig, A. H., Sarlan, A., & Raut, R. (2024). A hybrid recommendation system: A review. IEEE Access, 12, 157107–157126.
Chen, L., Chen, G., & Wang, F. (2021). Recommender systems based on deep learning: A review. IEEE Access, 9, 689–702.
Drachsler, H., Verbert, K., Santos, O. C., & Manouselis, N. (2015). Panorama of recommender systems in education: An overview of different approaches and their applications. International Journal of Technology Enhanced Learning, 7(1), 1–23.
Elahi, M., Kholgh, D. K., Kiarostami, M. S., Saghari, S., Rad, S. P., & Tkalčič, M. (2021). Investigating the impact of recommender systems on user-based and item-based popularity bias. Information Processing & Management, 58(5), 102655.
Field, A. (2024). Discovering Statistics Using IBM SPSS Statistics. Sage Publications Limited.
Hui, B., Zhang, L., Zhou, X., Wen, X., & Nian, Y. (2022). Personalized recommendation system based on knowledge embedding and historical behavior. Applied Intelligence, 52(1), 954–966.
Islam, F., Arman, M. S., Jahan, N., Sammak, M. H., Tasnim, N., & Mahmud, I. (2022). Model and popularity-based recommendation system: A collaborative filtering approach. In 2022 13th International Conference on Computing Communication and Networking Technologies (ICCCNT) (pp. 1–5). IEEE. DOI: 10.1109/ICCCNT54827.2022.9984348
Ko, H., Lee, S., Park, Y., & Choi, A. (2022). A survey of recommendation systems: Recommendation models, techniques, and application fields. Electronics, 11(1), 141. DOI: 10.3390/electronics11010141
Kundu, S. S., Sarkar, D., Jana, P., & Kole, D. K. (2020). Personalization in education using a recommendation system: An overview. In Computational Intelligence in Digital Pedagogy (pp. 85–111).
Lahiassi, J., Aammou, S., & Warraki, O. E. (2023). Enhancing personalized learning with a recommendation system in private online courses. Conhecimento & Diversidade, 15(39), 176–189.
Lenth, R. (2023). emmeans: Estimated Marginal Means, aka Least-Squares Means [R package]. https://cran.r-project.org/package=emmeans
Li, J., & Sun, Z. (2025, January). Development of a knowledge graph-based framework for music education in higher education: A technical and experimental approach. In Proceedings of the 2025 2nd International Conference on Informatics Education and Computer Technology Applications (pp. 341–345). DOI: 10.1109/ACCESS.2021.3069646
Li, J., & Ye, Z. (2020). Course recommendations in online education based on collaborative filtering recommendation algorithm. Complexity, 2020(1), 6619249.
Lynn, N. D., & Emanuel, A. W. R. (2021, March). A review of recommender systems for course selection in higher education. In IOP Conference Series: Materials Science and Engineering, 1098(3), 032039. IOP Publishing.
M. R. M., V., Rodriguez, C., Navarro Depaz, C., Concha, U. R., Pandey, B., S. Kharat, R., & Marappan, R. (2023). Machine learning-based recommendation system for web-search learning. Telecom, 4(1), 118–134. DOI: 10.3390/telecom4010008
Ma, F. (2025). Learning behavior analysis and personalized recommendation system of an online education platform based on machine learning. Computers and Education: Artificial Intelligence, 8, 100408.
Maphosa, M., Doorsamy, W., & Paul, B. (2020). A review of recommender systems for choosing elective courses. International Journal of Advanced Computer Science and Applications, 11(9), 287–295.
Resnick, P., & Varian, H. R. (1997). Recommender systems. Communications of the ACM, 40(3), 56–58.
Ricci, F., Rokach, L., & Shapira, B. (2021). Recommender systems: Techniques, applications, and challenges. In Recommender Systems Handbook, 1–35.
Ricci, F., Rokach, L., & Shapira, B. (2022). Recommender Systems Handbook (3rd ed.). Springer.
Shani, G., & Gunawardana, A. (2010). Evaluating recommendation systems. In Recommender Systems Handbook (pp. 257–297). Boston, MA: Springer US.
Son, J., & Kim, S. B. (2017). Content-based filtering for recommendation systems using multiattribute networks. Expert Systems with Applications, 89, 404–412.
The jamovi project. (2024). jamovi (Version 2.6) [Computer software]. https://www.jamovi.org
Tatineni, S. (2020). Recommendation systems for personalized learning: A data-driven approach in education. Journal of Computer Engineering and Technology (JCET), 4(2), 18–31.
Yin, S., Yang, K., & Wang, H. (2020, May). A MOOC course recommendation system based on learning behaviours. In Proceedings of the ACM Turing Celebration Conference-China (pp. 133–137). DOI: 10.1145/3393527.3393550
Zhang, S., Yao, L., Sun, A., & Tay, Y. (2019). Deep learning-based recommender system: A survey and new perspectives. ACM Computing Surveys, 52(1), 1–38. DOI: 10.1145/3285029
Citing literature
Copyright © 2026 The Author(s)
This article is licensed under the Creative Commons Attribution 4.0 International (CC BY 4.0) license.
