Comparative analysis of the impact of AI-based chatbots on language learning in comparison with traditional methods with empirical data from multilingual educational programs

Abstract

The study examines the effectiveness of AI-based chatbots in language learning compared to traditional methods in real multilingual programs. The relevance is defined by the shortage of personalized conversational practice and the need to scale individualized support. The goal is to quantitatively assess the chatbot's contribution to language competence development and learner engagement, testing the transferability of the effect across different languages and proficiency levels. The novelty of the work lies in a controlled comparison within the educational process, as well as in linking behavioral usage metrics with learning outcomes. Materials and methods include randomized assignment of 254 students into control and experimental groups, a total observation period of 16 weeks, pre- and post-testing in reading, writing, listening, and speaking, weekly vocabulary tests, chatbot interaction logs, and satisfaction surveys. The experimental group additionally practiced with a large language model-based chatbot for at least two hours per week. Statistical processing included descriptive measures, t-tests for independent samples, and correlation analysis controlling for baseline levels. Results show statistically significant superiority of the experimental group in engagement and learning outcomes. Average self-study time was 5.42 versus 2.85 hours per week, satisfaction scores 8.94 versus 7.18. Over the semester, an average of 568 versus 413 lexical items were acquired, with retention rates of 84,1% versus 68,3%. The greatest improvement was recorded in speaking; dialogue time with the chatbot strongly correlated with gains in this skill. The effect persists when accounting for initial differences and variance homogeneity. The discussion interprets the effect as a result of personalization, instant feedback, and reduction of the affective barrier. Practical recommendations include hybrid models integrating chatbots into assignments, progress analytics, and teacher training on designing dialogic scenarios. Limitations are related to a single educational environment, semester-long timeframe, and dependence on model quality; further longitudinal and cross-age studies are needed, as well as evaluation of long-term retention and academic integrity. The significance of the work lies in demonstrating a sustainable 25-40% increase in key metrics with moderate resource costs. Results are reproducible across different language tracks and proficiency levels. Conclusions are relevant for designing blended learning environments and courses.