RESEARCH
How practical is using AI for language acquisition?
When starting this project, I wanted to explore whether AI chatbots could be a good tool for language acquisition. This is the research I conducted to answer that question.
Summary
Problem Statement: Learning how to speak a new language is hard. Not only do you have to interpret what the other person is saying, but you also have to think of your answer, translate it, and speak it confidently. Could practicing with an AI-generated chatbot help language learners build confidence in speaking faster?
-
2023
-
-A Literature Review of academic sources around AI and language acquisition in adults.
-A Competitive Analysis to understand the landscape of AI language learning technology in 2023 and learn more about what capabilities are available with AI-based language chatbots, what they can do, and where there is opportunity for improvement before going into the next phase of testing.
-A Diary Study to understand if AI language learning technology can be successful in making users comfortable in speaking in another language.
-And finally a Survey to uncover the strengths and weaknesses of using an AI chatbot for language learning.
-
To understand the landscape of AI language learning technology in 2023.
To explore if an AI language learning chatbot can be successful in making users comfortable in speaking in another language.
Uncover the strengths and weaknesses of using an AI-based chatbot for language learning.
-
-What are the best ways for adults to become proficient in speaking in another language?
-What AI language learning chatbots exist on the market (if any) and what features/capabilities do they have?
-Does using these AI-based chatbots encourage users to continue using them, thus successfully learning to speak in another language?
-Do AI-based chatbots build comfort in users when speaking in another language?
-What issues do language learners have with learning how to speak in another language?
-What issues could arise in using an AI-based chatbot for language learning, rather going through more traditional routes?
Execution
Literature Review
I spent the first week exploring academic, business, and research publications, as well as reputable mainstream media articles, to understand how adults learn language, how AI has been applied to language learning from an academic level, and anecdotally if anyone in the media has been exploring the idea of using AI technology for language acquisition.
What I found was that other studies have been conducted on the possibility of using AI for language acquisition. The idea is promising due to users' high interest in the technology and the flexibility to practice language on the go. However, not enough research exists on how effective this method is as a learning tool.
For how adults learn language, I learned that even though immersion is still considered the best way, humans really need three things to learn a language: comprehensible input (reading and listening practice), comprehensible output (speaking and writing practice), and feedback (the opportunity to be corrected).
Finally, research supports that Duolingo is not effective alone in teaching language because it does not give users the autonomy to learn what they want to learn and does not give enough practice in the three variables I listed above.
Significant sources I referenced in my research are listed in the table below.
Competitive Analysis
For the competitive analysis, the 5 most prominent AI chatbot language applications were compared to each other to understand what existed on the market, their competitive advantages, and each user experience.
The 5 apps reviewed were LangoTalk, PolyglotAI, Memrise, Mondly, and ChatGPT-4. These were chosen based on having the most downloads at the time of the research.
The five chatbot applications were reviewed based on criteria relevant to their performance, user experience, and market standing: Estimated Number of Users, Company Profile, Key Competitive Advantage, Reputation, Target Market, Languages Offered, Features, Pricing, Distribution Channels, Quality, Strengths, and Weaknesses.
This exercise served as a way to choose an application for the upcoming diary study's participants. Based on this analysis, I went with LangoTalk for the diary study. LangoTalk allowed the user to practice speaking with an AI Chatbot about various subjects, and the conversations tested in Spanish sounded close to a natural human conversation, which were features I was looking for for my future study.
Details about the competitive analysis can be viewed below.
LangoTalk
-
Many features; Easy to navigate interface; Ability to speak in English or the target language to the bot; Written or spoken practice in the target language
-
Translation in Spanish needed work; Needed more helpful feedback when a mistake was made; Kinda just throws the user into situations without context; Points system doesn't make sense
Polyglot AI
-
Allows speaking, reading, and writing practice for learners
-
Limited features; Clunky user experience
Mondly
-
Interesting concept (AR application). Tries to physically put the user in situations they might find themselves in, in a foreign country, like ordering at a restaurant, for example; Tries to focus on functional language
-
The quality of the audio and visuals are pretty crappy; Sounds unnatural and choppy; Lessons are repetitive and very beginner level; Other courses are all the same; Just Google translated text so the quality of instruction is not high.
Memrise
-
Hear native speakers speak in the target language; Many features; More autonomy on what to study; Best Spanish translation with multiple dialects represented; Slang and natural conversation listening practice
-
AI bot is only text-based, so not a lot of speaking practice; Conversations are limited and repetitive like Duolingo
ChatGPT-4
-
Powerful chatbot that can answer the most questions in the most natural way; Can answer questions posed to it in other languages; Users can speak out loud to it like on other platforms; Updates consistently
-
It is not really tailored to a language-learning experience, but this model can be used for a language-learning chatbot.
Diary Study
Recruitment
I recruited 8 people who were:
Adults (18+)
Currently learning or have recently (within the past year) stopped learning Spanish
They must be residing within the United States
They must have tried either a language learning app or other method for learning a language within the past year and to be able to speak on their experience clearly
And they must be willing to participate in a week-long diary study
These 8 people were recruited through UserInterviews.com.
Execution
The diary studies were conducted as follows:
Participants were recruited from UserInterviews.com and were sent a Zoom link for a 20-minute introductory phone call
In the introductions, participants were asked about their background in language learning and thoughts on AI and briefed on the activities required to complete the diary study. This was also an opportunity for them to ask questions.
Participants were then emailed the link to the app they would be using for the rest of the week and the first diary study form.
The participants were asked to use the app for the first day, the 7th day, and at least 2 days in between and fill out a short form on those days asking about their experience with the app.
The diary study form was emailed to them each morning, and they could fill out that day’s form if they used the app.
The Diary study forms were short surveys created in Qualtrics for quick deployment and analysis.
Once all of the diary studies were completed, the information was downloaded and loaded into a spreadsheet that also linked the participants' introductory interview answers to their diary study responses. The diary studies, along with the survey, which was running at the same time, were then qualitatively coded and analyzed for common themes.
A draft of the diary study questions can be found here.
Survey
Survey
Survey Details
Built in Qualtrics
20 questions long
Collected the following information: Basic demographic information, information on how participants feel about AI and information about how participants have tried learning a new language
Participants were recruited through LinkedIn, Instagram, and Reddit channels specifically for survey recruitment, Surveyswap.io, and various NYU email lists and Slack channels
The survey had a built-in screener to filter out information from people outside the survey’s target audience
Target audience: adults (18+) who were currently or recently stopped learning a foreign language within the past year
I originally targeted 100 participants for the survey but eventually had to close it after 50 responses because getting (quality) survey responses on your own, for free, turned out to be very challenging. However, 50 responses were enough to qualitatively draw informed assumptions from.
Click the image to the left to view the survey.
Results
Research Key Insights
The main objective of this research was to discover the strengths and weaknesses of using an AI chatbot for language learning in adults and how it compares to other methods.
At the beginning of the Diary Study, all participants felt excited to use the AI tool, expressing excitement and the novelty factor of the tool. However, as the week wore on, 2 participants lost motivation and dropped out, expressing that the conversations with the tool were too monotonous or it was too difficult to come up with conversations on their own. The 4 of the remaining 6 participants rated their experience more favorably, but these participants were also more intermediate in Spanish. Feedback about the app included wishing the subscription was cheaper to unlock more conversations and liking the concept but wanting the execution to be better.
Diary Study Participant, Spanish Language Beginner
“I really don’t think that this tool will be helpful. I like the concept of having a conversation, but I don’t feel like my Spanish is advanced enough to hold a conversation.”
Diary Study Participant, Spanish Language Intermediate
“It’s a little helpful because it corrects me when I talk incorrectly. It’s a little hard because you already have to know a lot of Spanish to converse. It makes it fun to learn with the conversations.”
For the survey, participants relayed time and motivation as the top reasons they struggle with consistently learning a language. For language learning methods they have tried, mobile apps were the most popular and accessible method, but any method involving another human was ranked more favorably, suggesting human intervention is still the most effective method.
CONCLUSION
Whether AI tools can be used to make language learning easier and more efficient, the answer could be yes, depending on the need. Past studies point to AI tools being valuable to those who need more speaking practice in a low-pressure environment. This research supported those findings, with some exceptions. AI language tools need to be better tailored to people of different language proficiencies if they are to be effective. There is also a gap in the market for AI language tools that are free or low-cost to use for those who can’t afford an expensive subscription, excluding those who could benefit the most from this type of technology. Lastly, the technology still isn’t to a point where conversations with an AI bot are natural enough to give as much benefit to practicing with a live human. However, it can offer a better alternative or supplement to learning with popular apps on the market because this technology offers more autonomy for learning than previously available.
While this technology has the potential to be useful to adult language learners, there are significant gaps in the space of AI language learning applications that need to be addressed. These gaps include better learning outcomes, more beginner friendliness, and more user-led learning. Addressing these gaps is crucial to ensure the continued evolution and effectiveness of AI language tools in the future.
Follow along to the next section to see how I attempted to address these gaps through a new language application design.