[Paper Review] Curriculum Q-Learning for Visual Vocabulary Acquisition

Ahmed Zaidi, Russell Moore+1 more|arXiv (Cornell University)|Nov 29, 2017

Visual and Cognitive Learning Processes10 references3 citations

TL;DR

This paper proposes a curriculum Q-learning framework that personalizes visual vocabulary acquisition by adapting to individual learners' proficiency levels using reinforcement learning. The system dynamically selects words based on the learner's zone of proximal development, achieving improved learning rates and adaptive progression across beginner, intermediate, and advanced simulated students.

ABSTRACT

The structure of curriculum plays a vital role in our learning process, both as children and adults. Presenting material in ascending order of difficulty that also exploits prior knowledge can have a significant impact on the rate of learning. However, the notion of difficulty and prior knowledge differs from person to person. Motivated by the need for a personalised curriculum, we present a novel method of curriculum learning for vocabulary words in the form of visual prompts. We employ a reinforcement learning model grounded in pedagogical theories that emulates the actions of a tutor. We simulate three students with different levels of vocabulary knowledge in order to evaluate the how well our model adapts to the environment. The results of the simulation reveal that through interaction, the model is able to identify the areas of weakness, as well as push students to the edge of their ZPD. We hypothesise that these methods can also be effective in training agents to learn language representations in a simulated environment where it has previously been shown that order of words and prior knowledge play an important role in the efficacy of language learning.

Motivation & Objective

• To develop a personalized curriculum for visual vocabulary acquisition using reinforcement learning grounded in pedagogical theories.
• To simulate how an automated tutor can adapt to learners at different CEFR proficiency levels (A1 to C2).
• To evaluate whether the system can identify learner weaknesses and push them to the edge of their zone of proximal development (ZPD).
• To explore the parallels between reinforcement learning components and principles of language acquisition, such as Krashen’s i+1 hypothesis.
• To establish a foundation for future RL-based pedagogical systems that incorporate memory, spaced repetition, and deep learning.

Proposed method

• The agent acts as a tutor using Q-Learning to select visual vocabulary items (words) for students based on their proficiency level.
• The environment is a simulated student whose response probability is modeled using a negated Gompertz distribution based on the difference between student level (u) and item level (q).
• The Q-value update rule follows standard Q-Learning: $Q(s,a)\leftarrow Q(s,a)+\alpha[r+\gamma\max_{a'}Q(s',a')-Q(s,a)]$, with state defined by student proficiency and action by word selection.
• The system uses a curriculum strategy that prioritizes items within the learner’s ZPD, ensuring incremental progression.
• Proficiency levels are mapped to the CEFR scale (0 = A1, 6 = C2), and the agent selects items to maximize cumulative reward over 100 interactions.
• The model leverages the ε-greedy strategy to balance exploration and exploitation, aligning with the input hypothesis (i+1) in second language acquisition.

Experimental results

Research questions

• RQ1 Can a reinforcement learning agent learn to construct a personalized curriculum that adapts to individual learners’ vocabulary proficiency levels?
• RQ2 How effectively does the system identify and target the learner’s zone of proximal development (ZPD) across different proficiency levels?
• RQ3 To what extent does the curriculum strategy improve learning outcomes compared to a static or non-adaptive approach?
• RQ4 What are the parallels between Q-Learning components (e.g., ε-greedy, Q-value updates) and established theories of language acquisition like Krashen’s i+1 hypothesis?
• RQ5 Can the system be extended to incorporate memory, spaced repetition, and deep learning for improved long-term retention and scalability?

Key findings

• The agent successfully adapted to each student’s proficiency level, maintaining beginner students around A1–A2 and guiding intermediate students to B2 (level 3) over 100 interactions.
• Advanced students were challenged with items slightly above their level, resulting in cumulative rewards that indicated progression beyond their initial proficiency.
• The system demonstrated a clear downward slope in cumulative reward when students reached the limit of their current ability, indicating that the agent pushed them to the edge of their ZPD.
• The model showed distinct behavioral differences across proficiency levels, confirming that the curriculum was personalized and responsive to individual learning states.
• The use of Q-Learning ensured an optimal path toward language fluency by minimizing steps and maximizing learning efficiency.
• The framework provides a scalable testbed for evaluating the difficulty and appropriateness of new vocabulary items using metrics derived from learner interactions.