[Paper Review] A pilot study of the use of LogEx, lessons learned

Josje Lodder, Bastiaan Heeren+1 more|arXiv (Cornell University)|Jul 14, 2015

Intelligent Tutoring Systems and Adaptive Learning9 references3 citations

TL;DR

This pilot study evaluates LogEx, an e-learning tool for teaching propositional logic rewriting using standard equivalences. It finds that while students improve in rule application and formula normalization, they lack strategic insight due to insufficient feedback on solution strategies, prompting recommendations for enhanced feedback and logging in large-scale evaluations.

ABSTRACT

LogEx is a learning environment that supports students in rewriting propositional logical formulae, using standard equivalences. We organized a pilot study to prepare a large scale evaluation of the learning environment. In this paper we describe this study, together with the outcomes, which teach us valuable lessons for the large scale evaluation.

Motivation & Objective

• Evaluate the effectiveness of LogEx, an e-learning environment for teaching propositional logic rewriting using standard equivalences.
• Assess whether students learn core competencies such as rule application, formula normalization, and equivalence proving through the tool.
• Identify usability and feedback issues to inform the design of a large-scale evaluation of LogEx.
• Improve the tool’s feedback mechanisms, particularly on strategic decision-making, based on observed student behavior.
• Ensure logging infrastructure captures all user actions, including undo and hint usage, for accurate learning outcome analysis.

Proposed method

• Conducted a small-scale experiment with first-year students at Utrecht University using LogEx in a controlled classroom setting.
• Administered pre- and post-tests to measure learning gains across three exercise types: tautology proof, normal form rewriting, and equivalence proving.
• Collected detailed log data of student interactions, including step-by-step rewrites, rule applications, hints requested, and undo actions.
• Analyzed logs to detect patterns in rule usage, strategy development, and help-seeking behavior, especially regarding inefficient or incorrect approaches.
• Used semantic checking and a set of predefined 'buggy rules' to provide feedback on incorrect rewrites, including explanations for common errors.
• Evaluated the impact of feedback types (syntax, rule, semantic, and hint-based) on student performance and learning outcomes.

Experimental results

Research questions

• RQ1 To what extent do students improve in applying propositional logic equivalences after using LogEx?
• RQ2 How effective is LogEx in supporting students in rewriting formulas into DNF and CNF, and proving equivalence?
• RQ3 What are the key usability and feedback issues that hinder students from developing strategic insight in proof construction?
• RQ4 How do students use hints, next-step suggestions, and complete solutions, and what impact does this have on learning outcomes?
• RQ5 What logging limitations affect the ability to assess student learning and behavior accurately in the tool?

Key findings

• Students showed significant improvement in applying logical rules correctly and in rewriting formulas into normal forms, as evidenced by pre- and post-test results.
• Despite progress in rule application and normalization, students did not demonstrate improved strategic insight, as measured by solution efficiency and path choice.
• A student developed a non-optimal strategy involving repeated double negation and De Morgan’s law, which persisted due to lack of feedback on strategy.
• Only one student used hints or next-step features, and only one used the complete solution comparison, indicating widespread help avoidance.
• The system failed to log undo actions, limiting the ability to track solution evolution and detect when students backtracked or revised steps.
• Students struggled with commutative variants of distributivity, incorrectly assuming that commutative reordering was allowed in a single step, revealing unclear instruction on rule applicability.