[Paper Review] Chessboard and chess piece recognition with the support of neural networks

Maciej A. Czyzewski, Artur Laskowski+1 more|arXiv (Cornell University)|Aug 13, 2017

Image and Object Detection Techniques4 citations

TL;DR

This paper proposes a robust, end-to-end neural network-based method for detecting chessboards and recognizing chess pieces in real-world images, even under challenging lighting and angle conditions. The approach combines iterative line detection, lattice point localization, and piece classification using a custom convolutional neural network, achieving over 99.5% accuracy in lattice point detection and nearly 95% in piece recognition—significantly outperforming existing methods.

ABSTRACT

Chessboard and chess piece recognition is a computer vision problem that has not yet been efficiently solved. However, its solution is crucial for many experienced players who wish to compete against AI bots, but also prefer to make decisions based on the analysis of a physical chessboard. It is also important for organizers of chess tournaments who wish to digitize play for online broadcasting or ordinary players who wish to share their gameplay with friends. Typically, such digitization tasks are performed by humans or with the aid of specialized chessboards and pieces. However, neither solution is easy or convenient. To solve this problem, we propose a novel algorithm for digitizing chessboard configurations. We designed a method that is resistant to lighting conditions and the angle at which images are captured, and works correctly with numerous chessboard styles. The proposed algorithm processes pictures iteratively. During each iteration, it executes three major sub-processes: detecting straight lines, finding lattice points, and positioning the chessboard. Finally, we identify all chess pieces and generate a description of the board utilizing standard notation. For each of these steps, we designed our own algorithm that surpasses existing solutions. We support our algorithms by utilizing machine learning techniques whenever possible. The described method performs extraordinarily well and achieves an accuracy over $99.5\\%$ for detecting chessboard lattice points (compared to the $74\\%$ for the best alternative), $95\\%$ (compared to $60\\%$ for the best alternative) for positioning the chessboard in an image, and almost $95\\%$ for chess piece recognition.

Motivation & Objective

• To develop a robust, real-world applicable method for digitizing physical chessboards without requiring specialized hardware or controlled imaging conditions.
• To address the limitations of existing chessboard detection systems, which often fail under variable lighting, angles, or board styles.
• To improve accuracy in both chessboard localization and chess piece recognition beyond current state-of-the-art methods.
• To create a publicly available, enhanced dataset to support future research in chessboard recognition.
• To enable practical applications such as live tournament digitization, AI opponent integration, and game sharing among players.

Proposed method

• The method processes images iteratively through three core stages: straight line detection using a modified SLID algorithm that merges short segments into longer lines.
• Lattice points are detected using a custom-trained convolutional neural network (CNN) that identifies corners even in deformed or partially obscured boards.
• The chessboard is localized by analyzing geometric relationships between detected lines and lattice points, enabling robust positioning under various angles and lighting.
• Chess piece recognition is performed using a CNN-based classifier enhanced by probabilistic filtering via the Stockfish chess engine to reject implausible configurations.
• The system integrates feedback from Stockfish to refine predictions, reducing common errors such as misclassifying bishops as pawns.
• A novel, large-scale dataset was curated from real tournament broadcasts and shared publicly to support future research.

Experimental results

Research questions

• RQ1 Can a computer vision system achieve high-accuracy chessboard detection in real-world images with variable lighting, angles, and board styles?
• RQ2 How does a neural network-based lattice point detector compare to traditional methods like Hough transform or ChESS in handling deformed or obscured corners?
• RQ3 To what extent can integrating a chess engine's evaluation improve the accuracy of piece recognition in ambiguous cases?
• RQ4 Can the proposed method outperform existing open-source solutions in both board localization and piece classification under realistic conditions?
• RQ5 How effective is the method in practical applications such as digitizing live chess tournaments or enabling human-AI chess play from physical boards?

Key findings

• The proposed method achieves 99.5% accuracy in detecting chessboard lattice points, significantly outperforming the best alternative method, which reaches only 74%.
• Chessboard localization accuracy reaches 95%, compared to 60% for the best existing alternative, demonstrating robustness to image capture angles and lighting variations.
• Chess piece recognition accuracy is nearly 95%, marking the first method capable of reliable real-world deployment for game digitization.
• The use of a CNN for lattice point detection reduces misclassification of deformed points by over 50% compared to the ChESS classifier, which identifies only 43% of true positives.
• Integration with the Stockfish chess engine significantly improves classification by filtering out low-probability board configurations, reducing common misclassification errors.
• The method has already attracted practical interest, with multiple users expressing intent to deploy it in real applications, and the training data has been shared to support future research.