Erwin Wu

Learning an advanced skill in sports requires a huge amount of practice and players also have to overcome both physical difficulties and the dullness of repetitive training. Returning a fast spin shot in table tennis could be taken as an example, as athletes need to judge the spin type and decide the racket pose within a second, which is difficult for beginners. Therefore, in this paper, we show how to design an intuitive training system to acquire this specific skill using different cues in Vir

In this paper, we propose a novel mixed reality martial arts training system using deep learning based real-time human pose forecasting. Our training system is based on 3D pose estimation using a residual neural network with input from a RGB camera, which captures the motion of a trainer. The student wearing a head mounted display can see the virtual model of the trainer and his forecasted future pose. The pose forecasting is based on recurrent networks, to improve the learning quantity of the m

The automatic recognition of how people use their hands and fingers in natural settings -- without instrumenting the fingers -- can be useful for many mobile computing applications. To achieve such an interface, we propose a vision-based 3D hand pose estimation framework using a wrist-worn camera. The main challenge is the oblique angle of the wrist-worn camera, which makes the fingers scarcely visible. To address this, a special network that observes deformations on the back of the hand is requ

In most sports, the ability to forecast motions and trajectories is among the highest priority, which can be only earned from experience. How to predict the motion from image and visualize for training is a challenging topic for computer vision. In this paper, we present a real-time table tennis forecasting system using a long short-term pose prediction network. Our system can predict the landing point of a serve before the pingpong ball is even hit using the previous and present motions of a pl

Alpine ski training is restricted by environmental requirements and the incremental and cyclical ways of how movement and form are taught. Therefore, we propose a virtual reality ski training system based on an indoor ski simulator. The system uses two trackers to capture the motion of skis so that users can control them on the virtual ski slope. For training we captured the motion of professional athletes and replay them to the users to help them to improve their skills. In two studies, we expl

Alpine skiing has strong environmental dependencies and the way of teaching the movement is believed to be incremental and cyclical. Training alpine skiing on simulators is a challenging work, especially when supporting experienced learner to improve to higher level. In this paper, we propose several vision augmentations for learning from a recorded expert skier motion in the way to replay the motion as a virtual leading skier. The system uses an stationary indoor ski simulator and a VR System f

Humans’ ability to forecast motions and trajectories, are one of the most important abilities in many sports. With the development of deep learning and computer vision, it is becoming possible to do the same thing with real-time computing. In this paper, we present a real-time table tennis forecasting system using a long short-term pose prediction network. Our system can predict the trajectory of a serve before the pingpong ball is even hit based on the previous and present motions of a player,

We propose a real-time human motion forecasting system which visualize the future pose in virtual reality using a RGB camera. Our system consists of three parts: 2D pose estimation from RGB frames using a residual neural network, 2D pose forecasting using a recurrent neural network, and 3D recovery from the predicted 2D pose using a residual linear network. To improve the prediction learning quantity of temporal feature, we propose a special method using lattice optical flow for the joints movem

This paper propose a real-time human motion forecasting system which visualize the future pose in virtual reality using a RGB camera. Our system consists of three parts: 2D pose estimation from RGB frames using a residual neural network, 2D pose forecasting using a recurrent neural network, and 3D recovery from the predicted 2D pose using a residual linear network. To improve the prediction learning quantity of temporal feature, we propose a special method using lattice optical flow for the join

We propose SolePoser, a real-time 3D pose estimation system that leverages only a single pair of insole sensors. Unlike conventional methods relying on fixed cameras or bulky wearable sensors, our approach offers minimal and natural setup requirements. The proposed system utilizes pressure and IMU sensors embedded in insoles to capture the body weight’s pressure distribution at the feet and its 6 DoF acceleration. This information is used to estimate the 3D full-body joint position by a two-stre

Hand pose analysis is a key step to understanding dexterous hand performances of many high-level skills, such as playing the piano. Currently, most accurate hand tracking systems are using fabric-/marker-based sensing that potentially disturbs users’ performance. On the other hand, markerless computer vision-based methods rely on a precise bare-hand dataset for training, which is difficult to obtain. In this paper, we collect a large-scale high precision 3D hand pose dataset with a small workloa

Effective analysis of skills requires high-quality, multi-modal datasets, especially in the field of artificial intelligence. However, creating such datasets for extreme sports, such as alpine skiing, can be challenging due to environmental constraints. Optical and wearable sensors may not perform optimally under diverse lighting, weather, and terrain conditions. To address these challenges, we present a comprehensive skiing/snowboarding dataset using a professional motor-based simulator. Using

We propose a real-time dual-modal 3D pose estimation system leveraging a single pair of insole sensors. Unlike conventional methods requiring fixed cameras or bulky wearable sensors, our approach offers minimal and natural setup requirements. The system uses pressure and IMU sensors embedded in insoles to capture foot pressure distribution and 6 DoF acceleration, estimating 3D full-body joint positions via a two-stream transformer network. We introduce a novel double-cycle consistency loss and a

Using smartphones while walking is becoming a social problem. Recent works try to support this issue by different warning systems. However, most only focus on detecting obstacles, without considering the risk to the user. In this paper, we propose a deep learning-based context-aware risk prediction system using a built-in camera on smartphones, aiming to notify ”smombies” by a risk-degree based algorithm. The proposed system both estimates the risk degree of a potential obstacle and the user’s s