[Paper Review] On feature selection and evaluation of transportation mode prediction strategies

Mohammad Etemad, Amílcar Soares+1 more|arXiv (Cornell University)|Aug 9, 2018

Human Mobility and Location-Based Analysis25 references4 citations

TL;DR

This paper proposes a feature selection framework combining wrapper and information-theoretic methods to optimize trajectory-based transportation mode prediction. Using random forest with top 20 features, it achieves 69.5% accuracy (p=0.0431) against endo2016deep and 88.5% (p=0.0796) against dabiri2018inferring, outperforming both while ensuring model interpretability and lower computational cost.

ABSTRACT

Transportation modes prediction is a fundamental task for decision making in smart cities and traffic management systems. Traffic policies designed based on trajectory mining can save money and time for authorities and the public. It may reduce the fuel consumption and commute time and moreover, may provide more pleasant moments for residents and tourists. Since the number of features that may be used to predict a user transportation mode can be substantial, finding a subset of features that maximizes a performance measure is worth investigating. In this work, we explore wrapper and information retrieval methods to find the best subset of trajectory features. After finding the best classifier and the best feature subset, our results were compared with two related papers that applied deep learning methods and the results showed that our framework achieved better performance. Furthermore, two types of cross-validation approaches were investigated, and the performance results show that the random cross-validation method provides optimistic results.

Motivation & Objective

• To identify the optimal subset of trajectory features for transportation mode prediction using feature selection techniques.
• To compare the performance of multiple classifiers, including random forest and XGBoost, in predicting transportation modes from trajectory data.
• To evaluate the proposed framework against state-of-the-art deep learning models (endo2016deep and dabiri2018inferring) in terms of accuracy and computational efficiency.
• To investigate the impact of cross-validation strategies—specifically random vs. user-oriented cross-validation—on model evaluation reliability.
• To ensure model interpretability by using handcrafted, human-readable features rather than end-to-end representation learning.

Proposed method

• Employed a wrapper method (sequential forward selection) and an information-theoretic method (based on mutual information) to rank and select the most relevant trajectory features.
• Used 70 handcrafted trajectory features, including speed percentiles (e.g., $F_{p90}^{speed}$), distance, and acceleration metrics, to represent mobility behavior.
• Applied a five-fold cross-validation and user-oriented cross-validation to evaluate model generalization, ensuring users appear only in one split.
• Selected the top 20 features from the wrapper method and used them with a random forest classifier (50 estimators) for final evaluation.
• Conducted one-sample Wilcoxon signed-ranks tests to statistically compare performance against baseline models from endo2016deep and dabiri2018inferring.
• Avoided noise removal in testing to prevent data leakage and ensure realistic performance estimation.

Experimental results

Research questions

• RQ1 Which combination of feature selection method and classifier yields the highest accuracy in transportation mode prediction from trajectory data?
• RQ2 How does the performance of the proposed model compare to state-of-the-art deep learning models (endo2016deep and dabiri2018inferring) in terms of accuracy and computational cost?
• RQ3 What is the impact of different cross-validation strategies—random vs. user-oriented—on the evaluation of transportation mode prediction models?
• RQ4 Which trajectory features are most predictive of transportation mode, and why are they robust to noise?
• RQ5 Can interpretable, handcrafted features outperform end-to-end learned representations in this task?

Key findings

• The random forest classifier achieved the highest accuracy (69.5%) among the evaluated classifiers, significantly outperforming the SVM and matching XGBoost in performance.
• The wrapper method identified $F_{p90}^{speed}$ as the most important feature, which is robust to outliers due to its percentile-based calculation.
• The proposed model achieved 69.5% accuracy on the test set, significantly higher than endo2016deep’s 67.9% (p=0.0431), demonstrating superior performance.
• The model achieved 88.5% accuracy on the dabiri2018inferring benchmark, outperforming their reported 84.8% (p=0.0796), though the result was not statistically significant at p<0.05.
• User-oriented cross-validation produced more realistic and conservative performance estimates, while random cross-validation yielded overly optimistic results due to data leakage across users.
• The framework provides a more interpretable and computationally efficient alternative to deep learning models, with handcrafted features that are easier to understand and validate.