[Paper Review] San Francisco Crime Classification

Yehya Abouelnaga|arXiv (Cornell University)|Jul 13, 2016

Crime Patterns and Interventions17 citations

TL;DR

This paper presents a machine learning approach to predict crime categories in San Francisco using temporal, spatial, and categorical features. By leveraging PCA, feature engineering (especially on address and hour), and ensemble methods like Random Forest, the author achieved a top-18% ranking on the Kaggle competition with a log-loss of 2.39031.

ABSTRACT

San Francisco Crime Classification is an online competition administered by Kaggle Inc. The competition aims at predicting the future crimes based on a given set of geographical and time-based features. In this paper, I achieved a an accuracy that ranks at top %18, as of May 19th, 2016. I will explore the data, and explain in details the tools I used to achieve that result.

Motivation & Objective

• To predict crime categories in San Francisco using geospatial and temporal features from public crime data.
• To improve classification performance through advanced feature engineering, particularly on sparse address fields.
• To evaluate and compare multiple machine learning models, including XGBoost, k-NN, decision trees, and Random Forest.
• To achieve a high-ranking result in the Kaggle San Francisco Crime Classification competition using robust model tuning and dimensionality reduction.
• To explore future directions such as neural networks and classifier fusion for further performance gains.

Proposed method

• Extracted key temporal features (hour, day of week, month) from the 'Dates' field to enhance temporal pattern detection.
• Engineered new features from the 'Address' field, including 'StreetNo' and 'Block' indicators, to improve model generalization on sparse address data.
• Applied Principal Component Analysis (PCA) with 3 components to reduce dimensionality and mitigate overfitting.
• Evaluated multiple classifiers: k-Nearest Neighbors, XGBoost, Decision Trees, Bayesian, and Random Forest, using log-loss as the evaluation metric.
• Optimized hyperparameters via grid search, particularly tuning max_depth and n_estimators in Random Forest for optimal performance.
• Used classifier fusion and explored advanced techniques like Learning by Counting and neural networks for future improvements.

Experimental results

Research questions

• RQ1 Which temporal and spatial features most strongly correlate with crime category in San Francisco?
• RQ2 How effective is PCA in reducing dimensionality while preserving predictive power for crime classification?
• RQ3 Which machine learning model—k-NN, XGBoost, Decision Trees, or Random Forest—achieves the best log-loss performance on this dataset?
• RQ4 To what extent does feature engineering on the sparse 'Address' field improve classification accuracy?
• RQ5 Can neural networks or ensemble methods further improve performance beyond traditional tree-based models?

Key findings

• The hour of day emerged as the most predictive feature, showing the highest correlation with crime category.
• Random Forest with max_depth=13 and n_estimators=200 achieved the best validation log-loss of 2.41057.
• Adding StreetNo and Block features to the Random Forest model reduced log-loss to 2.366175731, significantly improving performance.
• The final model achieved a test log-loss of 2.39031, placing the author in the top 18% of the Kaggle competition (388 out of 2077 submissions).
• Decision Trees performed surprisingly well, with optimal performance at max_depth=10 and n_elements=256, yielding a log-loss of 2.50849507.
• XGBoost and Bayesian classifiers underperformed compared to ensemble methods, with XGBoost achieving a log-loss of 2.57428 at max_depth=5.