[論文レビュー] Interpretable Predictability-Based AI Text Detection: A Replication Study

Adam Skurla, Dominik Macko+1 more|arXiv (Cornell University)|Mar 16, 2026

Authorship Attribution and Profiling被引用数 0

ひとこと要約

The paper replicates and extends the AuTexTification 2023 system for AI-text attribution, tests newer multilingual models, adds 26 document-level stylometric features, and uses SHAP for interpretability, achieving comparable or better results with a unified multilingual configuration.

ABSTRACT

This paper replicates and extends the system used in the AuTexTification 2023 shared task for authorship attribution of machine-generated texts. First, we tried to reproduce the original results. Exact replication was not possible because of differences in data splits, model availability, and implementation details. Next, we tested newer multilingual language models and added 26 document-level stylometric features. We also applied SHAP analysis to examine which features influence the model's decisions. We replaced the original GPT-2 models with newer generative models such as Qwen and mGPT for computing probabilistic features. For contextual representations, we used mDeBERTa-v3-base and applied the same configuration to both English and Spanish. This allowed us to use one shared configuration for Subtask 1 and Subtask 2. Our experiments show that the additional stylometric features improve performance in both tasks and both languages. The multilingual configuration achieves the results that are comparable to or better than language-specific models. The study also shows that clear documentation is important for reliable replication and fair comparison of systems.

研究の動機と目的

• Aim to faithfully replicate the AuTexTification 2023 system and identify factors affecting reproducibility.
• Evaluate the impact of newer multilingual language models on both embedding- and predictability-based features.
• Extend the stylometric feature set with additional document-level indicators and assess their contribution.
• Use SHAP analysis to interpret feature importance and decision signals in both languages.
• Propose a unified multilingual configuration that performs well across tasks without language-specific adjustments.

提案手法

• Replicate the original system configuration, feature set, and training procedure using the AuTexTification 2023 dataset and official train–test splits.
• Replace base language models used for probabilistic features and contextual representations with newer multilingual model groups (XGLM, mGPT, Large) and multilingual encoders (XLM-R, mDeBERTa-v3) to test cross-language applicability.
• Extend the feature set with 26 document-level stylometric features and evaluate their impact on performance and interpretability.
• Apply SHAP analysis to identify which features most influence model decisions in LingRF and LingRF+PredOut configurations.
• Compare multilingual unified configurations against language-specific baselines to assess cross-language generalization and robustness.
• Conduct experiments on the AuTexTification 2023 data (English and Spanish) across Subtasks 1 (binary) and 2 (model attribution), with fixed splits and 20-epoch training where applicable.

Figure 1: Overview of the architecture proposed by Przybyla et al. ( 2023 ) . The upper part shows the evaluated model configurations, while the lower part illustrates the feature extraction pipeline used to derive token-level probabilistic features and document-level linguistic features.

実験結果

リサーチクエスチョン

• RQ1 RQ1: To what extent can the original AuTexTification 2023 system be faithfully reproduced?
• RQ2 RQ2: How does the choice of base language models influence performance and can a unified multilingual configuration achieve comparable results across languages?
• RQ3 RQ3: Do the added stylometric features improve classification performance and model interpretability?

主な発見

• Extended stylometric features consistently improve performance across languages and subtasks.
• A unified multilingual configuration (using mDeBERTa-v3 as encoder and Large probabilistic model group) can match or slightly outperform language-specific baselines in many settings.
• SHAP analysis reveals that several newly added stylometric features are among the most important predictors and complement probabilistic signals when combined with them.
• Replacing older models with newer multilingual encoders and probabilistic models enables a single configuration usable for both languages and subtasks without language-specific tuning.
• Replication highlights that small implementation details (data splits, early stopping, feature extraction nuances) substantially affect reproducibility, underscoring the need for full code release and precise methodological documentation.
• Compared to the state-of-the-art detector mdok in Appendix C.3, the approach can be competitive in binary subtask while maintaining interpretability.

Figure 2: SHAP Summary for Subtask 1 (English) - LingRF Style, Class 1