[Paper Review] Analyzing Hypersensitive AI: Instability in Corporate-Scale Machine Learning

Michaela Regneri, Malte Hoffmann+4 more|arXiv (Cornell University)|Jul 17, 2018

Topic Modeling14 references4 citations

TL;DR

This paper proposes a sensitivity analysis framework to measure instability in corporate-scale geometric machine learning models, particularly word2vec, by combining topological and output consistency metrics. It identifies that hierarchical softmax (HS) reduces variance, negative sampling increases sensitivity, and individual data points—especially those with rare or surprising associations—can drastically alter model outputs, enabling detection of high-information data points.

ABSTRACT

Predictive geometric models deliver excellent results for many Machine Learning use cases. Despite their undoubted performance, neural predictive algorithms can show unexpected degrees of instability and variance, particularly when applied to large datasets. We present an approach to measure changes in geometric models with respect to both output consistency and topological stability. Considering the example of a recommender system using word2vec, we analyze the influence of single data points, approximation methods and parameter settings. Our findings can help to stabilize models where needed and to detect differences in informational value of data points on a large scale.

Motivation & Objective

• To address the growing challenge of model instability in large-scale industrial machine learning systems where reliability and reproducibility are critical.
• To disentangle sources of instability in geometric models, particularly word2vec, into those from input data changes and algorithmic approximation methods.
• To develop scalable, application-oriented metrics that quantify both topological changes in embeddings and output consistency under parameter or data perturbations.
• To identify which data points contribute most to model instability, enabling better data quality assessment and model stabilization.
• To establish a foundation for predicting and measuring the business value of data points based on their impact on model stability and performance.

Proposed method

• The authors apply a hybrid recommender system using word2vec embeddings trained on e-commerce clickstream data to model item-to-item recommendations.
• They use topological data analysis (TDA) metrics such as number of clusters, noise points, and cluster purity to assess structural changes in the embedding space.
• Output consistency is measured via overlap of top-10 recommendations between models trained with and without a single data point or with different hyperparameters.
• A leave-one-out sensitivity analysis is conducted to isolate the impact of individual data points on model output and topology.
• The study compares two approximation methods—Hierarchical Softmax (HS) and Negative Sampling—under controlled conditions to evaluate their effect on stability.
• Randomness in word2vec is eliminated by fixing the Huffman tree and random seed, enabling reproducible comparisons across runs.

Experimental results

Research questions

• RQ1 How do different approximation methods (e.g., Hierarchical Softmax vs. Negative Sampling) affect the stability of word2vec embeddings?
• RQ2 To what extent do individual data points—especially rare or semantically complex ones—alter the final model structure and output?
• RQ3 How do topological properties of the embedding space (e.g., number of clusters, cluster purity) correlate with output consistency and model sensitivity?
• RQ4 Can we distinguish between instability caused by input data variation and that caused by algorithmic approximation strategies?
• RQ5 What is the relationship between data point information density and its influence on model stability and output quality?

Key findings

• Fixing the Huffman tree in Hierarchical Softmax (HS) eliminates variance in word2vec, enabling fully reproducible results on identical input data.
• Negative Sampling leads to higher sensitivity to data changes, while HS maintains greater output and topological stability.
• The number of clusters in the embedding space varies significantly between models, with larger changes observed when both Hamming distance and Huffman code changes are high.
• The average number of noise points remains nearly constant at 143,416 ± 99 across models, indicating a stable baseline for topological analysis.
• Cluster purity averages 0.910 ± 0.002 across all embeddings, with higher purity observed in models with fewer, more coherent clusters.
• Local embedding density correlates negatively with recommendation overlap: denser regions allow more variability in recommendations without loss of quality.