[Paper Review] Functional Transparency for Structured Data: a Game-Theoretic Approach

Guang-He Lee, Wengong Jin+2 more|arXiv (Cornell University)|Feb 26, 2019

Machine Learning in Materials Science4 citations

TL;DR

This paper introduces a game-theoretic framework for achieving functional transparency in neural models trained on structured data, where a flexible predictor is co-trained with local, interpretable witnesses (e.g., decision trees or linear functions) to ensure locally consistent behavior without sacrificing global expressivity. The approach improves model interpretability and stability in graph, sequence, and molecular representation learning tasks, with empirical gains in both fidelity and predictive performance.

ABSTRACT

We provide a new approach to training neural models to exhibit transparency in a well-defined, functional manner. Our approach naturally operates over structured data and tailors the predictor, functionally, towards a chosen family of (local) witnesses. The estimation problem is setup as a co-operative game between an unrestricted predictor such as a neural network, and a set of witnesses chosen from the desired transparent family. The goal of the witnesses is to highlight, locally, how well the predictor conforms to the chosen family of functions, while the predictor is trained to minimize the highlighted discrepancy. We emphasize that the predictor remains globally powerful as it is only encouraged to agree locally with locally adapted witnesses. We analyze the effect of the proposed approach, provide example formulations in the context of deep graph and sequence models, and empirically illustrate the idea in chemical property prediction, temporal modeling, and molecule representation learning.

Motivation & Objective

• To address the lack of interpretability in complex deep learning models, especially for structured data like graphs and sequences.
• To develop a method that ensures local functional consistency (e.g., linearity or substructure-based behavior) while preserving global model capacity.
• To provide a training-time regularization strategy that guides models toward transparent, locally interpretable behavior, rather than relying on post-hoc explanations.
• To enable stable and generalizable local explanations by co-optimizing a neural predictor with interpretable witnesses in a cooperative game setting.

Proposed method

• Formulates model transparency as a cooperative game between a neural predictor and a set of interpretable witnesses from a predefined family (e.g., decision trees, linear functions).
• Uses a loss function that penalizes discrepancies between the predictor and the witness in local neighborhoods, encouraging functional alignment.
• Employs local witness estimation (e.g., decision trees) on neighborhood data to define the transparency constraint, without requiring differentiability.
• Introduces a game coefficient λ to balance predictive accuracy and transparency, with symmetric and asymmetric game variants for training.
• Applies the framework to deep graph networks, sequence models, and variational autoencoders for molecules, using local witness fitting for interpretability.
• Employs deviation loss (e.g., KL divergence or RMSE) to measure local functional alignment and evaluates it during testing.

Experimental results

Research questions

• RQ1 Can a neural model be trained to exhibit locally interpretable behavior—such as linearity or substructure-based decisions—while remaining globally powerful?
• RQ2 How does the game-theoretic co-training of a predictor and interpretable witnesses affect model stability and fidelity of local explanations?
• RQ3 What is the impact of the game coefficient λ on the trade-off between predictive accuracy and functional transparency?
• RQ4 How does the proposed method compare to post-hoc explanation methods in terms of consistency and generalization across local regions?
• RQ5 Can the framework be effectively applied to structured data such as molecules, temporal sequences, and graphs, where interpretability is underexplored?

Key findings

• The Game model achieved better ELBO performance (-21.5) than the Deep model (-21.6) on the ZINC dataset, with significantly lower deviation (3.98 vs. 4.64), indicating improved local functional alignment.
• In temporal modeling, increasing λ improved transparency (lower deviation and TV) at the cost of higher error, but λ=0.1 yielded better accuracy than λ=0, showing regularization benefits.
• The Game model produced stable, generalizable local decision trees with consistent substructure-based splits, while the unregularized model generated unstable, sample-specific trees with poor generalization.
• The asymmetric game variant was more efficient (20.6 sequences/second) than the symmetric variant (14.6 sequences/second), with comparable performance, suggesting practical advantages.
• Witnesses in the molecule representation task primarily selected substructures in side chains, confirming that latent space variation reflects local molecular changes rather than scaffold-level alterations.
• Visualizations showed that the Game model’s linear weights remained stable across regions, while the Deep model relied more on biases, indicating less reliable interpretability.