[论文解读] Adapting Text Embeddings for Causal Inference

Victor Veitch, Dhanya Sridhar+1 more|arXiv (Cornell University)|May 29, 2019

Topic Modeling参考文献 28被引用 51

一句话总结

本文开发了因果充足的文本嵌入（C-BERT 和 Causal ATM），通过将监督语言表示与因果调整结合起来，从观测文本中识别并估计因果效应。经验的半合成实验表明，语言建模与监督学习相比基线方法能提升因果效应估计的准确性。

ABSTRACT

Does adding a theorem to a paper affect its chance of acceptance? Does labeling a post with the author's gender affect the post popularity? This paper develops a method to estimate such causal effects from observational text data, adjusting for confounding features of the text such as the subject or writing quality. We assume that the text suffices for causal adjustment but that, in practice, it is prohibitively high-dimensional. To address this challenge, we develop causally sufficient embeddings, low-dimensional document representations that preserve sufficient information for causal identification and allow for efficient estimation of causal effects. Causally sufficient embeddings combine two ideas. The first is supervised dimensionality reduction: causal adjustment requires only the aspects of text that are predictive of both the treatment and outcome. The second is efficient language modeling: representations of text are designed to dispose of linguistically irrelevant information, and this information is also causally irrelevant. Our method adapts language models (specifically, word embeddings and topic models) to learn document embeddings that are able to predict both treatment and outcome. We study causally sufficient embeddings with semi-synthetic datasets and find that they improve causal estimation over related embedding methods. We illustrate the methods by answering the two motivating questions---the effect of a theorem on paper acceptance and the effect of a gender label on post popularity. Code and data available at https://github.com/vveitch/causal-text-embeddings-tf2}{github.com/vveitch/causal-text-embeddings-tf2

研究动机与目标

• Motivate the problem of estimating causal effects from observational text with confounding features encoded in text.
• Propose causally sufficient embeddings that preserve information needed for causal adjustment while discarding linguistically irrelevant content.
• Develop two concrete embedding approaches (Causal BERT and Causal ATM) to predict treatment and outcome from text embeddings.
• Provide formal validity arguments showing when and why adjusting for the embeddings suffices for causal identification and estimation.
• Evaluate the methods on semi-synthetic experiments and illustrate with motivating applications (paper acceptance and Reddit post scores).

提出的方法

• Define the causal estimation setup with ATT and NDE using text as the confounder and W as the document text.
• Introduce z = f(W) as the causally sufficient low-dimensional embedding capturing information needed for propensity and outcome models.
• Adapt language models to learn embeddings that are predictive of both treatment and outcome (supervised dimensionality reduction).
• Implement Causal BERT by fine-tuning a BERT-based model to produce document embeddings and maps to g(λ) and Q(t, λ) for propensity and outcomes.
• Implement Causal Amortized Topic Model (Causal ATM) by adapting ATM to produce θi embeddings with learned mappings to g(θi) and Q(ti, θi).
• Provide joint training objectives that combine language modeling with prediction of treatment/outcome (and include overlap considerations).
• Present theoretical results (Theorems 3.1 and 3.2) showing that λ(W) suffices for identification and consistent estimation under stated conditions.

实验结果

研究问题

• RQ1 Can text embeddings be learned that are both linguistically meaningful and causally sufficient for adjustment?
• RQ2 Do supervised, language-aware representations (as opposed to unsupervised embeddings) improve causal effect estimation from text?
• RQ3 How do Causal BERT and Causal ATM perform in semi-synthetic settings and real motivating tasks (paper acceptance and Reddit post scores)?
• RQ4 Under what conditions is adjusting for the embedding sufficient to identify and consistently estimate causal effects?

主要发现

• Language modeling improves causal effect estimation compared to non-language-model baselines.
• Supervised representations (C-BERT, Causal ATM) outperform unsupervised or purely predictive baselines in semi-synthetic experiments.
• C-BERT and C-ATM effectively adjust for confounding in text across varying levels of confounding and outcome noise.
• The methods reduce biased treatment effect estimates and provide closer approximations to ground truth in Reddit and PeerRead simulations.
• Applying the methods to motivating examples suggests that much of the apparent treatment effects in text are due to confounding mediated by the text.