[Paper Review] Separate Training for Conditional Random Fields Using Co-occurrence Rate Factorization

Zhemin Zhu, Djoerd Hiemstra+2 more|arXiv (Cornell University)|Aug 9, 2010

Topic Modeling11 references3 citations

TL;DR

This paper proposes separate training for Conditional Random Fields (CRFs) using Co-occurrence Rate Factorization (CR-F), a novel method that decomposes the full graphical model into local, tractable factors for independent training. The approach eliminates the label bias problem, reduces training time from weeks to seconds, and achieves performance competitive with standard and piecewise training on linear-chain CRFs.

ABSTRACT

The standard training method of Conditional Random Fields (CRFs) is very slow for large-scale applications. As an alternative, piecewise training divides the full graph into pieces, trains them independently, and combines the learned weights at test time. In this paper, we present \emph{separate} training for undirected models based on the novel Co-occurrence Rate Factorization (CR-F). Separate training is a local training method. In contrast to MEMMs, separate training is unaffected by the label bias problem. Experiments show that separate training (i) is unaffected by the label bias problem; (ii) reduces the training time from weeks to seconds; and (iii) obtains competitive results to the standard and piecewise training on linear-chain CRFs.

Motivation & Objective

• Address the prohibitively slow training time of standard CRF training, especially for large-scale applications.
• Overcome the label bias problem inherent in discriminative models like MEMMs while enabling scalable training.
• Develop a factorization method that enables independent training of local components without sacrificing global consistency.
• Provide a theoretically grounded, general framework for factorizing undirected graphical models using co-occurrence rates.

Proposed method

• Introduce Co-occurrence Rate Factorization (CR-F), a novel factorization technique based on Pointwise Mutual Information (PMI) and the exponential of PMI, defining CR(X₁;…;Xₙ) = P(X₁,…,Xₙ)/(P(X₁)…P(Xₙ)).
• Use CR-F to decompose the joint probability of a Markov Random Field (MRF) into local factors over subsets of variables, ensuring exact factorization via binomial identities.
• Apply CR-F to derive a factorization over maximal cliques and separators, enabling the construction of local potentials that can be trained independently.
• Train each factor separately using local likelihood maximization, avoiding the need for global partition function computation.
• Reconstruct the global model at test time by combining independently trained local factors, maintaining consistency through the factorization structure.
• Ensure exactness by using a global fixed configuration in factor computation, and prove cancellation of non-clique terms via binomial identities.

Experimental results

Research questions

• RQ1 Can a factorization method be designed to enable separate, local training of CRFs without incurring the label bias problem of MEMMs?
• RQ2 Can Co-occurrence Rate Factorization (CR-F) provide a theoretically sound and exact decomposition of undirected graphical models into tractable local components?
• RQ3 Does separate training using CR-F achieve competitive performance compared to standard and piecewise training on linear-chain CRFs?
• RQ4 Can the proposed method drastically reduce training time while maintaining model accuracy?

Key findings

• Separate training using CR-F is unaffected by the label bias problem, unlike MEMMs, due to its undirected, globally conditioned nature.
• Training time is reduced from weeks (standard CRF) to seconds (separate training), demonstrating a radical speedup on large-scale POS tagging tasks.
• On linear-chain CRFs, separate training achieves performance competitive with both standard and piecewise training, as validated on POS tagging and other NLP tasks.
• The CR-F framework enables exact factorization of MRFs and junction tree cliques by ensuring cancellation of non-clique terms through binomial identities.
• The method generalizes to tree-structured graphs and can be extended to loopy graphs by breaking loops and reintroducing conditioned variables.
• The derived local potentials are normalized locally and can be trained independently, enabling scalable and efficient inference.