[Paper Review] Geometry of symmetric group-based models

Dimitra Kosta, Kaie Kubjas|arXiv (Cornell University)|May 25, 2017

14-3-3 protein interactions20 references3 citations

TL;DR

This paper develops an algebraic geometry framework for symmetric group-based phylogenetic models, using numerical algebraic geometry to analyze maximum likelihood estimation (MLE) and the embedding problem. It demonstrates that for the CFN model on a tripod tree, the MLE may not exist due to parameter blow-up, a result uncovered only via algebraic methods.

ABSTRACT

Phylogenetic models have polynomial parametrization maps. For symmetric group-based models, Matsen studied the polynomial inequalities that characterize the joint probabilities in the image of these parametrizations. We employ this description for maximum likelihood estimation via numerical algebraic geometry. In particular, we explore an example where the maximum likelihood estimate does not exist, which would be difficult to discover without using algebraic methods. We also study the embedding problem for symmetric group-based models, i.e. we identify which mutation matrices are matrix exponentials of rate matrices that are invariant under a group action.

Motivation & Objective

• To characterize the semialgebraic set of joint leaf probabilities in symmetric group-based models using polynomial inequalities.
• To investigate the existence and computation of maximum likelihood estimates (MLE) in phylogenetic models using numerical algebraic geometry.
• To solve the embedding problem by identifying which mutation matrices are matrix exponentials of group-invariant rate matrices.
• To provide a computational framework for detecting non-existent MLEs in phylogenetic models, particularly in non-compact parameter spaces.

Proposed method

• Employed Matsen’s Fourier coordinate description to derive polynomial inequalities that define the biologically meaningful region of leaf probabilities in symmetric group-based models.
• Relaxed strict inequalities to include boundary components and used numerical algebraic geometry tools (PHCpack, HCPack) to compute all complex critical points of the likelihood function.
• Applied Algorithm 1 with homotopy continuation to compute 167 critical points across 44 boundary components, including Zariski closure and lower-dimensional strata.
• Identified the MLE by evaluating log-likelihood values and checking feasibility against relaxed and stricter inequality constraints, including those derived from path-based dependencies.
• Used parametrization via rate matrices and their exponentials to test whether critical points lie in the model’s image, especially when parameters tend to infinity.
• Characterized G-embeddable mutation matrices as those that are symmetric, group-invariant, and matrix exponentials of group-invariant rate matrices, generalizing results for CFN and Kimura 3P models.

Experimental results

Research questions

• RQ1 Under what conditions does the maximum likelihood estimate fail to exist in symmetric group-based models, and how can this be detected algebraically?
• RQ2 How can the semialgebraic description of the joint leaf probabilities be derived using Fourier coordinates and polynomial inequalities?
• RQ3 What is the structure of the critical points of the likelihood function in symmetric group-based models, and how do they relate to the model’s boundary components?
• RQ4 Which symmetric, group-invariant mutation matrices can be expressed as matrix exponentials of group-invariant rate matrices, and how can this be characterized algebraically?

Key findings

• For the CFN model on the tripod tree $K_{1,3}$, the MLE does not exist for the data vector $(100,11,85,55,56,7,75,8)$, as the global maximum is achieved when a parameter tends to infinity.
• The likelihood function has 167 complex critical points across 44 ideals, with 97 real and 49 positive solutions, but none of the top candidates satisfy all required inequalities when $q_{ijk} = 0$.
• The top two likelihood values were $-0.0729$, achieved by two symmetric solutions, but both failed a path-dependent inequality $q_{000}q_{101} - q_{100}q_{001} eq 0$ when some $q_{ijk} = 0$, indicating they are not in the model’s image.
• The seventh critical point, with log-likelihood $-0.0737$, corresponds to a parametrization where one edge time tends to infinity ($ ilde{ heta}^{(e_2)} = (- ilde{ heta}^{(e_2)}, ilde{ heta}^{(e_2)}) o (- ilfty, ilde{ heta}^{(e_2)})$), confirming nonexistence of MLE.
• The Zariski closure of the CFN model on $K_{1,3}$ has degree 92, matching the ML degree computed in Hosten et al., validating the algebraic framework.
• The study reveals that standard numerical optimization may miss nonexistence of MLE, underscoring the necessity of algebraic methods for rigorous analysis in non-compact phylogenetic models.