[Paper Review] Bayesian Optimisation for Machine Translation

Yishu Miao, Ziyu Wang+1 more|arXiv (Cornell University)|Dec 22, 2014

Natural Language Processing Techniques13 references3 citations

TL;DR

This paper introduces Bayesian optimisation (BO) algorithms for minimum error rate training in statistical machine translation, using N-best lists and hypergraph representations to efficiently explore the translation space. The proposed HG-BO and CHG-BO methods achieve faster convergence and higher BLEU scores than MERT and MIRA, with REMBO enabling scalable tuning over high-dimensional sparse features.

ABSTRACT

This paper presents novel Bayesian optimisation algorithms for minimum error rate training of statistical machine translation systems. We explore two classes of algorithms for efficiently exploring the translation space, with the first based on N-best lists and the second based on a hypergraph representation that compactly represents an exponential number of translation options. Our algorithms exhibit faster convergence and are capable of obtaining lower error rates than the existing translation model specific approaches, all within a generic Bayesian optimisation framework. Further more, we also introduce a random embedding algorithm to scale our approach to sparse high dimensional feature sets.

Motivation & Objective

• To develop generic, efficient Bayesian optimisation algorithms for tuning linear model weights in statistical machine translation systems.
• To overcome the limitations of MERT and MIRA by treating the loss function as a black-box, avoiding costly error surface construction.
• To enable scalable hyperparameter tuning for high-dimensional sparse feature sets in SMT using random embeddings.
• To improve training efficiency and translation quality by leveraging Gaussian processes on fixed N-best lists or hypergraphs.
• To demonstrate the broader potential of Bayesian optimisation in natural language processing beyond translation models.

Proposed method

• Apply Bayesian optimisation to tune linear combination weights in SMT systems using a black-box loss function (e.g., BLEU) evaluated on fixed N-best lists or hypergraphs.
• Use Gaussian processes to model the objective function over the weight space, with expected improvement (EI) as the acquisition function to select next evaluation points.
• Iteratively reweight hypergraphs using sampled parameters, decode new translations via Viterbi algorithm, and update the GP with observed BLEU scores.
• Introduce a bounded search space around current weights to constrain exploration and improve stability during optimisation.
• Employ the REMBO algorithm with random embeddings to project high-dimensional sparse feature spaces into a lower-dimensional space for scalable BO.
• Use a two-step coordinate ascent process in HG-REMBO: first optimise default features, then update sparse features using a regularised random projection matrix.

Experimental results

Research questions

• RQ1 Can Bayesian optimisation outperform MERT and MIRA in terms of convergence speed and translation quality for SMT systems?
• RQ2 How does the use of hypergraphs instead of N-best lists affect the stability and performance of BO-based tuning?
• RQ3 Can random embeddings effectively scale Bayesian optimisation to high-dimensional sparse feature spaces in machine translation?
• RQ4 What impact does the size of the bounded search region have on the optimisation performance and generalisation of the final model?
• RQ5 Does the proposed BO framework generalize to other NLP tasks beyond statistical machine translation?

Key findings

• HG-BO achieved a test BLEU score of 29.9 on the fr-en dataset, outperforming MERT (26.8) and MIRA (26.8), with faster convergence in 1.5 hours compared to 4 hours for MERT.
• CHG-BO demonstrated superior stability, reaching a plateau in 5 iterations without the fluctuation seen in NBL-BO and HG-BO, particularly under poor initial weights.
• A search bound size of b=0.1 achieved comparable performance to b=0.01, while b=0.5 led to degraded test performance due to excessive noise in the search space.
• HG-REMBO achieved test BLEU scores of 30.6 on es-en and 28.0 on fr-en, nearly matching MIRA’s 30.6 and 28.3, respectively, despite requiring multiple runs to generate a stable transformation matrix.
• The BO algorithms required only 5 iterations to reach optimal performance, with maximum convergence typically achieved by 7 iterations, suggesting a safe upper bound of 10 iterations.
• The use of random embeddings enabled effective tuning over 207,952 sparse features in fr-en, demonstrating scalability to high-dimensional feature spaces previously intractable for standard BO.