[Paper Review] Let's Think Dot by Dot: Hidden Computation in Transformer Language Models

Jacob Pfau, William Merrill+1 more|arXiv (Cornell University)|Apr 24, 2024

Natural Language Processing Techniques4 citations

TL;DR

This paper demonstrates that transformers can perform complex algorithmic computations using meaningless filler tokens (e.g., '……') instead of chain-of-thought reasoning, achieving 100% accuracy on synthetic tasks like 3SUM where direct prompting fails. The results show that intermediate tokens provide computational benefits independent of their semantic content, revealing hidden, unauditable computation in LLMs.

ABSTRACT

Chain-of-thought responses from language models improve performance across most benchmarks. However, it remains unclear to what extent these performance gains can be attributed to human-like task decomposition or simply the greater computation that additional tokens allow. We show that transformers can use meaningless filler tokens (e.g., '......') in place of a chain of thought to solve two hard algorithmic tasks they could not solve when responding without intermediate tokens. However, we find empirically that learning to use filler tokens is difficult and requires specific, dense supervision to converge. We also provide a theoretical characterization of the class of problems where filler tokens are useful in terms of the quantifier depth of a first-order formula. For problems satisfying this characterization, chain-of-thought tokens need not provide information about the intermediate computational steps involved in multi-token computations. In summary, our results show that additional tokens can provide computational benefits independent of token choice. The fact that intermediate tokens can act as filler tokens raises concerns about large language models engaging in unauditable, hidden computations that are increasingly detached from the observed chain-of-thought tokens.

Motivation & Objective

• To investigate whether intermediate tokens in transformers provide computational power beyond semantic content.
• To determine if meaningless filler tokens (e.g., '……') can enable transformers to solve algorithmic tasks they otherwise cannot.
• To assess whether the performance gains from chain-of-thought prompting stem from structural computation or from interpretable reasoning.
• To evaluate the feasibility of learning to use filler tokens through supervised fine-tuning.
• To theoretically characterize the class of problems where filler tokens are useful, based on quantifier depth in first-order logic.

Proposed method

• The authors construct two synthetic datasets: 3SUM and 2SUM-Transform, designed to be unsolvable by direct prompting but solvable with intermediate tokens.
• Models are fine-tuned on sequences with filler tokens ('……') between input and output, using next-token prediction objective.
• Performance is evaluated on zero-shot generalization to longer and more complex inputs, comparing filler, chain-of-thought, and no-intermediate-token baselines.
• Theoretical analysis links filler token utility to quantifier depth in first-order logic, showing that problems with quantifier depth >2 become expressible with filler tokens.
• Ablation studies compare training on filler-only sequences versus chain-of-thought sequences to isolate learning difficulties.
• Empirical evaluation uses LLaMA-based models fine-tuned on synthetic data, with performance measured across multiple random seeds and hyperparameter settings.

Figure 1: A stylized example distinguishing between three LM question-answering protocols: chain of thought, filler tokens, and immediate answer. In the filler-tokens setting, the LM uses arbitrary, irrelevant intermediate tokens (e.g., ‘……’) before answering, but the filler tokens’ hidden-layer rep

Experimental results

Research questions

• RQ1 Can transformers solve algorithmic tasks using only filler tokens, without any semantic content in intermediate tokens?
• RQ2 To what extent do performance gains from chain-of-thought prompting stem from additional computation rather than faithful reasoning?
• RQ3 Is it possible to train transformers to use filler tokens effectively, and what kind of supervision is required?
• RQ4 Which classes of problems become solvable with filler tokens, and how does this relate to logical expressivity?
• RQ5 Does the use of filler tokens undermine the interpretability and auditability of LLM reasoning?

Key findings

• On the 3SUM task, models achieved 100% accuracy with filler tokens, compared to near-zero accuracy in the no-filler, immediate-answer setting.
• On the 2SUM-Transform task, models reached 94% accuracy with filler tokens, approaching 95.1% performance with chain-of-thought prompting.
• Performance with filler tokens increased significantly as input length and complexity grew, indicating a computational advantage beyond semantic content.
• Training on filler-only sequences yielded only ~71% accuracy, suggesting that learning to use filler tokens is difficult without parallelizable supervision.
• Models trained on instance-adaptive chain-of-thought data failed to generalize to filler-token sequences, indicating no transfer of knowledge from serial reasoning to filler use.
• Theoretical analysis shows that filler tokens extend expressive power within the TC⁰ complexity class, particularly for problems with high quantifier depth.

Figure 2: The performance gap between a transformer, Llama 34M, with and without filler tokens increases with 3SUM problem length up to length 12, showing that filler tokens reliably provide an advantage for sufficiently complex 3SUM problems. The no-filler models were trained for 5 $\times$ the num