[Paper Review] A Preliminary Study of the Intrinsic Relationship between Complexity and Alignment

Yingxiu Zhao, Bowen Yu+5 more|arXiv (Cornell University)|Aug 10, 2023

Topic Modeling4 citations

TL;DR

This paper proposes Tree-Instruct, a method to systematically increase instruction data complexity by adding nodes to semantic trees, enabling controlled study of complexity's impact on LLM alignment. Results show that increasing complexity consistently boosts performance—e.g., a 24% win rate gain with 1,000 instructions and 10 added nodes—outperforming both diverse simple data and curriculum learning strategies under fixed token budgets.

ABSTRACT

Training large language models (LLMs) with open-domain instruction data has yielded remarkable success in aligning to end tasks and human preferences. Extensive research has highlighted the importance of the quality and diversity of instruction data. However, the impact of data complexity, as a crucial metric, remains relatively unexplored from three aspects: (1)where the sustainability of performance improvements with increasing complexity is uncertain; (2)whether the improvement brought by complexity merely comes from introducing more training tokens; and (3)where the potential benefits of incorporating instructions from easy to difficult are not yet fully understood. In this paper, we propose Tree-Instruct to systematically enhance the instruction complexity in a controllable manner. By adding a specified number of nodes to instructions' semantic trees, this approach not only yields new instruction data from the modified tree but also allows us to control the difficulty level of modified instructions. Our preliminary experiments reveal the following insights: (1)Increasing complexity consistently leads to sustained performance improvements of LLMs. (2)Under the same token budget, a few complex instructions outperform diverse yet simple instructions. (3)Curriculum instruction tuning might not yield the anticipated results; focusing on increasing complexity appears to be the key.

Motivation & Objective

• To investigate the intrinsic relationship between instruction data complexity and LLM alignment performance.
• To determine whether performance gains from complexity stem from increased token count or inherent structural complexity.
• To evaluate the effectiveness of curriculum learning in instruction tuning, progressing from simple to complex instructions.
• To develop a method that increases complexity without introducing unwanted diversity or altering original instruction intent.

Proposed method

• Tree-Instruct generates complex instruction data by prompting LLMs to insert a specified number of semantic tree nodes into existing instruction trees.
• The number of added nodes serves as a controlled measure of complexity, preserving the original instruction's intent while increasing structural depth.
• Semantic trees are used instead of text sequences to avoid sequential ordering effects and enable precise complexity control.
• The method enables systematic comparison across complexity levels while holding other factors like data diversity and token count constant.
• Models are fine-tuned on instruction data with varying node counts (3, 6, 10), and performance is evaluated using AlpacaEval.
• Baselines include diverse simple data (Alpaca-4K), WizardLM’s evolved prompts, and curriculum learning setups with varying training orders.

Figure 1 : The scaling law of instruction complexity. We experiment with enhancing the complexity of semantic trees for 1,000 Alpaca instructions by adding extra 3, 6, and 10 nodes. We then evaluate models fine-tuned on instruction data of varying complexities against $\mathtt{text}$ - $\mathtt{davi

Experimental results

Research questions

• RQ1 Does increasing instruction data complexity lead to sustained performance improvements in LLM alignment, and if so, what is the scaling behavior?
• RQ2 To what extent do performance gains from complexity stem from increased token count rather than structural complexity?
• RQ3 Does curriculum learning—training from easy to hard instructions—improve LLM alignment compared to mixed or hard-first training?
• RQ4 Can a few complex instructions outperform a larger set of diverse but simple instructions under the same token budget?

Key findings

• Increasing complexity via Tree-Instruct leads to sustained performance gains: a 24% win rate increase was observed when using 1,000 instructions with 10 added nodes.
• With the same token budget, a small set of complex instructions (Tree-10-Nodes) outperformed a larger set of diverse simple instructions (Alpaca-4K), with a 22% performance gap in win rate.
• Curriculum learning from easy to hard did not yield significant gains; the performance improvement was minimal compared to training on the simplest data alone.
• The model trained on the most complex data (Tree-10-Nodes) achieved a 50.19% win rate on AlpacaEval, significantly outperforming the baseline Alpaca-1K (26.40%) and WizardLM (40.37%).
• The performance gain from complexity is partly due to increased tokens, but the core benefit lies in structural complexity, as complex data outperforms simple data even under equal token constraints.
• Modern LLMs with large parameter counts appear capable of learning directly from complex samples, reducing the need for foundational exposure to simpler examples.

Figure 2 : The instruction generated by different evolving methods: Tree-instruction after adding ten nodes and WizardLM by iteratively deepening three times. We also demonstrate how Tree-Instruct enhances the complexity of the original instruction’s semantic tree by introducing three nodes (orange)