[Paper Review] HTLM: Hyper-Text Pre-Training and Prompting of Language Models

Armen Aghajanyan, Dmytro Okhonko+5 more|arXiv (Cornell University)|Jul 14, 2021

Topic Modeling41 references32 citations

TL;DR

HTLM is a hyper-text language model trained on a large web crawl of simplified HTML, enabling structured prompting and auto-prompting; it matches or exceeds text-only LMs on zero-shot prompting and fine-tuning, and achieves state-of-the-art zero-shot summarization.

ABSTRACT

We introduce HTLM, a hyper-text language model trained on a large-scale web crawl. Modeling hyper-text has a number of advantages: (1) it is easily gathered at scale, (2) it provides rich document-level and end-task-adjacent supervision (e.g. class and id attributes often encode document category information), and (3) it allows for new structured prompting that follows the established semantics of HTML (e.g. to do zero-shot summarization by infilling title tags for a webpage that contains the input text). We show that pretraining with a BART-style denoising loss directly on simplified HTML provides highly effective transfer for a wide range of end tasks and supervision levels. HTLM matches or exceeds the performance of comparably sized text-only LMs for zero-shot prompting and fine-tuning for classification benchmarks, while also setting new state-of-the-art performance levels for zero-shot summarization. We also find that hyper-text prompts provide more value to HTLM, in terms of data efficiency, than plain text prompts do for existing LMs, and that HTLM is highly effective at auto-prompting itself, by simply generating the most likely hyper-text formatting for any available training data. We will release all code and models to support future HTLM research.

Motivation & Objective

• Motivate leveraging hyper-text signals (HTML structure) for pretraining to obtain richer document-level supervision.
• Introduce a BART-style denoising objective on simplified HTML (MHTML) with size hints to control output length.
• Demonstrate transfer of HTLM to diverse tasks (summarization, generation, classification) under zero-shot, few-shot, and full fine-tuning.
• Show data efficiency benefits of HTML-based prompts over plain text prompts for existing LMs.
• Exhibit HTLM's auto-prompting capability by generating HTML-formatted prompts from data.

Proposed method

• Pretrain HTLM on 23 TB of simplified HTML data extracted from Common Crawl (MHTML).
• Use a BART-Large–style denoising autoencoder with span masking, augmented with size hints that specify an estimated masked text length.
• Introduce size-hint tokens after each mask to control generated output length during prompting.
• Employ HTML-based prompts and templates (manual and auto-prompted) to perform generation and classification tasks.
• Evaluate zero-shot and few-shot prompting on summarization benchmarks (Gigaword, CNN/DM, XSum, Reddit TIFU, WebNLG, DART) and table-to-text datasets (E2E, WebNLG, DART).
• Compare HTLM against baselines (GPT-2, PEGASUS, RoBERTa, BART, T5) in generation and zero-shot classification, and assess fine-tuning performance on GLUE.

Experimental results

Research questions

• RQ1 Can HTLM’s hyper-text pretraining and prompting outperform text-only LMs on zero-shot and fine-tuning for classification and summarization tasks?
• RQ2 Do HTML-based prompts offer data efficiency advantages over plain-text prompts for existing LMs?
• RQ3 Can HTLM auto-prompt itself by generating effective HTML-formatted prompts from data?
• RQ4 How does the introduction of size hints affect output length control and task performance in prompting?
• RQ5 What is the impact of HTLM on table-to-text and other structured natural language generation tasks?

Key findings

• HTLM matches or exceeds text-only LMs on zero-shot prompting and full fine-tuning for GLUE-style classification benchmarks.
• HTLM achieves new state-of-the-art zero-shot summarization performance, with gains up to 8 ROUGE-1 points.
• Hyper-text prompts provide greater data efficiency than plain-text prompts for existing LMs, comparable to thousands of extra training examples.
• HTLM can auto-prompt by formatting available training data into HTML, often rivaling manually engineered prompts.
• HTLM demonstrates strong transfer across generation and classification tasks and shows improvements in fine-tuning when using prompt-based training.
• HTML-based prompts enable fine-grained control over output length via size hints, improving prompt robustness and task performance.