[Paper Review] Enriching Location Representation with Detailed Semantic Information

Muehlburger, Herbert, Wotawa, Franz|arXiv (Cornell University)|Jan 1, 2024

Natural Language Processing Techniques329 citations

TL;DR

Llama 3 is a new herd of multilingual foundation language models, including 8B, 70B, and 405B parameter variants, trained on 15T multilingual tokens with 3.8×10²⁵ FLOPs. It achieves state-of-the-art performance on benchmarks like MMLU, GSM8K, and HumanEval, matching or exceeding GPT-4 and other leading models, and is publicly released under the Llama 3 Community License.

ABSTRACT

Cyber-physical systems (CPS) are critical to modern infrastructure, but are vulnerable to faults and anomalies that threaten their operational safety. In this work, we evaluate the use of open-source Large Language Models (LLMs), such as Mistral 7B, Llama3.1:8b-instruct-fp16, and others to detect anomalies in two distinct datasets: battery management and powertrain systems. Our methodology utilises retrieval-augmented generation (RAG) techniques, incorporating a novel two-step process where LLMs first infer operational rules from normal behavior before applying these rules for fault detection. During the experiments, we found that the original prompt design yielded strong results for the battery dataset but required modification for the powertrain dataset to improve performance. The adjusted prompt, which emphasises rule inference, significantly improved anomaly detection for the powertrain dataset. Experimental results show that models like Mistral 7B achieved F1-scores up to 0.99, while Llama3.1:8b-instruct-fp16 and Gemma 2 reached perfect F1-scores of 1.0 in complex scenarios. These findings demonstrate the impact of effective prompt design and rule inference in improving LLM-based fault detection for CPS, contributing to increased operational resilience.

Motivation & Objective

• To develop a new generation of high-quality, multilingual foundation models that match or exceed state-of-the-art performance on diverse NLP tasks.
• To improve model quality through enhanced data curation, increased scale (15T tokens, 3.8×10²⁵ FLOPs), and better training procedures.
• To enable strong capabilities in reasoning, coding, tool use, and long-context understanding across multiple model sizes.
• To ensure safety and alignment through post-training with supervised fine-tuning, rejection sampling, and direct preference optimization.
• To release the largest model (405B parameters) publicly to accelerate research and responsible AI development.

Proposed method

• Pre-training Llama 3 on a 15T-token multilingual corpus using a dense Transformer architecture with up to 128K context length.
• Applying rigorous data curation and filtering pipelines to improve data quality during pre-training and post-training stages.
• Scaling the largest model to 405B parameters and 3.8×10²⁵ FLOPs, with longer training on smaller models to exceed compute-optimal performance.
• Using a simplified post-training pipeline combining supervised fine-tuning (SFT), rejection sampling (RS), and direct preference optimization (DPO) for alignment.
• Integrating multimodal capabilities (image, video, speech) via a compositional approach, though these are not yet publicly released.
• Leveraging the 405B model to distill quality improvements into smaller models during post-training for better performance at lower inference cost.

Experimental results

Research questions

• RQ1 Can a large-scale, multilingual foundation model trained on high-quality, curated data achieve performance on par with leading models like GPT-4?
• RQ2 How does increased model scale and improved data quality affect performance across reasoning, coding, and multilingual understanding tasks?
• RQ3 Can a simpler post-training pipeline (SFT + RS + DPO) achieve strong alignment and helpfulness without complex reinforcement learning?
• RQ4 To what extent can a large 405B-parameter model improve the performance of smaller models (8B, 70B) through distillation during post-training?
• RQ5 How well does a compositional approach to multimodal integration perform on image, video, and speech recognition benchmarks compared to SOTA?

Key findings

• The Llama 3 405B model achieves 87.3% on MMLU (5-shot) and 88.6% on IFEval, outperforming GPT-3.5 Turbo and Gemma 2 9B, and approaching GPT-4 performance.
• On HumanEval (0-shot), Llama 3 405B achieves 89.0% accuracy, surpassing GPT-3.5 Turbo (68.0%) and close to GPT-4 (86.6%) and GPT-4o (90.2%).
• On GSM8K (8-shot, CoT), Llama 3 405B reaches 96.8% accuracy, exceeding GPT-3.5 Turbo (81.6%) and nearing the SOTA of 96.4% (Claude 3.5 Sonnet).
• The Llama 3 70B model achieves 86.0% on MMLU (0-shot, CoT), outperforming Mistral 7B (60.5%) and Mixtral 8x22B (79.9%), demonstrating strong reasoning and knowledge retention.
• In long-context tasks, Llama 3 405B achieves 95.2% on ZeroSCROLLS/QuALITY and 98.1% on NIH/Multi-needle, indicating strong performance on long-context reasoning and retrieval.
• The Llama 3 405B model achieves 91.6% on MGSM (0-shot, CoT), significantly outperforming models like Mistral 7B (29.9%) and Gemma 2 9B (53.2%), showing strong multilingual generalization.