[Paper Review] What comes after transformers? -- A selective survey connecting ideas in deep learning

Johannes Schneider|arXiv (Cornell University)|Aug 1, 2024

Advanced Neural Network Applications4 citations

TL;DR

This paper surveys recent innovations in deep learning beyond Transformers, focusing on alternative architectures like state space models, capsule networks, and novel layer designs. It identifies recurring design patterns—such as multi-X, higher-order layers, data-controlled gating, moving averages, and matrix decomposition—that unify successful advancements and guide future research toward disruptive, non-incremental progress beyond the current Transformer paradigm.

ABSTRACT

Transformers have become the de-facto standard model in artificial intelligence since 2017 despite numerous shortcomings ranging from energy inefficiency to hallucinations. Research has made a lot of progress in improving elements of transformers, and, more generally, deep learning manifesting in many proposals for architectures, layers, optimization objectives, and optimization techniques. For researchers it is difficult to keep track of such developments on a broader level. We provide a comprehensive overview of the many important, recent works in these areas to those who already have a basic understanding of deep learning. Our focus differs from other works, as we target specifically novel, alternative potentially disruptive approaches to transformers as well as successful ideas of recent deep learning. We hope that such a holistic and unified treatment of influential, recent works and novel ideas helps researchers to form new connections between diverse areas of deep learning. We identify and discuss multiple patterns that summarize the key strategies for successful innovations over the last decade as well as works that can be seen as rising stars. Especially, we discuss attempts on how to improve on transformers covering (partially) proven methods such as state space models but also including far-out ideas in deep learning that seem promising despite not achieving state-of-the-art results. We also cover a discussion on recent state-of-the-art models such as OpenAI's GPT series and Meta's LLama models and, Google's Gemini model family.

Motivation & Objective

• To identify and synthesize promising, non-Transformer architectures and components that could succeed or replace Transformers in deep learning.
• To uncover recurring design patterns across recent deep learning advancements that explain their success.
• To provide researchers with a holistic, unified overview of recent breakthroughs in model architectures, layers, and optimization beyond narrow, task-specific surveys.
• To highlight both proven methods (e.g., state space models) and far-out ideas with potential for future impact, even if not yet state-of-the-art.
• To stimulate new research by connecting ideas across NLP, computer vision, and general deep learning through shared design principles.

Proposed method

• The survey synthesizes influential works from 2016 onward, prioritizing those with high usage metrics from Paperswithcode.com and rapid adoption in the research community.
• It categorizes and analyzes key architectural innovations, including alternative attention mechanisms, state space models, and novel layer types such as Mish and GELU.
• The authors identify and formalize five recurring design patterns: multi-X, higher-order layers, data-controlled gating, moving averages, and matrix decomposition.
• The methodology relies on empirical validation via benchmark leaderboards and citation analysis, supplemented by manual verification via Google Scholar and survey literature.
• It contrasts Transformer-based models (e.g., GPT, Llama, Gemini) with emerging alternatives, assessing their theoretical and practical potential.
• The paper avoids coverage of data/data augmentation, model compression, and distributed training to maintain focus on architectural and optimization innovations.

Figure 1 : Overview: Part 1 deals with transformers, Part 2 with deep learning (Lower graph based on [ 103 ] )

Experimental results

Research questions

• RQ1 What are the most promising architectural alternatives to Transformers that could disrupt the current deep learning landscape?
• RQ2 Which recurring design patterns underlie the success of recent deep learning innovations beyond incremental improvements?
• RQ3 How do emerging models like state space models and capsule networks compare to Transformers in terms of efficiency, expressivity, and scalability?
• RQ4 Why do some far-out ideas in deep learning, despite not achieving SOTA, still hold significant potential for future breakthroughs?
• RQ5 To what extent do current deep learning advancements represent incremental progress versus fundamental paradigm shifts?

Key findings

• The paper identifies five key design patterns—multi-X, higher-order layers, data-controlled gating, moving averages, and matrix decomposition—that recurrently underlie successful deep learning innovations.
• State space models are highlighted as a leading alternative to attention mechanisms, offering linear complexity and strong performance in sequence modeling.
• Data-controlled gating mechanisms, such as those in attention, LSTM, GELU, and mixture-of-experts, are shown to be a powerful abstraction for dynamic information flow.
• Despite their dominance, Transformers are found to violate the 'no free lunch' theorem by performing well across diverse tasks, suggesting potential for specialized architectures to outperform them in specific domains.
• The survey reveals a 'winner-takes-all' dynamic in research, where platform-based metrics may overlook innovative but less popular ideas, risking the stagnation of truly disruptive innovation.
• The authors conclude that while recent progress is highly creative, it remains largely incremental, and future breakthroughs will require fundamentally new approaches beyond current architectural trends.

Figure 2 : Transformer with the four basic blocks on top and the encoder and decoder at the bottom Figure from [ 103 ]