[Paper Review] Improving Anytime Prediction with Parallel Cascaded Networks and a Temporal-Difference Loss

Michael L. Iuzzolino, Michael C. Mozer+1 more|arXiv (Cornell University)|Feb 19, 2021

Advanced Memory and Neural Computing4 citations

TL;DR

This paper proposes a cascaded ResNet architecture with parallel, stateful updates and propagation delays to enable anytime prediction—producing increasingly accurate outputs over time. It introduces a temporal-difference (TD) loss with λ < 1 that significantly improves the speed-accuracy trade-off, outperforming state-of-the-art anytime prediction methods by leveraging dynamic output trajectories for better inference and robustness.

ABSTRACT

Although deep feedforward neural networks share some characteristics with the primate visual system, a key distinction is their dynamics. Deep nets typically operate in serial stages wherein each layer completes its computation before processing begins in subsequent layers. In contrast, biological systems have cascaded dynamics: information propagates from neurons at all layers in parallel but transmission occurs gradually over time, leading to speed-accuracy trade offs even in feedforward architectures. We explore the consequences of biologically inspired parallel hardware by constructing cascaded ResNets in which each residual block has propagation delays but all blocks update in parallel in a stateful manner. Because information transmitted through skip connections avoids delays, the functional depth of the architecture increases over time, yielding anytime predictions that improve with internal-processing time. We introduce a temporal-difference training loss that achieves a strictly superior speed-accuracy profile over standard losses and enables the cascaded architecture to outperform state-of-the-art anytime-prediction methods. The cascaded architecture has intriguing properties, including: it classifies typical instances more rapidly than atypical instances; it is more robust to both persistent and transient noise than is a conventional ResNet; and its time-varying output trace provides a signal that can be exploited to improve information processing and inference.

Motivation & Objective

• To explore the computational consequences of biologically inspired parallel hardware and transmission delays in deep neural networks.
• To address the limitations of serial, layer-by-layer deep networks by modeling cascaded dynamics akin to primate cortical processing.
• To improve anytime prediction performance through a novel training objective that exploits temporal dynamics.
• To demonstrate that cascaded networks with TD loss outperform existing anytime prediction methods in accuracy and robustness.
• To show that the time-varying output trajectory of cascaded models conveys exploitable information for downstream tasks like OOD detection.

Proposed method

• Constructs a cascaded ResNet by introducing propagation delays into each residual block while enabling all blocks to update in parallel in a stateful manner.
• Uses skip connections to allow faster transmission of early features, enabling functional depth to increase over time and support anytime prediction.
• Applies a temporal-difference (TD) loss with λ < 1 to train the network, optimizing for intermediate predictions rather than just final outputs.
• Employs a time-unrolled, stateful inference process where each block’s output evolves over time based on delayed input from previous blocks.
• Represents the model’s output at each time step as max softmax, entropy, class posterior, or logits for use in downstream tasks like OOD detection.
• Validates the method on CIFAR-10, ImageNet, LSUN, and SVHN, using AUROC and FPR@95%TPR to evaluate OOD detection performance with temporal trajectories.

Experimental results

Research questions

• RQ1 Can cascaded dynamics with parallel updates and propagation delays improve anytime prediction performance compared to standard serial ResNets?
• RQ2 Does a temporal-difference loss with λ < 1 yield better speed-accuracy trade-offs than standard cross-entropy or λ=1 TD losses?
• RQ3 Can the time-varying output trajectory of a cascaded network provide additional information for downstream tasks such as out-of-distribution detection?
• RQ4 How does the cascaded architecture handle noise and atypical inputs compared to conventional ResNets?
• RQ5 To what extent do the dynamic properties of cascaded networks—such as faster classification of prototypical instances—arise from their architecture and training?

Key findings

• The cascaded ResNet with TD(λ) loss achieves a strictly superior speed-accuracy trade-off compared to state-of-the-art anytime prediction methods.
• The model classifies typical (prototypical) instances more rapidly than atypical or outlier instances, reflecting coarse-to-fine semantic processing.
• The cascaded architecture is more robust to both persistent and transient noise than standard ResNets, due to its dynamic, time-evolving inference process.
• The temporal output trajectory of the cascaded model provides a valuable signal for out-of-distribution detection, improving AUROC and reducing FPR@95%TPR compared to using only the final asymptotic output.
• Training with λ < 1 in the TD loss leads to better performance than λ = 1, demonstrating that intermediate supervision improves representation learning and prediction quality.
• The cascaded model’s dynamics enable improved inference beyond the asymptotic output, showing that time-varying outputs can be exploited to enhance information processing.