[Paper Review] Learning Motion in Feature Space: Locally-Consistent Deformable Convolution Networks for Fine-Grained Action Detection

Khoi-Nguyen C. Mac, Dhiraj Joshi+4 more|arXiv (Cornell University)|Nov 21, 2018

Human Pose and Action Recognition29 references4 citations

TL;DR

This paper proposes Locally-Consistent Deformable Convolution (LCDC), a novel single-stream network that models fine-grained motion by learning temporal dynamics in feature space through adaptive receptive field changes and enforcing local coherency. LCDC achieves state-of-the-art performance on 50 Salads (F1: 80.22%) and GTEA (F1: 75.39%) with significantly fewer parameters than optical flow-based baselines, outperforming both two-stream and standard deformable convolution methods by jointly learning spatio-temporal features without relying on pixel-level motion estimation.

ABSTRACT

Fine-grained action detection is an important task with numerous applications in robotics and human-computer interaction. Existing methods typically utilize a two-stage approach including extraction of local spatio-temporal features followed by temporal modeling to capture long-term dependencies. While most recent papers have focused on the latter (long-temporal modeling), here, we focus on producing features capable of modeling fine-grained motion more efficiently. We propose a novel locally-consistent deformable convolution, which utilizes the change in receptive fields and enforces a local coherency constraint to capture motion information effectively. Our model jointly learns spatio-temporal features (instead of using independent spatial and temporal streams). The temporal component is learned from the feature space instead of pixel space, e.g. optical flow. The produced features can be flexibly used in conjunction with other long-temporal modeling networks, e.g. ST-CNN, DilatedTCN, and ED-TCN. Overall, our proposed approach robustly outperforms the original long-temporal models on two fine-grained action datasets: 50 Salads and GTEA, achieving F1 scores of 80.22% and 75.39% respectively.

Motivation & Objective

• To address the challenge of fine-grained action detection, where subtle motion differences make actions hard to distinguish even for humans.
• To improve short-term spatio-temporal feature extraction by modeling motion directly in feature space, avoiding reliance on computationally expensive and noisy optical flow.
• To develop a compact, single-stream backbone network that jointly learns spatial and temporal features using adaptive receptive fields with local coherency constraints.
• To create a flexible feature extractor that enhances performance when combined with existing long-temporal modeling networks like ST-CNN, DilatedTCN, and ED-TCN.

Proposed method

• Proposes Locally-Consistent Deformable Convolution (LCDC), an extension of standard deformable convolution that models motion via changes in adaptive receptive fields across consecutive frames.
• Introduces a local coherency constraint to enforce consistency in motion vectors across neighboring spatial locations, reducing redundant parameters and improving robustness.
• Uses feature space motion representation instead of pixel-level optical flow, computing motion energy as aggregated motion vectors across all deformable convolution layers.
• Leverages deformable convolution offsets to dynamically adjust receptive fields, focusing on salient regions and capturing motion patterns more efficiently than fixed kernels.
• Designs a single-stream network that jointly learns spatio-temporal features without decoupling spatial and temporal streams, enabling end-to-end training.
• Employs a parameter-efficient design where the number of deformation parameters is reduced by a factor of 36 compared to standard deformable convolutions, due to the coherency constraint.

Experimental results

Research questions

• RQ1 Can motion be effectively modeled in feature space using changes in adaptive receptive fields, rather than relying on optical flow or pixel-level motion estimation?
• RQ2 Does enforcing local coherency in deformable convolution offsets improve motion modeling robustness and reduce parameter complexity?
• RQ3 How does a single-stream, end-to-end network that jointly learns spatio-temporal features compare to two-stream or two-stage approaches in fine-grained action detection?
• RQ4 To what extent can LCDC features improve performance when integrated into existing long-temporal modeling networks like ST-CNN or TCN?
• RQ5 What is the impact of the local coherency constraint on model efficiency and accuracy in low-motion or high-similarity action scenarios?

Key findings

• LCDC achieves an F1 score of 80.22% on the 50 Salads dataset, outperforming the original ST-CNN baseline and optical flow-based two-stream models.
• On the GTEA dataset, LCDC achieves an F1 score of 75.39%, demonstrating strong generalization across diverse egocentric action datasets.
• LCDC reduces the number of parameters for deformable convolutions by a factor of 36 compared to standard deformable convolution, due to the local coherency constraint.
• The model outperforms the two-stream baseline (TwoStreamNet) in accuracy while using significantly fewer parameters, proving its efficiency and effectiveness.
• Ablation studies confirm that the local coherency constraint is essential for performance, as removing it leads to a drop in accuracy from 73.77% to 72.25%.
• LCDC features are robust and effective even without optical flow, outperforming appearance-only baselines and motion streams based on optical flow on both datasets.