[Paper Review] Convolutional Collaborative Filter Network for Video Based Recommendation Systems

Cheng-Kang Hsieh, Miguel Del Campo+4 more|arXiv (Cornell University)|Oct 18, 2018

Recommender Systems and Techniques7 references4 citations

TL;DR

This paper proposes a Convolutional Collaborative Filter Network (CCFN) that models temporal sequences of objects in movie trailers using 1D convolutional layers over frame-level features to improve video-based recommendation performance. By capturing dynamic visual patterns like shot duration and repetition, the model outperforms average-pooling baselines and enables model explainability through activation analysis of learned object-sequences across genres.

ABSTRACT

This analysis explores the temporal sequencing of objects in a movie trailer. Temporal sequencing of objects in a movie trailer (e.g., a long shot of an object vs intermittent short shots) can convey information about the type of movie, plot of the movie, role of the main characters, and the filmmakers cinematographic choices. When combined with historical customer data, sequencing analysis can be used to improve predictions of customer behavior. E.g., a customer buys tickets to a new movie and maybe the customer has seen movies in the past that contained similar sequences. To explore object sequencing in movie trailers, we propose a video convolutional network to capture actions and scenes that are predictive of customers' preferences. The model learns the specific nature of sequences for different types of objects (e.g., cars vs faces), and the role of sequences in predicting customer future behavior. We show how such a temporal-aware model outperforms simple feature pooling methods proposed in our previous works and, importantly, demonstrate the additional model explain-ability allowed by such a model.

Motivation & Objective

• To improve cold-start movie recommendation by modeling temporal dynamics in movie trailers beyond static frame features.
• To address the limitation of average pooling in video representation, which discards sequential and temporal information.
• To enhance model explainability by identifying which visual sequences (e.g., close-ups, rapid cuts) drive predictions.
• To integrate video-level temporal patterns with collaborative filtering for better user behavior prediction.
• To demonstrate that sequence-aware video representations are more predictive than frame-level pooling in hybrid recommendation systems.

Proposed method

• The model uses a pre-trained image feature extractor to generate dense embeddings for each video frame in a trailer.
• These frame-level features are stacked over time and processed through multiple 1D convolutional layers to learn temporal patterns in object presence and transitions.
• The convolutional layers apply filters across consecutive frames to detect recurring visual sequences (e.g., rapid cuts, sustained close-ups) that signal genre or narrative style.
• A residual connection is used in the first convolutional layer to improve gradient flow and feature learning, especially with 1-frame filters.
• The final feature map is globally average pooled and fed into a collaborative filtering layer to predict user preferences based on historical behavior.
• The model is trained end-to-end using a combination of video representation learning and collaborative filtering loss.

Experimental results

Research questions

• RQ1 Can modeling temporal sequences of objects in movie trailers improve the accuracy of video-based recommendation systems?
• RQ2 How do different convolutional filter sizes and network architectures affect the performance of sequence-aware video representation in recommendation tasks?
• RQ3 To what extent can learned convolutional filters provide interpretable insights into the visual patterns that predict user preferences?
• RQ4 How does the proposed method compare to average pooling of frame features in terms of predictive power and explainability?
• RQ5 Which types of visual sequences (e.g., shot duration, repetition) are most predictive of user behavior across different genres?

Key findings

• A sufficiently large receptive field (filter size > 4 frames) is crucial for model performance, with performance saturating at filter sizes above 8 frames.
• The residual connection significantly improves performance, particularly when using 1-frame convolutional filters, suggesting it enhances inter-frame correlation learning.
• The model successfully learns genre-specific visual templates: for example, Channel 4 activates on slow, intense close-ups typical of horror films, while Channel 8 responds to high-energy stunts common in action or superhero movies.
• The model outperforms simple average pooling baselines by capturing dynamic, sequence-level features that are more informative for predicting user behavior in cold-start scenarios.
• Activation maps reveal that the model learns semantically meaningful object-sequences across diverse genres, including action, animation, romance, horror, monster, and war.
• The explainability provided by filter activations allows researchers to interpret which visual patterns (e.g., shot frequency, object focus) are most predictive of user preferences.