[Paper Review] MOI-Mixer: Improving MLP-Mixer with Multi Order Interactions in Sequential Recommendation

Hojoon Lee, Dongyoon Hwang+4 more|arXiv (Cornell University)|Aug 17, 2021

Recommender Systems and Techniques38 references4 citations

TL;DR

This paper proposes MOI-Mixer, a sequential recommendation model that enhances MLP-Mixer by introducing a Multi-Order Interaction (MOI) layer to explicitly capture high-order feature interactions. By replacing standard MLP layers with MOI layers that model arbitrary-order interactions while preserving linear complexity, MOI-Mixer achieves performance on par with Transformer-based models—especially in long-sequence settings—without the quadratic computational cost.

ABSTRACT

Successful sequential recommendation systems rely on accurately capturing the user's short-term and long-term interest. Although Transformer-based models achieved state-of-the-art performance in the sequential recommendation task, they generally require quadratic memory and time complexity to the sequence length, making it difficult to extract the long-term interest of users. On the other hand, Multi-Layer Perceptrons (MLP)-based models, renowned for their linear memory and time complexity, have recently shown competitive results compared to Transformer in various tasks. Given the availability of a massive amount of the user's behavior history, the linear memory and time complexity of MLP-based models make them a promising alternative to explore in the sequential recommendation task. To this end, we adopted MLP-based models in sequential recommendation but consistently observed that MLP-based methods obtain lower performance than those of Transformer despite their computational benefits. From experiments, we observed that introducing explicit high-order interactions to MLP layers mitigates such performance gap. In response, we propose the Multi-Order Interaction (MOI) layer, which is capable of expressing an arbitrary order of interactions within the inputs while maintaining the memory and time complexity of the MLP layer. By replacing the MLP layer with the MOI layer, our model was able to achieve comparable performance with Transformer-based models while retaining the MLP-based models' computational benefits.

Motivation & Objective

• To address the performance gap between MLP-based and Transformer-based models in sequential recommendation despite MLPs' computational advantages.
• To investigate whether explicit high-order feature interactions can close the performance gap in sequential recommendation.
• To design a novel layer that captures arbitrary-order interactions while maintaining linear memory and time complexity.
• To evaluate the effectiveness of high-order interactions in channel-mixing operations for long-sequence modeling.

Proposed method

• Proposes the Multi-Order Interaction (MOI) layer, which models arbitrary-order interactions among input features using a tensor decomposition approach inspired by PARAFAC.
• Integrates the MOI layer into the MLP-Mixer architecture, replacing standard MLP layers in both token-mixing and channel-mixing components.
• Employs a parameter-efficient design where interaction order is controlled by a hyperparameter k, allowing flexible modeling of feature interactions.
• Uses Hadamard product and residual connections to preserve gradient flow and stability during training.
• Applies LayerNorm after the Hadamard product in the MOI layer, based on ablation results showing it outperforms other normalization placements.
• Adapts positional embeddings but finds them ineffective due to inherent sequence-order sensitivity of the MOI layer.

Experimental results

Research questions

• RQ1 Can explicit high-order feature interactions close the performance gap between MLP-based and Transformer-based models in sequential recommendation?
• RQ2 Does modeling higher-order interactions in the channel-mixing layer improve recommendation accuracy?
• RQ3 Can a model maintain linear computational complexity while capturing complex interactions comparable to self-attention?
• RQ4 How does the interaction order (k) affect model performance in different components of the architecture?
• RQ5 Is positional embedding necessary in an MLP-based model that inherently respects sequence order?

Key findings

• MOI-Mixer achieves NDCG@10 of 0.4873 on the ML-1m dataset, outperforming baseline MLP-Mixer and matching state-of-the-art Transformer-based models.
• The second-order interaction (k=2) in the channel-mixing layer yields the best performance, with further increases in order degrading results.
• Replacing the MLP layer with a single linear layer in the token-mixer reduces performance by only 0.3% in NDCG@10 but introduces quadratic complexity, limiting scalability.
• The model maintains linear FLOPs and VRAM usage even at sequence lengths up to 1000, unlike BERT4Rec and gMLP, which scale quadratically.
• Normalization after the Hadamard product in the MOI layer yields the best performance, while Layer Scale and L2 normalization show no benefit or even degrade results.
• Positional embeddings do not improve performance, as the MOI layer already captures sequence order through its interaction mechanism.