[Paper Review] Learning to Advertise with Adaptive Exposure via Constrained Two-Level Reinforcement Learning.

Weixun Wang, Junqi Jin+9 more|arXiv (Cornell University)|Sep 10, 2018

Advanced Bandit Algorithms Research21 references3 citations

TL;DR

This paper proposes a constrained two-level reinforcement learning framework for adaptive advertising exposure in e-commerce, decoupling slot allocation and ad selection while enforcing query-level and day-level constraints via a per-state constrained MDP. It achieves higher advertising revenue with improved training stability and speed using a novel constrained hindsight experience replay mechanism, validated on real-world data.

ABSTRACT

For online advertising in e-commerce, the traditional problem is to assign the right ad to the right user on fixed ad slots. In this paper, we investigate the problem of advertising with adaptive exposure, in which the number of ad slots and their locations can dynamically change over time based on their relative scores with recommendation products. In order to maintain user retention and long-term revenue, there are two types of constraints that need to be met in exposure: query-level and day-level constraints. We model this problem as constrained markov decision process with per-state constraint (psCMDP) and propose a constrained two-level reinforcement learning to decouple the original advertising exposure optimization problem into two relatively independent sub-optimization problems. We also propose a constrained hindsight experience replay mechanism to accelerate the policy training process. Experimental results show that our method can improve the advertising revenue while satisfying different levels of constraints under the real-world datasets. Besides, the proposal of constrained hindsight experience replay mechanism can significantly improve the training speed and the stability of policy performance.

Motivation & Objective

• To address the challenge of dynamic ad slot allocation and placement in e-commerce advertising, where slot count and positions adapt based on product scores.
• To maintain user retention and long-term revenue by enforcing two types of constraints: query-level (per search query) and day-level (per day) on ad exposure.
• To model the adaptive advertising exposure problem as a per-state constrained Markov Decision Process (psCMDP) for precise constraint handling.
• To decouple the complex optimization into two independent sub-optimization problems for improved training efficiency and policy stability.
• To accelerate policy training and enhance performance stability through a novel constrained hindsight experience replay mechanism.

Proposed method

• Formulate the adaptive advertising exposure problem as a per-state constrained MDP (psCMDP), where constraints are enforced at the state level.
• Decompose the original problem into two independent sub-optimization tasks: (1) slot selection based on product scores and (2) ad selection per slot.
• Apply constrained two-level reinforcement learning to optimize the two sub-problems separately while maintaining constraint compliance.
• Introduce a constrained hindsight experience replay mechanism that stores and replays past experiences with constraint-aware sampling to improve data efficiency.
• Use a dual-optimization approach to handle constraints, ensuring that policy updates do not violate query-level or day-level exposure limits.
• Leverage off-policy learning with prioritized experience replay, enhanced with constraint validation during replay sampling to stabilize training.

Experimental results

Research questions

• RQ1 Can a two-level reinforcement learning framework effectively decouple slot allocation and ad selection in adaptive advertising exposure?
• RQ2 How well can the proposed method maintain user retention and long-term revenue while satisfying query-level and day-level exposure constraints?
• RQ3 To what extent does the constrained hindsight experience replay mechanism improve training speed and policy performance stability?
• RQ4 How does the method perform in real-world e-commerce datasets compared to baseline approaches under varying constraint levels?
• RQ5 Does the psCMDP formulation enable better constraint adherence and optimization efficiency than standard unconstrained RL methods?

Key findings

• The proposed method improves advertising revenue compared to baseline methods while strictly satisfying both query-level and day-level exposure constraints.
• The constrained two-level RL framework successfully decouples the optimization problem, enabling more stable and efficient training.
• The constrained hindsight experience replay mechanism significantly accelerates policy training and enhances the stability of policy performance.
• Experimental results on real-world e-commerce datasets demonstrate consistent improvements in revenue and constraint compliance across different constraint levels.
• The method maintains high performance even under tight constraint regimes, indicating strong robustness and scalability in real-world deployment scenarios.