[Paper Review] LADDER: A Human-Level Bidding Agent for Large-Scale Real-Time Online Auctions

Yu Wang, Jiayi Liu+6 more|arXiv (Cornell University)|Aug 18, 2017

Auction Theory and Applications20 references13 citations

TL;DR

LADDER is a deep reinforcement learning agent based on DASQN that learns optimal bidding strategies directly from raw textual descriptions of real-time online auctions, achieving human-level performance. It increased JD's ad revenue by over 50% during a major sale while improving advertisers' ROI, outperforming a manually engineered benchmark policy.

ABSTRACT

We present LADDER, the first deep reinforcement learning agent that can successfully learn control policies for large-scale real-world problems directly from raw inputs composed of high-level semantic information. The agent is based on an asynchronous stochastic variant of DQN (Deep Q Network) named DASQN. The inputs of the agent are plain-text descriptions of states of a game of incomplete information, i.e. real-time large scale online auctions, and the rewards are auction profits of very large scale. We apply the agent to an essential portion of JD's online RTB (real-time bidding) advertising business and find that it easily beats the former state-of-the-art bidding policy that had been carefully engineered and calibrated by human experts: during JD.com's June 18th anniversary sale, the agent increased the company's ads revenue from the portion by more than 50%, while the advertisers' ROI (return on investment) also improved significantly.

Motivation & Objective

• To develop a reinforcement learning agent capable of learning optimal bidding strategies in large-scale real-time online auctions without relying on handcrafted features.
• To enable direct learning from high-level semantic inputs such as plain-text state descriptions, mimicking human decision-making in complex, incomplete-information environments.
• To outperform expert-designed bidding policies in real-world industrial applications, particularly in revenue generation and ROI for advertisers.
• To demonstrate the feasibility of end-to-end deep reinforcement learning in real-world, high-stakes, large-scale online advertising systems.

Proposed method

• The agent uses an asynchronous stochastic variant of Deep Q-Network (DQN), named DASQN, to stabilize training and improve sample efficiency in large-scale environments.
• Raw textual descriptions of auction states—containing semantic information about bids, time, and market conditions—are used as direct input to the neural network.
• The agent is trained using sparse, high-reward signals representing auction profits, which are dense and scalable across millions of auction events.
• The architecture processes sequential auction states to model temporal dependencies and adapt bidding behavior dynamically in real time.
• The training process leverages experience replay and target networks to stabilize learning, typical in DQN-based methods.
• The system is deployed in a real-world RTB (real-time bidding) advertising platform at JD.com, operating under production-scale constraints.

Experimental results

Research questions

• RQ1 Can a deep reinforcement learning agent learn effective bidding policies directly from raw textual state descriptions in large-scale real-time auctions?
• RQ2 Does end-to-end deep reinforcement learning outperform manually engineered, expert-tuned bidding strategies in real-world industrial settings?
• RQ3 Can such an agent achieve human-level performance in terms of revenue generation and ROI improvement in a live advertising platform?
• RQ4 How does the agent generalize across varying market conditions and auction dynamics without explicit feature engineering?

Key findings

• LADDER achieved a 50% increase in ad revenue during JD's June 18th anniversary sale compared to the previous state-of-the-art bidding policy.
• Advertisers' return on investment (ROI) improved significantly, indicating better efficiency and targeting by the agent.
• The agent outperformed a carefully calibrated, human-engineered bidding strategy that had been optimized over months of manual tuning.
• The system successfully learned from raw textual inputs without requiring handcrafted features or domain-specific preprocessing.
• The agent demonstrated robustness and scalability in a real-world, large-scale RTB environment with high-frequency, incomplete-information auctions.