[Paper Review] Convergence Analysis of No-Regret Bidding Algorithms in Repeated Auctions

Zhe Feng, Guru Guruganesh+3 more|arXiv (Cornell University)|Sep 14, 2020

Advanced Bandit Algorithms Research16 references4 citations

TL;DR

This paper analyzes the convergence of no-regret bidding algorithms in repeated auctions, showing that mean-based learning rules—such as Exp3, UCB, and ε-greedy—drive bidders to converge to the truthful Nash equilibrium in second-price and VCG auctions, and to the Bayesian Nash equilibrium in first-price auctions. The key result is high-probability convergence to equilibrium under mild distributional assumptions, validated by experiments with both contextual bandits and Deep Q-Learning agents.

ABSTRACT

The connection between games and no-regret algorithms has been widely studied in the literature. A fundamental result is that when all players play no-regret strategies, this produces a sequence of actions whose time-average is a coarse-correlated equilibrium of the game. However, much less is known about equilibrium selection in the case that multiple equilibria exist. In this work, we study the convergence of no-regret bidding algorithms in auctions. Besides being of theoretical interest, bidding dynamics in auctions is an important question from a practical viewpoint as well. We study repeated game between bidders in which a single item is sold at each time step and the bidder's value is drawn from an unknown distribution. We show that if the bidders use any mean-based learning rule then the bidders converge with high probability to the truthful pure Nash Equilibrium in a second price auction, in VCG auction in the multi-slot setting and to the Bayesian Nash equilibrium in a first price auction. We note mean-based algorithms cover a wide variety of known no-regret algorithms such as Exp3, UCB, $ε$-Greedy etc. Also, we analyze the convergence of the individual iterates produced by such learning algorithms, as opposed to the time-average of the sequence. Our experiments corroborate our theoretical findings and also find a similar convergence when we use other strategies such as Deep Q-Learning.

Motivation & Objective

• To understand which equilibrium emerges when bidders use no-regret learning in repeated auctions with multiple equilibria.
• To establish theoretical conditions under which mean-based learning rules converge to specific equilibria in second-price, VCG, and first-price auctions.
• To analyze both time-averaged and individual iterates of learning dynamics, moving beyond coarse correlated equilibria.
• To validate theoretical findings experimentally using ε-greedy and Deep Q-Learning agents in first- and second-price auction settings.

Proposed method

• Uses mean-based learning rules—such as Exp3, UCB, and ε-greedy—as the core bidding strategy for bidders in repeated auctions.
• Applies a thickness assumption on the distribution of rival bids (Assumption 4.2) to ensure sufficient density for convergence.
• Proves high-probability convergence of individual bids (not just time-averages) to equilibrium strategies under the thickness condition.
• Employs theoretical analysis of regret and deviation bounds to show truthful bidding dominates in second-price and VCG auctions.
• Extends analysis to first-price auctions by modeling Bayesian Nash equilibrium under i.i.d. value draws and proving convergence to equilibrium bids.
• Validates results via simulations using contextual ε-greedy and Deep Q-Learning agents in first- and second-price auctions with uniform value distributions.

Experimental results

Research questions

• RQ1 Under what conditions do no-regret learning algorithms converge to a specific equilibrium in repeated auctions with multiple equilibria?
• RQ2 Does the individual sequence of bids (not just the time-average) converge to equilibrium under mean-based learning?
• RQ3 Can mean-based learning rules like ε-greedy and Deep Q-Learning lead bidders to the truthful equilibrium in second-price and VCG auctions?
• RQ4 How does the convergence behavior differ between first-price and second-price auctions under the same learning dynamics?
• RQ5 Does the thickness assumption on the distribution of rival bids ensure convergence to equilibrium in repeated auction settings?

Key findings

• With high probability, each γt-mean-based learner converges to truthful bidding in repeated multi-position VCG auctions, provided the thickness assumption holds and T₀ is sufficiently large.
• In second-price and VCG auctions, truthful bidding is selected with probability at least 1 - Hγt - 4exp(−τ²ρ²T₀/(32n²H²)) for any fixed valuation.
• In first-price auctions with i.i.d. uniform valuations, ε-greedy and Deep Q-Learning agents converge to the Bayesian Nash equilibrium, where each bidder bids half their true value in the two-bidder case.
• Experiments show that both ε-greedy and Deep Q-Learning agents converge to the BNE in both first- and second-price auctions, with mean rewards approaching theoretical equilibrium utilities.
• The convergence is robust to increasing agent count, as shown in experiments with more than two bidders, where convergence to BNE is still observed.
• Even when exploration probability ε is bounded at 0.05, the agents' bidding behavior converges closely to the BNE, with only a small gap in mean reward.