[Paper Review] Inferring agent objectives at different scales of a complex adaptive system

Dieter Hendricks, Adam D. Cobb+3 more|arXiv (Cornell University)|Dec 4, 2017

Complex Systems and Decision Making4 citations

TL;DR

This paper proposes a multi-scale inverse reinforcement learning framework to infer agent objectives in financial market microstructure by estimating scale-specific reward functions from aggregate market behavior. Using maximum entropy IRL on temporally aggregated state-action trajectories, it reveals that feature vectors have differing relative attractiveness across time scales—indicating distinct agent objectives at high-frequency and longer-term horizons—suggesting a hierarchical structure in market dynamics.

ABSTRACT

We introduce a framework to study the effective objectives at different time scales of financial market microstructure. The financial market can be regarded as a complex adaptive system, where purposeful agents collectively and simultaneously create and perceive their environment as they interact with it. It has been suggested that multiple agent classes operate in this system, with a non-trivial hierarchy of top-down and bottom-up causation classes with different effective models governing each level. We conjecture that agent classes may in fact operate at different time scales and thus act differently in response to the same perceived market state. Given scale-specific temporal state trajectories and action sequences estimated from aggregate market behaviour, we use Inverse Reinforcement Learning to compute the effective reward function for the aggregate agent class at each scale, allowing us to assess the relative attractiveness of feature vectors across different scales. Differences in reward functions for feature vectors may indicate different objectives of market participants, which could assist in finding the scale boundary for agent classes. This has implications for learning algorithms operating in this domain.

Motivation & Objective

• To understand how different agent classes in financial markets operate at distinct time scales with potentially divergent objectives.
• To develop a method for inferring effective reward functions at multiple temporal scales from aggregate market data.
• To investigate whether differences in reward functions across scales indicate distinct agent objectives or strategic behaviors.
• To support the design of multi-scale learning algorithms by identifying scale-specific state representations and objectives.
• To explore the existence of a scale boundary between agent classes such as high-frequency traders, market makers, and long-term investors.

Proposed method

• Applies Maximum Entropy Inverse Reinforcement Learning (MaxEnt IRL) to estimate reward functions from observed state-action trajectories at different time scales.
• Uses temporal state trajectories and action sequences derived from aggregate market data, segmented into 5-, 15-, 30-, and 60-minute intervals.
• Employs a linear reward function r = θ⊤fζ, where fζ represents feature counts along trajectories, and optimizes θ to match empirical feature expectations.
• Computes expected state visitation frequencies Dsi via backward and forward passes through the MDP transition model T to estimate gradients.
• Normalizes and clusters states by feature vector similarity (K-means, k=6) to compare relative rewards across scales for the same feature patterns.
• Visualizes and compares normalized rewards across scales using node size (time scale) and color (reward magnitude) in cluster-based plots.

Experimental results

Research questions

• RQ1 Do agent objectives in financial markets differ across time scales, as indicated by variations in effective reward functions?
• RQ2 Can scale-specific reward functions be reliably inferred from aggregate market behavior using inverse reinforcement learning?
• RQ3 Are certain feature vectors more attractive at specific time scales, suggesting different strategic behaviors among agent classes?
• RQ4 What evidence exists for a hierarchical structure in market causality, with distinct models governing different temporal scales?
• RQ5 How do volume imbalance and spread features influence state attractiveness differently across time horizons?

Key findings

• Feature vectors with similar values exhibit differing relative rewards across time scales, indicating that the same market state can be more or less attractive depending on the observer's time horizon.
• Cluster 3 shows a negative spread and large negative volume imbalance that is favorable at the 5-minute scale but unfavorable at 60 minutes, suggesting time-scale-dependent strategic responses.
• Cluster 5, with positive volume imbalance and negative spread, yields positive rewards at 5 minutes (favorable for short-term selling) but negative rewards at longer scales, indicating potential price pressure effects.
• Cluster 6, with negative volume imbalance and positive spread, shows consistently positive rewards across all scales, suggesting favorable conditions for buying at lower prices with upward price pressure.
• The relative reward for the same feature vector varies significantly across scales, implying that agent objectives—such as high-frequency trading vs. long-term investing—are not uniform and may be scale-dependent.
• The framework provides empirical evidence for a multi-scale structure in market dynamics, supporting the existence of distinct agent classes operating at different temporal resolutions.