[论文解读] Structuring Value Representations via Geometric Coherence in Markov Decision Processes

Zuyuan Zhang, Zeyu Fang+1 more|arXiv (Cornell University)|Feb 3, 2026

Reinforcement Learning in Robotics被引用 0

一句话总结

This paper reframes value learning in RL as learning a sequence of super-poset refinements to enforce geometric coherence, using soft and hard enforcement methods to regularize symmetry and partial order constraints, with theoretical convergence guarantees and empirical gains.

ABSTRACT

Geometric properties can be leveraged to stabilize and speed reinforcement learning. Existing examples include encoding symmetry structure, geometry-aware data augmentation, and enforcing structural restrictions. In this paper, we take a novel view of RL through the lens of order theory and recast value function estimates into learning a desired poset (partially ordered set). We propose \emph{GCR-RL} (Geometric Coherence Regularized Reinforcement Learning) that computes a sequence of super-poset refinements -- by refining posets in previous steps and learning additional order relationships from temporal difference signals -- thus ensuring geometric coherence across the sequence of posets underpinning the learned value functions. Two novel algorithms by Q-learning and by actor--critic are developed to efficiently realize these super-poset refinements. Their theoretical properties and convergence rates are analyzed. We empirically evaluate GCR-RL in a range of tasks and demonstrate significant improvements in sample efficiency and stable performance over strong baselines.

研究动机与目标

• Propose an order-theoretic view of value learning by modeling value estimates as a poset over state-action pairs.
• Develop GCR-RL to refine posets progressively via TD signals while preserving geometric coherence.
• Introduce soft and hard enforcement mechanisms for symmetry and order constraints with theoretical guarantees.
• Provide convergence analysis and empirical evaluation across grid, MiniGrid, Atari, and non-transitive chain tasks.
• Demonstrate improvements in sample efficiency, stability, and reduced Bellman residuals compared to baselines.

提出的方法

• Represent value learning as constructing a sequence of super-poset refinements over symmetry-quotiented elements X/∼.
• Learn near-automorphisms with a symmetry module to enforce Eq(G) softly and reduce variance.
• Construct a TD-driven partial order via a DAG from TD targets and bootstrap a differentiable isotonic projection to Mono(D).
• Provide two enforcement modes: (i) soft coherence regularization combining L_sym and L_ord losses; (ii) hard manifold enforcement projecting updates onto a constrained feasible set M.
• Prove monotonic refinement (Theorem 4.6), automorphism identifiability (Theorems 4.7–4.9), and convergence rate O(sqrt(R(N)/N)) (Theorem 4.10).
• Optionally implement a group-parameter closure mechanism to maintain symmetry constraints via a three-stage projection/closure/alignment process.

(a) RotMirror-Grid (E1): symmetry with controlled breaking

实验结果

研究问题

• RQ1 Can value learning in RL be recast as learning a poset over state–action pairs that reflects the optimal action structure?
• RQ2 How can TD signals be used to progressively refine a poset while ensuring geometric coherence and antisymmetry?
• RQ3 Do soft (regularized) and hard (projected) enforcement strategies for symmetry and order constraints improve stability and sample efficiency in RL?
• RQ4 What are the theoretical guarantees (convergence, variance reduction, identifiability) for GCR-RL under standard RL assumptions?
• RQ5 Do empirical results on grid, MiniGrid, Atari, and non-transitive tasks show improvements over strong baselines in sample efficiency and stability?

主要发现

• GCR-RL yields significant improvements in sample efficiency and stable performance over strong baselines across tasks.
• Learning a sequence of super-poset refinements enforces geometric coherence underpinning learned value functions.
• Soft enforcement via a learned symmetry module and a differentiable order alignment reduces variance and speeds convergence.
• Hard enforcement via a batchwise manifold projection maintains valid posets and provides convergence guarantees.
• Theoretical results include monotonic refinement, automorphism identifiability, reduced Bellman residuals, and a convergence rate of O(sqrt(R(N)/N)).
• Empirical evaluation on grid, Minigrid, Atari, and non-transitive chain tasks demonstrates improved stability and reduced Bellman residuals.

(b) Noisy-RPS Chain (E4): local non-transitive cycles under noise