[논문 리뷰] Neural Predictive Belief Representations

Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar외 3명|arXiv (Cornell University)|2018. 11. 15.

Domain Adaptation and Few-Shot Learning참고 문헌 42인용 수 47

한 줄 요약

The paper investigates unsupervised neural methods (one-step frame prediction, CPC, and CPC|Action) to learn belief-state representations in partially observable environments, showing these representations encode state and uncertainty and that multi-step, action-conditioned CPC yields best results in visually complex settings.

ABSTRACT

Unsupervised representation learning has succeeded with excellent results in many applications. It is an especially powerful tool to learn a good representation of environments with partial or noisy observations. In partially observable domains it is important for the representation to encode a belief state, a sufficient statistic of the observations seen so far. In this paper, we investigate whether it is possible to learn such a belief representation using modern neural architectures. Specifically, we focus on one-step frame prediction and two variants of contrastive predictive coding (CPC) as the objective functions to learn the representations. To evaluate these learned representations, we test how well they can predict various pieces of information about the underlying state of the environment, e.g., position of the agent in a 3D maze. We show that all three methods are able to learn belief representations of the environment, they encode not only the state information, but also its uncertainty, a crucial aspect of belief states. We also find that for CPC multi-step predictions and action-conditioning are critical for accurate belief representations in visually complex environments. The ability of neural representations to capture the belief information has the potential to spur new advances for learning and planning in partially observable domains, where leveraging uncertainty is essential for optimal decision making.

연구 동기 및 목표

• Motivate learning belief-state representations that summarize past observations and actions in partially observable environments.
• Evaluate whether unsupervised methods can recover ground-truth state and uncertainty from observations.
• Compare one-step frame prediction, CPC, and CPC|Action on DeepMind Lab tasks.
• Assess how well learned representations encode agent position, trajectory, and object locations under varying visual complexity.
• Examine how prediction horizon and action-conditioning affect representation quality in complex visual settings.

제안 방법

• Employ three representation learning objectives: one-step frame prediction (FP), Contrastive Predictive Coding (CPC), and CPC|Action (action-conditioned CPC).
• Use a GRU-based history encoder to produce belief states b_t from past observations and actions.
• For CPC/CPC|Action, predict future observations o_{t+k} from b_t via a CPC classifier with positive/negative samples drawn from the batch.
• For FP, predict the next observation o_{t+1} from b_t using a transposed-convolutional decoder.
• Evaluate learned belief representations by training auxiliary predictors to recover ground-truth state information (e.g., agent position/orientation, past trajectory, object positions) without backpropagating through the representation.
• Architecture includes a CNN to embed observations to z_t, a belief GRU to produce b_t, an action-GRU (for CPC|Action) to process future actions, and an MLP to predict ground-truth quantities.
• Algorithm 1 describes CPC|Action training: sample sub-trajectories, compute beliefs, unroll future actions, compute CPC loss with positive future observations and a negative sample, average losses, and update.]
• research_questions: [

실험 결과

연구 질문

• RQ1 Can unsupervised methods learn belief-state representations that encode ground-truth state information and uncertainty in partially observable environments?
• RQ2 How do prediction horizon (1 vs 30 steps) and action-conditioning affect the quality of learned belief representations in visually rich domains?
• RQ3 Do learned representations capture agent position, past trajectory, and object locations, and to what extent do they handle uncertainty?

주요 결과

Env	Algorithm	(x,y,θ)	Past (x,y,θ)	Objects (x,y)
fixed	FP	0.118±0.015	0.121±0.007	0.043±0.006
fixed	CPC 1	0.579±0.067	0.132±0.010	0.049±0.005
fixed	CPC 30	0.562±0.204	0.118±0.010	0.045±0.004
fixed	CPC|Action 1	0.689±0.057	0.137±0.006	0.049±0.004
fixed	CPC|Action 30	0.240±0.030	0.100±0.007	0.040±0.003
room	FP	0.517±0.123	0.285±0.017	0.484±0.005
room	CPC 1	2.010±0.142	0.311±0.017	0.498±0.008
room	CPC 30	0.482±0.157	0.257±0.022	0.481±0.005
room	CPC|Action 1	2.274±0.117	0.308±0.018	0.484±0.005
room	CPC|Action 30	0.689±0.066	0.276±0.029	0.484±0.008
maze	FP	0.178±0.207	0.233±0.029	0.322±0.008
maze	CPC 1	0.622±0.158	0.278±0.055	0.330±0.009
maze	CPC 30	0.244±0.058	0.213±0.031	0.325±0.015
maze	CPC|Action 1	0.638±0.094	0.264±0.028	0.323±0.010
maze	CPC|Action 30	0.182±0.034	0.206±0.029	0.323±0.010
terrain	FP	1.831±0.162	0.405±0.077	0.181±0.084
terrain	CPC 1	3.393±0.252	0.417±0.074	0.307±0.174
terrain	CPC 30	2.280±0.853	0.340±0.104	0.131±0.185
terrain	CPC|Action 1	3.348±0.482	0.414±0.042	0.312±0.049
terrain	CPC|Action 30	1.589±0.358	0.344±0.065	0.139±0.136

• All three methods (FP, CPC with 1-step and 30-step forecasts, and CPC|Action) can learn belief representations that encode agent position and orientation and past trajectory.
• Representations also encode uncertainty about the state and objects, which decreases as the agent gathers information from observations and actions.
• In visually simple environments, FP often best encodes position/orientation, while in visually complex terrains, multi-step CPC approaches (especially CPC|Action 30) perform best and are more computationally efficient than FP.
• CPC-based methods better capture distributions over future observations than FP, with CPC|Action providing further gains by conditioning on future actions.
• Object-position information is more reliably captured when objects significantly alter future observations (e.g., teleport interactions); otherwise, objects are harder to encode, indicating reliance on map-specific cues or episodic memory.
• Predicting further into the future (30 steps) and incorporating actions (CPC|Action) substantially improve belief quality in terrain-like environments, compared to single-step predictive approaches.