[논문 리뷰] AtmoRep: A stochastic model of atmosphere dynamics using large scale representation learning

Christian Lessig, I. Luise외 4명|arXiv (Cornell University)|2023. 08. 25.

Meteorological Phenomena and Simulations인용 수 20

한 줄 요약

AtmoRep은 대규모 표현 학습을 사용하는 태스크 독립적 확률적 대기 역학 모델로, nowcasting, 보간, 모델 보정, 반사실적에 걸쳐 확률 예측과 제로샷 기능을 제공합니다.

ABSTRACT

The atmosphere affects humans in a multitude of ways, from loss of life due to adverse weather effects to long-term social and economic impacts on societies. Computer simulations of atmospheric dynamics are, therefore, of great importance for the well-being of our and future generations. Here, we propose AtmoRep, a novel, task-independent stochastic computer model of atmospheric dynamics that can provide skillful results for a wide range of applications. AtmoRep uses large-scale representation learning from artificial intelligence to determine a general description of the highly complex, stochastic dynamics of the atmosphere from the best available estimate of the system's historical trajectory as constrained by observations. This is enabled by a novel self-supervised learning objective and a unique ensemble that samples from the stochastic model with a variability informed by the one in the historical record. The task-independent nature of AtmoRep enables skillful results for a diverse set of applications without specifically training for them and we demonstrate this for nowcasting, temporal interpolation, model correction, and counterfactuals. We also show that AtmoRep can be improved with additional data, for example radar observations, and that it can be extended to tasks such as downscaling. Our work establishes that large-scale neural networks can provide skillful, task-independent models of atmospheric dynamics. With this, they provide a novel means to make the large record of atmospheric observations accessible for applications and for scientific inquiry, complementing existing simulations based on first principles.

연구 동기 및 목표

• 계산적으로 효율적이고 확률적이며 정확하고 다재다능한 대기 모델의 필요성을 제시한다.
• 역사적 관측 데이터로부터 학습하여 복잡한 대기 역학을 포착하는 태스크-아그노스틱(비특정 태스크) 확률 모델을 제안한다.
• 여러 작업에 대한 고유의 제로샷 능력을 보여주고, 추가 데이터로 모델 확장이나 편향 보정이 어떻게 가능한지 보여준다.

제안 방법

• AtmoRep를 대기 상태에 대해 p_theta(y|x, alpha)를 예측하는 3.5십억 개의 매개변수 트랜스포머 모델로 소개한다.
• 데이터 분포와의 거리를 몬테카를로 추정치를 통해 최소화하는 자기지도(self-supervised) 목표로 p_theta(y|x, alpha)를 학습한다.
• 학습된 분포에서 샘플링하기 위해 예측 헤드의 앙상블을 사용하여 확률적 예측을 가능하게 한다.
• 토큰으로 나뉜 4D 시공간 이웃에서 작동하며, 자기지도 학습을 위해 마스킹/왜곡을 사용한다.
• 필드별 변환기를 교차 주의(cross-attention)로 결합한 Multiformer 아키텍처와 효율적인 앙상블 학습 전략을 채택한다.
• 관측 데이터를 사용한 바이어스 보정과 다운스케일링 등 태스크 특화 테일(tails)을 통해 확장성을 확보한다.

Figure 1: The AtmoRep model provides a numerical representation $p_{\theta}(y|x,\alpha)$ of the conditional probability $p(y|x,\alpha)$ for atmospheric states $x$ , $y$ subject to external conditions $\alpha$ , e.g. the time of $x$ , $y$ or their location on the globe. It is implemented as a transfo

실험 결과

연구 질문

• RQ1 Can a large-scale, task-agnostic neural model capture the stochastic dynamics of the atmosphere from observational history?
• RQ2 What are the intrinsic capabilities of AtmoRep for nowcasting, temporal interpolation, model correction, and counterfactuals without task-specific training?
• RQ3 How does AtmoRep perform relative to established forecasting systems in short-term prediction and probabilistic skill?
• RQ4 Can AtmoRep be extended to additional tasks such as downscaling and bias correction using external observational datasets?

주요 결과

• AtmoRep achieves state-of-the-art or competitive nowcasting skill with no or minimal task-specific training.
• The model provides probabilistic forecasts via an ensemble whose spread reflects observational variability.
• AtmoRep enables temporal interpolation with substantially lower RMSE than linear interpolation.
• The model can correct higher-frequency content in inputs toward ERA5 through its learned representation.
• Counterfactuals demonstrate that AtmoRep can generate samples reflecting alternative external conditions under learned distributions.
• Downscaling with AtmoRep outperforms a GAN baseline in RMSE and can adjust distributions toward higher-resolution references.

Figure 2: AtmoRep can be used for a diverse set of applications without task-specific training (shaded areas depict one standard deviation). Nowcasting: Short-term forecasting can be realized by masking tokens at the future-most time step(s) (bottom right inset). Skill is compared to Pangu-Weather a