[论文解读] Taming Vision Priors for Data Efficient mmWave Channel Modeling

Zhenlin An, Longfei Shangguan+6 more|arXiv (Cornell University)|Mar 11, 2026

Millimeter-Wave Propagation and Modeling被引用 0

一句话总结

VisRFTwin 利用视觉推导的先验来初始化和校准一个可微分光线追踪器，用于毫米波信道，显著减少标定数据量同时保持多路径精度。

ABSTRACT

Accurately modeling millimeter-wave (mmWave) propagation is essential for real-time AR and autonomous systems. Differentiable ray tracing offers a physics-grounded solution but still facing deployment challenges due to its over-reliance on exhaustive channel measurements or brittle, hand-tuned scene models for material properties. We present VisRFTwin, a scalable and data-efficient digital-twin framework that integrates vision-derived material priors with differentiable ray tracing. Multi-view images from commodity cameras are processed by a frozen Vision-Language Model to extract dense semantic embeddings, which are translated into initial estimates of permittivity and conductivity for scene surfaces. These priors initialize a Sionna-based differentiable ray tracer, which rapidly calibrates material parameters via gradient descent with only a few dozen sparse channel soundings. Once calibrated, the association between vision features and material parameters is retained, enabling fast transfer to new scenarios without repeated calibration. Evaluations across three real-world scenarios, including office interiors, urban canyons, and dynamic public spaces show that VisRFTwin reduces channel measurement needs by up to 10$ imes$ while achieving a 59% lower median delay spread error than pure data-driven deep learning methods.

研究动机与目标

• Motivate data-efficient mmWave channel modeling suitable for real-time AR and autonomous systems.
• Leverage vision priors to initialize electromagnetic properties of scene surfaces.
• Integrate vision-derived materials with differentiable ray tracing to enable rapid calibration with limited channel measurements.
• Address scene dynamics with incremental updates and region-local refinements for practicality.

提出的方法

• Use multi-view RGB images to reconstruct a 3D scene and extract semantic features via a frozen Vision-Language Model (CLIP).
• Train a NeRF-based semantic extractor to produce a dense 3D semantic field aligned with CLIP embeddings through a semantic loss.
• Translate voxel-wise semantic features into frequency-dependent EM parameters via a lightweight, physics-regularized translator.
• Feed the geometry and vision-informed EM maps into a differentiable Sionna ray tracer to compute multipath channels.
• Perform few-shot calibration by optimizing EM parameters with channel measurements using differentiable ray tracing gradients, ensuring physically valid parameterizations.
• Maintain adaptability to dynamics by localizing updates to affected regions and re-optimizing locally.

实验结果

研究问题

• RQ1 Can vision-derived priors initialize electromagnetic parameters to enable rapid calibration of differentiable ray tracers for mmWave channels?
• RQ2 How few channel measurements are needed to achieve accurate multipath channel modeling when guided by vision priors?
• RQ3 Does the vision-guided calibration generalize across diverse environments such as indoor offices, urban canyons, and dynamic spaces?
• RQ4 How well does the proposed translator map open-vocabulary semantic features to physically meaningful EM parameters?
• RQ5 What is the impact on delay-spread modeling accuracy compared to data-driven baselines in zero-shot and few-shot settings?

主要发现

• Channel measurement requirements can be reduced by up to 10x with VisRFTwin.
• In zero-shot settings, VisRFTwin achieves 59% lower median delay-spread error than pure data-driven models trained on 20% of the data.
• The framework maintains stable performance across LOS and NLoS regions and across multiple environment types (office interiors, urban canyons, dynamic spaces).
• Vision priors initialize EM parameters that are refined with only a handful of channel soundings, enabling rapid digital-twin calibration.
• The approach retains physical interpretability by grounding EM parameters in physics-based relations and by leveraging differentiable ray tracing for gradient-based refinement.