[Paper Review] Adaptive Signal Detection and Parameter Estimation in Unknown Colored Gaussian Noise

Bo Tang, Haibo He+1 more|arXiv (Cornell University)|Jul 27, 2016

Radar Systems and Signal Processing10 references3 citations

TL;DR

This paper proposes three adaptive signal detectors—two GLRT-based and one Rao test-based—for detecting signals in unknown colored Gaussian noise modeled as an autoregressive (AR) process. By deriving closed-form maximum likelihood estimates and asymptotic distributions, the detectors achieve constant false alarm rate (CFAR) and demonstrate strong performance in frequency-modulated signal and time-series moving object detection applications.

ABSTRACT

This paper considers the general signal detection and parameter estimation problem in the presence of colored Gaussian noise disturbance. By modeling the disturbance with an autoregressive process, we present three signal detectors with different unknown parameters under the general framework of binary hypothesis testing. The closed form of parameter estimates and the asymptotic distributions of these three tests are also given. Given two examples of frequency modulated signal detection problem and time series moving object detection problem, the simulation results demonstrate the effectiveness of three presented detectors.

Motivation & Objective

• Address the limitation of existing detectors that assume white Gaussian noise, which degrades performance in real-world colored noise environments.
• Formulate a general signal detection problem under binary hypothesis testing with unknown signal amplitude and unknown AR parameters modeling colored noise.
• Develop three sub-optimal detectors (GLRT and Rao test variants) for different unknown parameter configurations in AR-driven colored noise.
• Derive closed-form maximum likelihood estimates for unknown parameters and asymptotic distributions of test statistics to enable practical implementation.
• Demonstrate the effectiveness of the proposed detectors through simulations on stepped-frequency signal detection and time-series moving object detection.

Proposed method

• Model the colored Gaussian noise as an autoregressive (AR) process of order p, with unknown coefficients α₁,…,αₚ and innovation variance σ².
• Apply the generalized likelihood ratio test (GLRT) framework to construct two detectors: one with unknown signal amplitude and AR coefficients, and another with unknown signal amplitude and noise variance.
• Derive the Rao test detector under the same hypothesis framework, leveraging the Cramér-Rao lower bound and asymptotic distribution theory.
• Use maximum likelihood estimation (MLE) to compute closed-form expressions for unknown parameters, including AR coefficients and noise variance, via Yule-Walker equations.
• Transform the original detection problem into a linear model using a transformation matrix T and vector c, enabling efficient computation of test statistics.
• Establish the asymptotic distribution of the Rao test as a non-central chi-squared distribution, and show equivalence to GLRT under large sample conditions.

Experimental results

Research questions

• RQ1 How can signal detection be effectively performed in the presence of unknown colored Gaussian noise modeled as an AR process?
• RQ2 What are the closed-form expressions for maximum likelihood estimates of unknown parameters (signal amplitude, AR coefficients, noise variance) in such a model?
• RQ3 How do the asymptotic distributions of the GLRT and Rao test detectors behave under different unknown parameter configurations?
• RQ4 Can the proposed detectors maintain constant false alarm rate (CFAR) performance when the noise covariance is unknown?
• RQ5 How do the detectors perform in practical applications such as stepped-frequency signal detection and time-series moving object detection?

Key findings

• The proposed GLRT detectors achieve exact chi-squared and non-central chi-squared distributions under H₀ and H₁, respectively, enabling precise threshold setting.
• The Rao test detector asymptotically matches the performance of the GLRT under large sample conditions (N → ∞), validating its optimality in the limit.
• Closed-form MLEs for AR coefficients and noise variance are derived using Yule-Walker equations, enabling efficient implementation.
• Simulation results show strong detection performance in stepped-frequency signal detection, confirming robustness to colored noise.
• The detectors are successfully extended to time-series moving object detection, demonstrating wide applicability in real-world signal processing tasks.
• All detectors maintain constant false alarm rate (CFAR) behavior, a critical property for radar and sonar applications.