Bayes Estimator With Absolute Loss Is Median

Bayes Estimator with Absolute Loss is the Median

This article explores the fascinating relationship between Bayesian estimation and the median. Specifically, we'll demonstrate why, under an absolute loss function, the Bayes estimator is the median of the posterior distribution. Understanding this connection provides valuable insights into Bayesian inference and offers a practical alternative to using the mean, especially when dealing with skewed distributions.

What is a Bayes Estimator?

In Bayesian statistics, we aim to estimate an unknown parameter θ based on observed data x. We achieve this using a Bayes estimator, which is a function of the data that minimizes expected loss. The expected loss is calculated using the posterior distribution of θ given the data, p(θ|x). The formula for the posterior distribution is given by Bayes' theorem:

p(θ|x) = [p(x|θ)p(θ)] / p(x)

Where:

• p(θ|x) is the posterior distribution of θ given x.
• p(x|θ) is the likelihood function, representing the probability of observing x given θ.
• p(θ) is the prior distribution, reflecting our initial beliefs about θ.
• p(x) is the marginal likelihood (evidence), a normalizing constant.

The Absolute Loss Function

The choice of loss function significantly impacts the resulting Bayes estimator. A common loss function is the absolute loss function, defined as:

L(θ, θ̂) = |θ - θ̂|

where θ is the true value of the parameter and θ̂ is our estimate. This function measures the absolute difference between the true value and the estimate. Minimizing this loss function leads to an estimator that is robust to outliers and less sensitive to extreme values compared to squared error loss.

Deriving the Median as the Bayes Estimator

To find the Bayes estimator under absolute loss, we need to minimize the expected loss:

E[L(θ, θ̂)|x] = E[|θ - θ̂||x] = ∫ |θ - θ̂| p(θ|x) dθ

To minimize this expected value, we take the derivative with respect to θ̂ and set it to zero. However, the absolute value function is not differentiable at zero. Instead, we can use a more intuitive approach.

Consider the integral:

∫ |θ - θ̂| p(θ|x) dθ = ∫ (θ - θ̂) p(θ|x) dθ for θ > θ̂ + ∫ (θ̂ - θ) p(θ|x) dθ for θ < θ̂

Minimizing this integral is equivalent to finding the median of the posterior distribution p(θ|x). Intuitively, the median minimizes the expected absolute distance to all points in the distribution. Points above the median contribute positively to the integral, while points below contribute negatively. The median balances these contributions, resulting in the minimum expected absolute loss.

In Summary

The Bayes estimator under an absolute loss function is the median of the posterior distribution. This offers a valuable alternative to using the mean, particularly when dealing with skewed distributions or data containing outliers, where the median provides a more robust and representative estimate. This understanding solidifies the connection between Bayesian inference and robust statistical methods. The choice between mean and median as a Bayes estimator hinges entirely on the chosen loss function, highlighting the importance of careful consideration when selecting an appropriate loss function for a given problem.