No σ for you!

[carey]

dm319, thank you for asking these questions!

(11-26-2024 08:12 PM)dm319 Wrote:  

(11-25-2024 03:58 PM)carey Wrote:  It works, but working with absolute values in subsequent equations becomes unwieldly.

In what way?

Deriving equations when some terms include absolute values can be challenging, e.g., moving these terms from one side of an equation to the other and eventually removing the absolute value brackets and getting the signs and inequality symbols correct.

(11-26-2024 08:12 PM)dm319 Wrote:  

(11-25-2024 04:49 PM)carey Wrote:  e.g., the standard deviation is needed to define some continuous functions, e.g., the Gaussian (normal) distribution, Since the Central Limit Theorem ensures Gaussian distributions occur over a wide range of typical experimental conditions, this suggests that the standard deviation is Nature's way to characterize variation.

\[
Gaussian function = f(x) = \frac{1}{\sqrt{2 \pi \sigma^2}} \exp\left(-\frac{(x - \mu)^2}{2 \sigma^2}\right),
\]

This seems to be more convincing, but I'll need to think about this more. In my mind SD had little to do with normal distribution other than we are familiar with the SD in a normal distribution covering 68/95/99.7 in 1/2/3. Having said that, the SD is equally spaced along the normal distribution - is this what you mean? I.e. that the SD somehow defines the normal distribution? I need to think about this one...!

The normal distribution cannot be defined without the standard deviation (it can be approximated without the standard deviation, but that's not the same). The actual definition of the normal distribution function requires the standard deviation and since the normal distribution is the typical distribution of experimental uncertainties (guaranteed by the Central Limit Theorem), it is not an exaggeration to claim that the standard deviation is Nature's measure of variation or uncertainty.

(11-26-2024 08:12 PM)dm319 Wrote:  In my mind, squaring the values, and then finding the mean of the variance is going to bias towards larger deviations.

Yes, but note how this is bias cancels a similar bias in applications involving the standard deviation, e.g., (i) defining the normal distribution and (ii) the outlier problem in least squares regression.

1) The numerator of the normal distribution includes squared deviations that suffer from a similar bias. However, the denominator of the normal distribution includes the square of the standard deviation. This has the effect of weighting contributions of deviations inversely with the squares of their uncertainties. Hence, large deviations (which usually have larger uncertainties) contribute less.

2) This is also how the outlier problem of ordinary least squares regression (where large deviations are given extra weight due to squaring) is solved. By dividing by the square of the standard error, each contribution is weighted inversely with the square of its uncertainty. As large deviations usually are more uncertain they are weighted less. The name for least squares weighted inversely by the square of the standard errors is called chi-squared (related to, but not the same as, the chi-squared used in counting experiments).
\[
\chi^{2} = \sum{\left(\frac{(x - \mu)^2}{\sigma^2}\right)}
\]

(11-26-2024 08:12 PM)dm319 Wrote:  Also, it seems like it's using a property of maths for convenience (square then square-root giving a positive value). Maybe instead of RMSD, we could have had MRSD - take the root first then then mean, which gives you the mean of absolute deviation.

While SD is more convenient than using absolute values, the rationale for the RMSD algorithm is rooted in its occurrence, by necessity (not convenience), in the naturally occurring normal distribution.