Least squares in calibration: Dealing with uncertainty in x

Abstract

The least-squares (LS) analysis of data with error in x and y is generally thought to yield best results when carried out by minimizing the “total variance” (TV), defined as the sum of the properly weighted squared residuals in x and y. Alternative “effective variance” (EV) methods project the uncertainty in x into an effective contribution to that in y, and though easier to employ are considered to be less reliable. In the case of a linear response function with both σ_x and σ_y constant, the EV solutions are identically those from ordinary LS; and Monte Carlo (MC) simulations reveal that they can actually yield smaller root-mean-square errors than the TV method. Furthermore, the biases can be predicted from theory based on inverse regression — x upon y when x is error-free and y is uncertain — which yields a bias factor proportional to the ratio σ²_x/σ²_xmof the random-error variance in x to the model variance. The MC simulations confirm that the biases are essentially independent of the error in y, hence correctable. With such bias corrections, the better performance of the EV method in estimating the parameters translates into better performance in estimating the unknown (x₀) from measurements (y₀) of its response. The predictability of the EV parameter biases extends also to heteroscedastic y data as long as σ_x remains constant, but the estimation of x₀ is not as good in this case. When both x and y are heteroscedastic, there is no known way to predict the biases. However, the MC simulations suggest that for proportional error in x, a geometric x-structure leads to small bias and comparable performance for the EV and TV methods.