Fuller is the standard reference and derives what I called the Adjusted Least Squares estimator in his sections 1.1 and 1.2. He awkwardly calls it “least squares corrected for attentuation, but following AGL it would be appropriate to call it “Attenuation-Corrected LS”, or ACLS. (In fact AGL call it ACOLS, but if OLS is modified, it is no longer ordinary, so the O becomes unnecessary or even incorrect.)

Then in his section 1.3, Fuller derives the TLS estimator. Unfortunately he does not mention either term, “TLS” or “Orthogonal LS”. He provides three different derivations of the formula, but unfortunately does not include the nice SVD approach. He shows that it is consistent (p. 32), and gives its asymptotic covariance matrix. On p. 47, he discusses a method for obtaining confidence intervals that I haven’t digested, but which sounds similar to the classical Fieller confidence sets for CCE, because “The confidence sets constructed by the methods of this section will not always be a single interval and can be the whole line.”

AGL embrace ACLS as a solution to the inconsistency of ICE (Inverse Calibration Estimation) when estimating past temperatures from noisy proxies. In doing so, they invent what might be called “Backward LS” (BLS).

They have a case in which their “X” is a noisy proxy (such as an isotope ratio), and “Y” is instrumental temperature, so that var(f) is important but var(e) is nearly 0 and the usual roles of “X” and “Y” are reversed. Instead of just regressing “X” on “Y” and inverting to reconstruct Y from X as in CCE (Classical Calibration Estimation), they regress “Y” on “X” and then correct this ICE estimate for attenuation using ACLS. In order to obtain an estimate of var(f) they do regress X on Y and compute the variance of the residuals, but otherwise discard the consistent CCE estimate of the slope.

Simulations indicate that AGL’s BLS substantially mis-corrects for attenuation in the slope (and therefore also its reciprocal) when the significance of the slope is marginal (expected t = 2), if the standard unbiased estimator s^2 of the X on Y regression error variance is used to estimate var(f) and the adjustment factor is taken as infinity whenever var(x) < var(f). But surprisingly, BLS appears to exactly give the CCE estimator of the slope (and therefore the OLS estimator of its reciprocal) when the Mean Squared Error (dividing by n instead of n-2) is used to estimate var(f). So at best it has no advantage over CCE and at worst it is a lot worse.

AGL propose a further modification of ACLS when both types of error are present, using a cross-validation procedure involving only the X and Y data to estimate their "k". However, this can't be valid, since as Fuller points out, the slope is not identified unless there is some external information about the absolute or relative size of the two types of error. If var(e) (in this case the temperature measurement error) is known externally, it can just be used with ACLS to modify the CCE estimator as I noted in an earlier comment here.

The method I tried to use was adapted from my tls.alt function above, but re-weighting the variables with the estimated variances from the previous step. However, the re-weights had the effect of stalling the coefficients on the OLS values and sending one of the weights off to infinity.

I think that your comments on the number of unknowns in TLS exceeding the amount of information in the observations is particularly on target here. As is, that approach seems to be a bit of a dead end.

I agree that CCE is ordinarily the way to go with proxies, since the regression error is much bigger than the measurement error in the instrumental temperature series (we hope!). But TLS sounds good on paper and is popular with climate people, so it’s good to figure out why it does or doesn’t help.

According to Wiki, the emminent economist Paul Samuelson was enthusiastic about TLS back in the 1940s when he was an eager-beaver grad student. But now the econometric texts don’t even mention it.

http://www.amazon.com/Statistical-Regression-Measurement-Error-Statistics/dp/0340614617

seems to be a good book to start. If the ratio of the _error_ variances is known, the _data_ can be scaled so that this ratio becomes one. Then the maximum likelihood solution of the normal ME (measurement error) regression is orthogonal regression (p.9).

For the calibration problem the book offers interesting reference for comparing ICE, CCE and ME,

C-L. Cheng and C-L. Tsai (1994). “The comparisons of three different linear calibration estimators in measurement error models.”

It is listed in here, http://www.stat.sinica.edu.tw/clcheng/index.htm but it is unpublished manuscript…

In my opinion CCE is the only way to go with proxies. ICE implies structural model ( we know a lot about past temperatures prior to looking at proxies ), and for orthogonal regression the assumption seems to be that we know the ratio of the error variances prior to looking at proxies. But I’m quite unfamiliar with this topic, so I might be wrong.

After reading your comments, I also tried a sequential reweighted LS approach as well. At each step, the variances of e and f were re-estimated and the regression repeated using the new weights.

How did you do this? Originally I was hoping to iteratively estimate the variance of e from the TLS residuals, starting from knowledge of var(f) and an initial guess of var(e) (say from OLS of y on x). But then I realized that TLS does not give back the true relative variance of f and e, unless the slope happens to be +/- 1 which would only be by accident.

Although TLS is ML-motivated, it does not necessarily have the nice properties of ML. ML is consistent if a fixed number of parameters are being estimated with an increasing number of observations. But in TLS the number of unknowns increases 1-for-1 with the number of observations, so that x* and y* are not consistently estimated. It appears that as a consequence, the variance ratio is not consistently estimated by the residuals, even though the slope appears to be consistently estimated (from my limited simulations).

Please e-mail me (and the gang) if and when you add to this interesting discussion, as it proceeds rather slowly.

I have looked at the situation further and did some more analysis which seemed to add further insight in regards to some of your other points:

Although TLS sounds good on paper, in practice it is almost never useful, since it requires knowing the relative variances of the two errors e and f.

I believe the problems with TLS to go deeper than that. If the relative variances are known, TLS can be applied after weighting the the SS components in the head post inversely to the variances of the e and f “errors” (thus reducing the problem to the equal variance case). However, the residuals for e and f are still perfectly linearly correlated although the ratio between them is changed to take the weighting into account.

After reading your comments, I also tried a sequential reweighted LS approach as well. At each step, the variances of e and f were re-estimated and the regression repeated using the new weights. However, all of the cases I tried converged to the end result of a LS where the variable with the higher variance has its error term reduced to zero variability. I also looked very briefly at using maximum likelihood with normally distributed errors, but it wasn’t clear that this would produce any better results.

With regard to the consistency of the parameter estimates, my gut feeling would be that in most cases the results would be consistent although there might be demonstrable bias in the parameters (that converged to zero as the sample size increased). Where it could be bothersome is in the case that X and Y are uncorrelated with approximately equal variability. However, |I am just guessing on this aspect and it could be wrong.

The possible use of Adjusted OLS is intriguing, but I would need to do some reading to say more about that.

Getting insight on the behavior of the estimates and their standard errors is made more difficult because the mechanics of the calculation involve portions of the principal components of the variables – in the multivariate case particularly. Something that might make it easier to comprehend is to do the calculations recursively and examine what happens in that process. I wrote a short program for this:

tls.alt = function(yvec,xvec,wts = c(1,1),tol=1E-6) {
  #wts = (y-variable weight, x-variable weight)
  # typically weights would be 1/ variable variance
  #variables do not have to be centered
    dif = tol+1
    ywt = wts[1]; xwt = wts[2]
    newx = xvec
  while (dif>tol) {
   lastx = newx
   regco = coef(lm(yvec~newx))
   newx = (ywt*regco[2]*(yvec-regco[1]) + xwt*xvec)/(xwt+ ywt*regco[2]^2)
   dif = max(abs(newx-lastx),na.rm=T)}
   yscore = regco[1]+regco[2]*newx
   xres = xvec-newx
   yres = yvec-yscore
   xss = sum(xres^2, na.rm=T)   
   yss = sum(yres^2, na.rm=T)
 list(coefs=regco,yscore = yscore, xscore = newx, xresid =xres,yresid=yres,xss=xss,yss=yss)} 

% TLS0
% univariate TLS with pre-centered and pre-normalized data
% (zero in name indicates restricted problem)
% x, y ~ n x 1
% x = xstar + f, y = ystar + e
% ystar = beta * xstar, var(f) + var(e) = min.
% x and y have been pre-normalized so that var(e) = var(f)
% by Hu McCulloch 1/28/12, after P. de Groen, “An Introduction to
% Total Least Squares,” Nieuw Archief voor Wiskunde, 1996, 14:237-53,
% http://www.freescience.info
function [beta] = TLS0(x, y)
B = [x y];
[U, S, V] = svd(B, ‘econ’); % B*V = U*S, S is 2×2, V is 2×2
beta = -V(1,2)/V(2,2); % This correctly gives inf if V(2,2) = 0.
end
% xstar and ystar are in U somewhere, but I’m not certain how to
% recover them.
return

It’s not really necessary to define “B”, but I’ve included it for clarity.