Approximating the following system using ordinary least squares regression y = p 0 + p 1 x 1 + p 2 x 2 + . . . p M x M = ξ PI know that a property of the least squares estimator is that the sum of the residuals, r i = y ^ i − y i , is equal to zero. However, what I am finding is that the sum of the residuals multiplied by one of the regressors, ξ m ∈ ( 1 , x 1 , x 2 , . . . , x M ), is also equal to zero. I have found this to be true in all cases in simulation, but I am not sure how to prove it. ∑ i = 0 N r i ξ m = 0

Question

Approximating the following system using ordinary least squares regression  y  =      p    0    +      p    1        x    1    +      p    2        x    2    +  .  .  .      p    M        x    M    =  ξ  PI know that a property of the least squares estimator is that the sum of the residuals,       r    i    =              y      ^        i    −      y    i  , is equal to zero. However, what I am finding is that the sum of the residuals multiplied by one of the regressors,       ξ    m    ∈  (  1  ,      x    1    ,      x    2    ,  .  .  .  ,      x    M    ), is also equal to zero. I have found this to be true in all cases in simulation, but I am not sure how to prove it.      ∑          i      =      0          N        r    i        ξ    m    =  0

Kole Weber · Accepted Answer

Consider a regression model      y    i    =      X    i    ′    β  +      ϵ    i  The least squares problem is to minimize      ∑          i      =      1          n    (      y    i    −      X    i    ′    β      )    2  The first order condition for the optimal coefficient vecto r      β    ^   is      ∑          i      =      1          n    2  (      y    i    −      X    i    ′        β    ^    )      X    i    =  0or      ∑          i      =      1          n        ϵ    i        X    i    =  0which says that the sample autocovariance between the residual and each regressor is zero.

kvasilw0 · Accepted Answer

The property is true because, in linear regression, you are projecting (orthogonally) your target vector onto the subspace generated by the features (columns of the design matrix) and solving an approximate system  X  w  =      y    ~  where       y    ~   is the projection of   y onto the above mentioned subspace. Then the difference   y  −      y    ~   is orthogonal to the subspace, so it is orthogonal to each of the feature columns and, therefore, the dot products are equal to zero.

Answered question

Answer & Explanation

New Questions in High school statistics