Show that in a simple linear regression y = X β + ϵ, when the mean is subtracted from any feature (i.e. some column of the matrix X), the t-statistics of all the non-intercept coefficients do not change.

Question

Show that in a simple linear regression   y  =  X  β  +  ϵ, when the mean is subtracted from any feature (i.e. some column of the matrix   X), the   t-statistics of all the non-intercept coefficients do not change.

inmholtau5 · Accepted Answer

Write   X  =  (      x          i      j            )          i      =      1      ,      .      .      .      ,      n      ;      j      =      1      ,      .      .      .      ,      p      . The first column   j  =  1  ) contains only   (  1  ,  1.  ,  ,  ,  1  )  ∈      R    n   because this is the intercept. Say, you center all the columns   j  =  2  ,  .  .  .  ,  p, i.e. you build the matrix             X      ~        =  (                    x        ~                    i      j        ) with                     x        ~                    i      j        =      x          i      j        −                    x        ¯              j   where                     x        ¯              j   is the mean       1    n        ∑          i      =      1        n        x          i      j      . The question is whether the   i-th element of   (                    X        ~              T              X      ~            )          −      1       is the same as the   i-th element of   (      X    T    X      )          −      1      .So   X      e    1   is the vector of size   n with all entries equal to one, and let             x      ¯        ∈      R    p   the vector with entries   0  ,                    x        ¯              2    ,  .  .  .  ,                    x        ¯              p  . Then the construction of             X      ~       is equivalent to             X      ~        =  X  −  (  X      e    1    )                    x        ¯              T    =  X  (      I    p    −      e    1                      x        ¯              T    ).Now                     X        ~              T              X      ~        =  (      I    p    −            x      ¯            e    1    T    )      X    T    X  (      I    p    −      e    1                      x        ¯              T    ) and one can check that   (      I    p    −            x      ¯            e    1    T    ) is invertible, because the image of canonical basis vectors are linearly independent:   (      I    p    −            x      ¯            e    1    T    )      e    j    =      e    j   for   j  ≥  2 and   (      I    p    −            x      ¯            e    1    T    )      e    1   has the first component equal to   1 due to             x      ¯       having   0 in the first component.For   j  ≥  2,      e    j    T    (                    X        ~              T              X      ~            )          −      1            e    j    =      e    j    T    (      I    p    −      e    1                      x        ¯              T        )          −      1        (      X    T    X      )          −      1        (      I    p    −            x      ¯            e    1    T        )          −      1            e    j  and one can check that   (      I    p    −            x      ¯            e    1    T        )          −      1            e    j    =      e    j   because       e    1    T        e    j    =  0. This implies that                     X        ~              T              X      ~       and       X    T    X have the same diagonal elements for   j  ≥  2.

Show that in a simple linear regression y=Xbeta+epsilon, when the mean is subtracted from any feature (i.e. some column of the matrix X), the t-statistics of all the non-intercept coefficients do not change.

Answered question

Answer & Explanation

New Questions in College Statistics