In Linear regression, we have θ = ( X T X ) − 1 X T y .In Ridge regression, we have θ = ( λ I + X T X ) − 1 X T y .I learnt somewhere that while X T X is not guaranteed to be invertible, λ I + X T X is guaranteed to be invertible.Is this true? If so, why?

Question

In Linear regression, we have   θ  =                    (                  X          T                X        )                    −        1                    X      T        y  .In Ridge regression, we have   θ  =                    (        λ        I        +                  X          T                X        )                    −        1                    X      T        y  .I learnt somewhere that while             X      T        X   is not guaranteed to be invertible,       λ    I    +          X      T        X   is guaranteed to be invertible.Is this true? If so, why?

Marco Meyer · Accepted Answer

Can you write the eigenvalues of   λ  I  +      X    ⊤    X in terms of the eigenvalues of       X    ⊤    X?The eigenvalues of       X    ⊤    X are nonnegative. (Why?) It is possible for it to be not invertible (specifically, if some of the eigenvalues are zero). However, the eigenvalues of   λ  I  +      X    ⊤    X are of the form   λ  +  μ where   μ is an eigenvalue of       X    ⊤    X. (Why?) Since   λ  &amp;gt;  0 and   μ  ≥  0, the eigenvalues of   λ  I  +      X    ⊤    X are all strictly positive, so it is invertible.

Direkotogbkmn · Accepted Answer

First, replace   λ  →  −      λ    ~  . Then the given expression      X    T    X  −      λ    ~    Iis similar to the eigenvalue equation of the matrix       X    T    X. As       X    T    X is a positive semi-definite matrix we know that its eigenvalues are all   ≥  0. This implies that for   λ the corresponding values are   ≤  0. If you state that   λ  &amp;gt;  0 then you rule out all possible eigenvalues. Hence, the eigenvalue equation  det      [          X      T        X    −            λ      ~        I    ]    =  0does not have any solutions. Hence,      [          X      T        X    −            λ      ~        I    ]    =      [          X      T        X    +    λ    I    ]   is invertible for   λ  &amp;gt;  0

In Linear regression, we have θ = (

Answered question

Answer & Explanation

New Questions in Inferential Statistics