Matrix derivative in multiple linear regression modelThe basic setup in multiple linear regression model is Y = [ y 1 y 2 ⋮ y n ] X = [ 1 x 11 … x 1 k 1 x 21 … x 2 k ⋮ … … 1 x n 1 … x n k ] β = [ β 0 β 1 ⋮ β k ] ϵ = [ ϵ 1 ϵ 2 ⋮ ϵ n ] The regression model is Y = X β + ϵTo find least square estimator of β vector, we need to minimize S ( β ) = Σ i = 1 n ϵ i 2 = ϵ ′ ϵ = ( y − x β ) ′ ( y − x β ) = y ′ y − 2 β ′ x ′ y + β ′ x ′ x β ∂ S ( β ) ∂ β = 0My question: how to get − 2 x ′ y + 2 x ′ x β?

Question

Matrix derivative in multiple linear regression modelThe basic setup in multiple linear regression model is                    Y                            =                  [                                                                      y                                      1                                                                                                                        y                                      2                                                                                                      ⋮                                                                                      y                                      n                                                                                ]                                        X                            =                  [                                                    1                                                              x                                      11                                                                              …                                                              x                                      1                    k                                                                                                      1                                                              x                                      21                                                                              …                                                              x                                      2                    k                                                                                                      ⋮                                            …                                            …                                                                    1                                                              x                                      n                    1                                                                              …                                                              x                                      n                    k                                                                                ]                                        β                            =                  [                                                                      β                                      0                                                                                                                        β                                      1                                                                                                      ⋮                                                                                      β                                      k                                                                                ]                                        ϵ                            =                  [                                                                      ϵ                                      1                                                                                                                        ϵ                                      2                                                                                                      ⋮                                                                                      ϵ                                      n                                                                                ]                    The regression model is   Y  =  X  β  +  ϵTo find least square estimator of   β vector, we need to minimize   S  (  β  )  =      Σ          i      =      1        n        ϵ    i    2    =      ϵ    ′    ϵ  =  (  y  −  x  β      )    ′    (  y  −  x  β  )  =      y    ′    y  −  2      β    ′        x    ′    y  +      β    ′        x    ′    x  β            ∂      S      (      β      )              ∂      β        =  0My question: how to get   −  2      x    ′    y  +  2      x    ′    x  β?

Royce Morrison · Accepted Answer

The sum of the squared errors can be written as            ‖      ϵ      ‖        2    =            ‖      Y      −      X      β      ‖        2    =  (  Y  −  X  β      )    T    (  Y  −  X  β  )  =      Y    T    Y  −      β    T        X    T    Y  −      Y    T    X  β  +      β    T        X    T    X  β  =            ‖      Y      ‖        2    −  2      Y    T    X  β  +            ‖      X      β      ‖        2  Then, finding the gradient             d                        ‖          ϵ          ‖                2                    d      β                  d                        ‖          ϵ          ‖                2                    d      β        =  −  2      X    T    Y  +  2      X    T    X  βReviewing term by term in that differentiation (since differentiation is linear operator!)            ‖      Y      ‖        2   does not depend on   β and becomes   0  2      Y    T    X  β is a is a sum where the       i          t      h       term is   2      β    i        Y    T        x    i   where       x    i   is the       i          t      h       row of   X. Therefore, the gradient of this term evaluates to   −  2      X    T    Y            ‖      X      β      ‖        2   can be differentiated using the product ruleIn general, the Jacobian of   f  (  x  )  =  A  x is       J    f    =      A    T

Yair Valentine · Accepted Answer

Instead of expanding     ε   before differentiating, do the differentiation first.                    ε                    =        X        β        −        Y                            λ                    =                  ε          T                ε                            d        λ                    =        2                  ε          T                          d          ε                =        2                  ε          T                                      (            X                        d            β            )                          =                              (            2                          X              T                        ε            )                    T                d        β                                                  ∂            λ                                ∂            β                                      =        2                  X          T                          ε                =        2                  X          T                                      (            X            β            −            Y            )                              To find the minimizer, set this gradient to zero and solve for   β                              X          T                X        β                    =                  X          T                Y                            β                    =                                            (                              X                T                            X              )                                      −              1                                            X            T                          Y                =                                      X            +                          Y            where       X    +   denotes the pseudoinverse of   X

Matrix derivative in multiple linear regression model

Answered question

Answer & Explanation

New Questions in Inferential Statistics