Avoiding subtraction for finite difference with log and expI want to approximate the derivative of f(x)Finite difference f ′ ( x ) ≈ f ( x + h ) − f ( x ) h I was taught that the error from the subtraction is blown up for small h. This I can verify with MATLAB.Smartass methodSo I though maybe the following would fix the problem: f ′ ( x ) ≈ log ⁡ exp ⁡ ( f ( x + h ) ) exp ⁡ ( f ( x ) ) h However, I get the same relative errors. (Which also start increasing for h smaller than approx. ϵ m a c h / 2Why is this?

Question

Avoiding subtraction for finite difference with log and expI want to approximate the derivative of f(x)Finite difference      f    ′    (  x  )  ≈            f      (      x      +      h      )      −      f      (      x      )        h  I was taught that the error from the subtraction is blown up for small h. This I can verify with MATLAB.Smartass methodSo I though maybe the following would fix the problem:      f    ′    (  x  )  ≈            log      ⁡                                    exp            ⁡            (            f            (            x            +            h            )            )                                exp            ⁡            (            f            (            x            )            )                                h  However, I get the same relative errors. (Which also start increasing for h smaller than approx.       ϵ          m      a      c      h            /    2Why is this?

Kaeden Bishop · Accepted Answer

First of all, you should always be careful to choose   h with a very slight dependence on   x such that   x  ±  h is exactly machine representable. For instance, if   x  =  1, then you should at least round the   h you are tempted to use to the nearest multiple of       2          −      22       for 32-bit precision or       2          −      51       for 64-bit doubles.Then, as Manazoni points out, the symmetric formula is superior; it gets a precision of about two orders of magnitude better working with 64-bit numbers. You should choose an   h somewhere near   x      ϵ          1              /            3      , where   ϵ is the fractional machine precision (thus in 64-bit doubles,   h of roughly       10          −      5        x is good. The fractional error in the derivative should be about       ϵ          2              /            3      Yet better, unless you suspect really weird behavior near   x, is to use the next higher order symmetric formula      f          ′        (  x  )  =            f      (      x      −      2      h      )      −      4      f      (      x      −      h      )      +      4      f      (      x      +      h      )      −      f      (      x      +      2      h      )        6    +  O      (          h      4        f          ′      ′      ′      ′        (    x    )    )  using   h  ≈  x      ϵ          1              /            5      , which gives fractional accuracy of about       ϵ          4              /            5      Higher order methods begin to probe too sensitively for slightly irregular function behavior and begin to rely too heavily on accuracy of the function evaluations. Unfortunately, unlike the case for integration where there are adaptive step size methods, there is no robust method of choosing order and step size that reliably gives you the best achievable estimate of the derivative. But there are methods that are yet better than the one I showed here, see chapter 5.7 of Numerical recipes.

Nashorn0o · Accepted Answer

Your solution doesn&#039;t help as it is essentially the same thing.   log  ⁡            exp      ⁡      (      f      (      x      +      h      )      )              exp      ⁡      (      f      (      x      )      )       is not going to be a more accurate representation of   f  (  x  +  h  )  −  f  (  x  ) than evaluating it directly (on average, I would imagine it will be less accurate).There isn&#039;t really much you can do when you are working that close to machine error afaik. Practically speaking, you don&#039;t need to use such a tiny   h with reasonable functions; the practical solution is that when you are dividing by a small number, don&#039;t let that number be close to       ϵ          m      a      c      h

Avoiding subtraction for finite difference with log and exp I want to approximate the derivative of f(x) Finite difference f(x)~~(f(x+h)-f(x))/(h) I was taught that the error from the subtraction is blown up for small h.

Open question

Answer & Explanation

New Questions in Algebra II