Download Python Scientific lecture notes Release 2012.3 (EuroScipy - e

Python Scientific lecture notes, Release 2012.3 (EuroScipy 2012)
The conjugate gradient solves this problem by adding a friction term: each step depends on the two last values of
the gradient and sharp turns are reduced.
Table 13.3: Conjugate gradient descent
An ill-conditionned
non-quadratic function.
An ill-conditionned very
non-quadratic function.
Methods based on conjugate gradient are named with ‘cg’ in scipy. The simple conjugate gradient method to
minimize a function is scipy.optimize.fmin_cg():
>>> def f(x):
# The rosenbrock function
...
return .5*(1 - x[0])**2 + (x[1] - x[0]**2)**2
>>> optimize.fmin_cg(f, [2, 2])
Optimization terminated successfully.
Current function value: 0.000000
Iterations: 13
Function evaluations: 120
Gradient evaluations: 30
array([ 0.99998968, 0.99997855])
These methods need the gradient of the function. They can compute it, but will perform better if you can pass
them the gradient:
>>> def fprime(x):
...
return np.array((-2*.5*(1 - x[0]) - 4*x[0]*(x[1] - x[0]**2), 2*(x[1] - x[0]**2)))
>>> optimize.fmin_cg(f, [2, 2], fprime=fprime)
Optimization terminated successfully.
Current function value: 0.000000
Iterations: 13
Function evaluations: 30
Gradient evaluations: 30
array([ 0.99999199, 0.99997536])
Note that the function has only been evaluated 30 times, compared to 120 without the gradient.
13.2.3 Newton and quasi-newton methods
Newton methods: using the Hessian (2nd differential)
Newton methods use a local quadratic approximation to compute the jump direction. For this purpose, they rely
on the 2 first derivative of the function: the gradient and the Hessian.
13.2. A review of the different optimizers
251