L22 Numerical Methods part 2 Homework Review Alternate Equal Interval Golden Section Summary Test 4...

L22 Numerical Methods part 2

• Homework• Review• Alternate Equal Interval• Golden Section• Summary• Test 4

1

2

Problem 10.4

3

2 2 21 2 3 1 2 2 3: ( ) 2 2 2 2

( 3,10, 12) (1,2,3)

Determine whether is a descent direction. . 0

given f x x x x x x xand at

i e

xd x

dc d

1 2

2 1 3

3 2 *

2 2 2(1) 2(2) 6( *) 4 2 2 4(2) 2(1) 2(3) 16

4(3) 2(2) 164 2x

x xf x x x

x x

c x

1 1 2 2 3 3

03

6 16 16 10

12

6( 3) 16(10) 16( 12)18 160 19250 0

c d c d c d

T

c d

c d c d

Yes, descent direction

Prob 10.10

4

2 21 2 2 3

:( ) ( ) ( )

(4,8,4) (1,1,1)

Determine whether is a descent direction. . 0

givenf x x x xand at

i e

xd x

dc d

1 2

1 2 2 3

2 3 *

2( ) 2(1 1) 4( *) 2( ) 2( ) 2(1 1) 2(1 1) 8

2(1 1) 42( )x

x xf x x x x

x x

c x

1 1 2 2 3 3

04

4 8 4 8

4

4(4) 8(8) 4(4)16 64 1696 0

c d c d c d

T

c d

c d c d

No, not a descent direction

Prob 10.19

5

1 2

2 1 3

3 2 *

2 2 2(2) 2(4) 12( *) 4 2 2 4(4) 2(2) 2(10) 40

4(10) 2(4) 484 2

1212 40 48 40

4812( 12) 40( 40) 48( 48)

(144 1600 2304)4048

x

x xf x x x

x x

c x

c d

2 2 21 2 3 1 2 2 3

:

( ) 2 2 2 2( 12, 40, 48) (2,4,10)

What is the slope of the function at the point?Find the optimum step size.

given

f x x x x x x xand at

xd x

Slope

Prob 10.19 cont’d

6

( 1) ( ) ( )

( ) ( )

1 1 1

2 2 2

3 3 3( ) ( )

1

2

3

1

2

2

2 124 4010 48

2 ( 12) 2 124 ( 40) 4 4010 ( 48) 10 48

k k k

new old

new old

x x dx x dx x d

xxxxxx

x x d

2 2 21 2 3 1 2 2 3

2 2 2

( ) 2 2 2 2

(2 12 ) 2(4 40 ) 2(10 48 ) 2(2 12 )(4 40 ) 2(4 40 )(10 48 )

( ) 4048 25504 0* 4048 / 25504 0.15873

f x x x x x x x

f

x

Prob 10.19 cont’d

7

alpha 0.15872

x1 0.095358x2 -2.34881x3 2.38143

f(x) 10.75031

( 1) ( ) ( )

( ) ( )

1

2

3

1

2

2

2 124 0.15873 4010 48

2 0.15873( 12) 0.0954 0.15873( 40) 2.3510 0.15873( 48) 2.38

k k k

new oldxxx

xxx

x x d

Prob 10.30

8

( 1) ( ) ( )

( ) ( )

1

2

3

4

1

2

3

4

2221

4 23 2

2 ( 2)1 ( 2)4 ( 2)3 ( 2)

k k k

new oldxxxx

xxxx

x x d

2 2 2 2

2

( ) ((2 2 ) 1) ((1 2 ) 2) ((4 2 ) 3) ((3 2 ) 4)

( ) 16 16 4

f

f

2 2 2 21 2 3 4( ) ( 1) ( 2) ( 3) ( 4)

(2,1,4,3) ( 2,2, 2,2)( )

f x x x x

find f

xx d

The Search Problem

• Sub Problem AWhich direction to head next?

• Sub Problem BHow far to go in that direction?

9

( )kx

Search Direction… Min f(x): Let’s go downhill!

10

( *)Tf f x d

1( ) ( *) ( *)( *) ( *) ( *)( *)

2T Tf f f R x x x x x x x H x x x

1( * ) ( *) ( *)

2T Tf f f R x d x x d d H d

( ) ( )let ( *) then new oldor d x x x x * d x x d

( *) 0Tf x dDescent condition

( *)Tlet fc x

0c d

Step Size?

How big should we make alpha?Can we step too “far?”

i.e. can our step size be chosen so big that we step over the “minimum?”

11

12

Figure 10.2 Conceptual diagram for iterative steps of an optimization method.

We are hereWhich direction should we head?

Some Step Size Methods

• “Analytical”Search direction = (-) gradient, (i.e. line search)Form line search function f(α)Find f’(α)=0

• Region Elimination (“interval reducing”)Equal intervalAlternate equal intervalGolden Section

13

14

Figure 10.5 Nonunimodal function f() for 0

Nonunimodal functions

Unimodal if stay in locale?

Monotonic Increasing Functions

15

Monotonic Decreasing Functions

16

continous

17

Figure 10.4 Unimodal function f().

Unimodal functions

monotonic increasing then monotonic decreasing

monotonic decreasing then monotonic increasing

Some Step Size Methods

• “Analytical”Search direction = (-) gradient, (i.e. line search)Form line search function f(α)Find f’(α)=0

• Region Elimination (“interval reducing”)Equal intervalAlternate equal intervalGolden Section

18

19

Figure 10.3 Graph of f() versus .

Analytical Step size

( 1) ( )( ) ( + ) ( )k kf f f x x d

( )

( ) ( ) ( )

( 1) ( ) ( )

given , and letthen

old

new old k

k k k

d xx x dx x d

( 1) ( )( ) ( + ) ( )'( )=0

k kf f ff

x x dSlope of line

search=c d

Slope of line at fmin0c d

Analytical Step Size Example

20

2 21 2: ( ) ( 2) ( 1)

44

find optimal step size *and ( *)!

given f x x

and let and at

f

x

d c x

2 21 2

2 2

2

( ) ( 2) ( 1)

( ) ((4 4 ) 2) ((4 6 ) 1)( ) 52 52 13( ) 2(52 ) 52 0

* 1/ 2

f x x

fff

1

2

2 21 2

2 2

4 1/ 2( 4) 24 1/ 2( 6) 12

*1

( *) ( 2) ( 1)

(2 2) (1 1) 0

xx

f x x

x

x

1

2

2( 2) 2(4 2)2( 1) 2(4 1)

46

xx

d c

( 1) ( ) ( )

( ) ( )

1 1 1

2 2 2

1

2

4 ( 4) 4 44 ( 6) 4 6

k k k

new oldx x dx x dxx

x x d

2 2( ) ((4 4 ) 2) ((4 6 ) 1)( ) 2((4 4 ) 2)( 4) 2((4 6 ) 1)( 6) 0( *) (2 4 )( 4) (3 6 )( 6) 0

8 16 18 36 26 52 0* 26 / 52 1 / 2

fff

Alternative Analytical Step Size

21

( 1) ( )

( 1) ( 1) ( 1)

( 1) ( ) ( )

( 1)( )

( 1) ( )

( 1) ( )

( ) ( + ) ( )'( )=0( ) ( ) ( )

0

since( )

( ) 0

0

k k

k T k k

k k k

kk

k k

k k

f f ffdf f d

d d

d

df

x x d

x x x

xx x d

xd

x d

c d

( 1) ( ) ( )

1

2

4 ( 4) 4 44 ( 6) 4 6

k k k

xx

x x d

( 1) ( ) 04

4 8 , 6 126

4(4 8 ) 6(6 12 ) 016 32 36 72 052 104 0

52 / 104 1/ 2

k k

T

c d

( 1) 1

2

2( 2)2( 1)

2(4 4 2)2(4 6 1)

4 86 12

k xx

c

New gradient must be orthogonal to d for ' ( )=0f

Some Step Size Methods

• “Analytical”Search direction = (-) gradient, (i.e. line search)Form line search function f(α)Find f’(α)=0

• Region Elimination (“interval reducing”)Equal intervalAlternate equal intervalGolden Section

22

23

Figure 10.6 Equal-interval search process. (a) Phase I: initial bracketing of minimum. (b) Phase II: reducing the interval of uncertainty.

“Interval Reducing”Region elimination

“bounding phase”

Interval reductionphase”

( 1)( 1)

2

l

u

u l

qq

I

2 delta!

24

Successive-Equal Interval Algorithm

25

x f(x)-5.0000 22.0067-4.0000 18.0183-3.0000 14.0498-2.0000 10.1353-1.0000 6.36790.0000 3.00001.0000 0.71832.0000 1.38913.0000 10.08554.0000 40.59825.0000 130.4132

( ) 2 - 4 exp( )f x x x x f(x)

0.0000 3.00000.2000 2.42140.4000 1.89180.6000 1.42210.8000 1.02551.0000 0.71831.2000 0.52011.4000 0.45521.6000 0.55301.8000 0.84962.0000 1.3891

x f(x)1.2000 0.52011.2400 0.49561.2800 0.47661.3200 0.46341.3600 0.45621.4000 0.45521.4400 0.46071.4800 0.47291.5200 0.49221.5600 0.51881.6000 0.5530

x f(x)1.3600 0.4561931.3680 0.4554881.3760 0.4550341.3840 0.4548331.3920 0.4548881.4000 0.4552001.4080 0.4557721.4160 0.4566051.4240 0.4577021.4320 0.4590651.4400 0.460696

x lower -5x upper 5

delta 1

02

0.2

1.21.6

0.04

“Interval” of uncertainty

1.361.44

0.008

More on bounding phase I

Swan’s method

Fibonacci sequence

26

( 1) ( )

( 1) ( )

210

k k k

k k k

0

(1.618)k

jq

j

Successive Alternate Equal Interval

27

Assume bounding phase has found Min can be on either side of

But for sure its not in this region!

1

32 1

3 3

b l

b l u

I

I I

u land

b

Point values… not a line

Successive Alt Equal Int

28

ME 482 Optimal Design alt_eq_int.xls RJE 4/11/2012

Alternate Equal Interval Search for locating minimum of f(x)x lower 0.5x upper 2.618

Iteration xl 1/3(xu+2xl) 1/3(2xu+xl) xu Interval Optimal

1 x 0.5000000 1.2060000 1.9120000 2.6180000 2.118000 1.2060000f(x) 1.6487213 0.5160975 1.1186085 5.2362796 0.5160975

2 x 0.5000000 0.9706667 1.4413333 1.9120000 1.412000 1.4413333f(x) 1.6487213 0.7570370 0.4609938 1.1186085 0.4609938

3 x 0.9706667 1.2844444 1.5982222 1.9120000 0.941333 1.2844444 f(x) 0.7570370 0.4748826 0.5513460 1.1186085 0.47488264 x 0.9706667 1.1798519 1.3890370 1.5982222 0.627556 1.3890370 f(x) 0.7570370 0.5344847 0.4548376 0.5513460 0.45483765 x 1.1798519 1.3193086 1.4587654 1.5982222 0.418370 1.3193086

f(x) 0.5344847 0.4635997 0.4655851 0.5513460 0.4635997

( ) 2 - 4 exp( )f x x x

Requires two function evaluations per iteration

29

Figure 10.8 Initial bracketing of the minimum point in the golden section method.

Fibonacci Bounding

30

Figure 10.9 Graphic of a section partition.

Golden section

22

1,2

1 1 4(1)( 1)4

2 2(1)

1 5 1 2.236

2 20.618, 1.618

b b ac

a

2

leftside = rightside(1 )

[ ] (1 )[ ] (1 )

1 0

I II I

Golden Section Example

31

ME 482 Optimal Design alt_gold.xls RJE 4/11/2012

Golden Section Region Elimiation Search for locating minimum of f(x)x lower 0.5 1-τ= 0.381966x upper 2.618 τ = 0.618034

Iteration xl xL+(1-τ) I xL+ τ I xu Interval I Optimal

1 x 0.5 1.309004 1.808996 2.618 2.118 1.30900399f(x) 1.6487213 0.4664682 0.8683316 5.2362796 0.4664682

2 x 0.5 0.999992 1.309004 1.808996 1.308996 1.30900404f(x) 1.6487213 0.7182921 0.4664682 0.8683316 0.46646819

3 x 0.999992 1.309004 1.499984 1.808996 0.809004 1.30900401 f(x) 0.7182921 0.4664682 0.4816814 0.8683316 0.46646824 x 0.999992 1.1909719 1.309004 1.499984 0.499992 1.30900403 f(x) 0.7182921 0.5263899 0.4664682 0.4816814 0.466468195 x 1.1909719 1.309004 1.3819519 1.499984 0.3090121 1.38195187

f(x) 0.5263899 0.4664682 0.4548602 0.4816814 0.45486022

( ) 2 - 4 exp( )f x x x

Summary• General Opt Algorithms have two sub problems:

search direction, and step size

• Descent condition assures correct direction• For line searches…in local neighborhood…

we can assume unimodal!

• Step size methods: analytical, region elimin.• Region Elimination (“interval reducing”)

Equal interval Alternate equal interval Golden Section 32