13 Nonlinear

8/13/2019 13 Nonlinear

1/63

Non-Linear Optimization

Distinguishing Features

Common Examples

EOQ

Balancing Risks

Minimizing Risk

15.057 Spring 03 Vande Vate 1


2/63

Hierarchy of Models

Network Flows

Linear Programs

Mixed Integer Linear Programs



3/63


A More Academic View

Network Flows

Linear Programs

Convex Optimization

Mixed Integer

LinearPrograms Non-Convex Optimization


4/63


A More Academic View

Networks & Linear Models

Convex Optimization

Integer ModelsNon-Convex Optimization


5/63

ConvexityThe Distinguishing FeatureSeparates Hard from Easy

Convex Combination

Weighted AverageNon-negative weights

Weights sum to 1



6/63

Convex Functions

Thefunctionlies

belowthe line

Convex Function

800

700

600

500

400

300

200

100

0

Examples?

Convexcombinations of the

values

0 5 10 15 20 25 3015.057 Spring 03 Vande Vate 6


7/63

Whats Easy

Find the minimum of a Convex Function

A local minimum is a global minimum



8/63

Convex Set

A set S is CONVEX if every convexcombination of points in S is also in S

The set of points above a convex functionConvex Function

800

700

600

500

400

300

200

100

0

0 5 10 15 20 25 30



9/63

Whats EasyFind the minimum of a Convex

SetFunction over (subject to) a Convex



10/63

Concave FunctionConcave Function

Thefunctionlies

ABOVEthe line

0

50

100

150

200

250

Examples?

5 10 15 20 25 3015.057 Spring 03 Vande Vate 10


11/63

Whats Easy

Find the maximum of a Concave

Set.Function over (subject to) a Convex



12/63

Academic Questions

Is a linear function convex or concave?

Do the feasible solutions of a linearprogram form a convex set?

program form a convex set?

12

Do the feasible solutions of an integer

15.057 Spring 03 Vande Vate


13/63

Ugly - Hard

Cost

Volume15.057 Spring 03 Vande Vate 13


14/63

Integer Programming is Hard

Bands

Coils

Bands Limit

Coils Limit

Production Capacity

Why?

3

2

1

1 2 3 415.057 Spring 03 Vande Vate 14


15/63

Review

Convex Optimization

Convex (min) or Concave (max) objective

Convex feasible region

Non-Convex Optimization

Stochastic Optimization

Incorporates Randomness



16/63

Agenda

Convex Optimization

Unconstrained Optimization

Constrained Optimization

Non-Convex Optimization

Convexification

Heuristics



17/63

Convex Optimization


If the partial derivatives exist (smooth)

find a point where the gradient is 0

Otherwise (not smooth)

find point where 0 is a subgradient



18/63

Unconstrained Convex

OptimizationSmooth

Find a point where the Gradient is 0Find a solution to f(x) = 0

Analytically (when possible)

Iteratively otherwise



19/63

Solving f(x) = 0Newtons Method

Approximate using gradientf(y) f(x) + (y-x)tHx(y-x)

Computing next iterate involves inverting HxQuasi-Newton Methods

Approximate H and update the approximationso we can easily update the inverse

(BFGS) Broyden, Fletcher, Goldfarb, Shanno15.057 Spring 03 Vande Vate 19


20/63

Line Search

Newton/Quasi-Newton Methods yielddirection to next iterate

1-dimensional search in this directionSeveral methods



21/63

Unconstrained Convex

Optimization

Non-smooth

Subgradient Optimization

Find a point where 0 is a subgradient



22/63


Whats a Subgradient

f(y) = f(x) - 2(y-x)

x

a0 is a subgradient if and only if ...1 x -2f(y) = f(x) + (y-x)

Like a gradient

f(y) f(x) +x(y-x) f(x) is a minimum point


23/63

Steepest Descent

If 0 is not a subgradient at x, subgradientindicates where to go

Direction of steepest descent

Find the best point in that directionline search



24/63

Examples

EOQ Model

Balancing Risk

Minimizing Risk



25/63

EOQHow large should each order beTrade-off

Cost of Inventory (known)

Cost of transactions (what?)Larger orders

Higher Inventory Cost

Lower Ordering Costs



26/63

The Idea

Increase the order size until theincremental cost of holding the last itemequals the incremental savings in ordering

costsIf the costs exceed the savings?

If the savings exceed the costs?



27/63

Modeling CostsQ is the order quantity

Average inventory level isQ/2

h*c is the Inv. Cost. in $/unit/year

Total Inventory Costh*c*Q/2

Last item contributes what to inventory

cost?h*c/2



28/63

Modeling CostsD is the annual demand

How many orders do we place?D/Q

Transaction cost is A per transactionTotal Transaction Cost

AD/Q



29/63

Total CostTotal Cost = h*cQ/2 + AD/Q

Total Cost

120

100

80

60

40

20

What kind of function?

0

0 10 20 30 40 50 60 70

Order Quantity



30/63

Incremental SavingsWhat does the last item save?

Savings of Last ItemAD/(Q-1) - AD/Q

[ADQ - AD(Q-1)]/[Q(Q-1)] ~ AD/Q2

Order up to the point that extra carryingcosts match incremental savings

h*c/2 = AD/Q2

Q2 = 2AD/(h*c)

Q = 2AD/(h*c)



31/63

Key Assumptions?

Known constant rate of demand



32/63


33/63

The EOQ Trade off

Known valuesAnnual Demand D

Product value c

Inventory carrying percentage h

Unknown transaction cost AFor each value of A

Calculate Q = 2AD/(h*c)

Calculate Inventory Investment cQ/2Calculate Annual Orders D/Q



34/63

The Tradeoff Benchmark

EOQ Trade off

$3,000

$2,500

$2,000

$1,500

$1,000

$500

InventoryInvestment

Where are you?

Where can you be?But wait!What prevents getting there?

$0

0 50 100 150 200 250 300

Orders Per Year



35/63

Balancing Risks



36/63


37/63


38/63


39/63

Theoretical vs EmpiricalEmpirical Distribution

Based on observationsValue 2 3 4 5 6 7 8 9 10 11 12

Number of Outcomes 1 2 1 5 3 9 8 3 3 1 -

Fraction of Outcomes 0.03 0.06 0.03 0.14 0.08 0.25 0.22 0.08 0.08 0.03 -

Theoretical DistributionBased on a model

Value 2 3 4 5 6 7 8 9 10 11 12

Fraction of Outcomes 0.03 0.06 0.08 0.11 0.14 0.17 0.14 0.11 0.08 0.06 0.03



40/63


41/63


42/63

Continuous vs Discrete

Discrete

Value of dice

Number of units sold

Continuous

Essentially, if we measure it, its discreteTheoretical convenience



43/63

ProbabilityDiscrete: Whats the probability we roll a12 with two fair die:

1/36Continuous: Whats the probability thetemperature will be exactly 72.00o F

tomorrow at noon EST?Zero!

Events: Whats the probability that the

temperature will be at least 72o Ftomorrow at noon EST?



44/63

Continuous DistributionStandard Normal Distribution0.45

0.4

0.35

0.3

0.25

0.2

0.15

0.1

0.05

Probability the random variableis greater than 2 is the area

under the curve above 2

0

-10 -8 -6 -4 -2 0 2 4 6 8 10



45/63

Total ProbabilityEmpirical, Theoretical, Continuous,

Discrete, Probability is between 0 and 1Total Probability (over all possibleoutcomes) is 1



46/63

Summary StatsThe Mean

Weights each outcome by itsprobability

AKA

Expected ValueAverage

May not even be possible

Example:Win $1 on Heads, nothing on Tails



47/63


48/63

VarianceNomal D istributions with Different Variances

0.45

0.4

0.35

0.3

0.25

0.2

0.15

0.1

0.05

Variance 1

Variance 9

0

-10 -8 -6 -4 -2 0 2 4 6 8 10



49/63

Std. Deviation

Variance is measured in units squaredThink sum of squared errors

Standard Deviation is the square rootIts measured in the same units as the

random variableThe two rise and fall together

Coefficient of VariationStandard Deviation/MeanSpread relative to the Average



50/63

Balancing RiskBasic Insight

Bet on the outcome of a variable processChoose a value

You pay $0.5/unit for the amount your bet

exceeds the outcomeYou earn the smaller of your value and theoutcome

Question: What value do you choose?



51/63

Similar to...

Anything you are familiar with?



52/63


53/63

The Idea

Balance the risks

Look at the last item

What did it promise?

What risk did it pose?

If Promise is greater than the risk?If the Risk is greater than thepromise?



54/63

Measuring Risk and ReturnRevenue from the last item

$1 if the Outcome is greater,$0 otherwise

Expected Revenue

$1*Probability Outcome is greater than our choice

Risk posed by last item$0.5 if the Outcome is smaller, $0 otherwise

Expected Risk

$0.5*Probability Outcome is smaller than our choice



55/63

Balancing Risk and RewardExpected Revenue

$1*Probability Outcome is greaterthan our choice

Expected Risk

$0.5*Probability Outcome is smallerthan our choice

How are probabilities Related?



56/63

Risk & Reward

Distribution

0

0.05

0.1

0.15

0.2

0.35

0.4

0.45

Prob. Outcome is smaller

Prob. Outcome is larger

Our choice

How are they

related?

0 2 4 6 8 10 1215.057 Spring 03 Vande Vate 56


57/63


58/63

Making the ChoiceDistribution0.45

0.4

0.35

0.3

0.25

0.2

0.15

0.1

0.05

Prob. Outcome is smaller

Our choice

2/3

Cumulative

Probability

0

0 2 4 6 8 10 12



59/63


Feasible Direction techniquesEliminating constraints

Implicit Function

Penalty Methods

Duality



60/63

Feasible Directions

Unconstrained OptimizationXStart at a point: x0Identify an^ improving direction: dFeasibleFind a best ^ solution in direction d: x + d

FeasibleRepeatA Feasible direction: one you can move inA Feasible solution: dont move too far.Typically for Convex feasible region



61/63


Penalty Methods

Move constraints to objective with penaltiesor barriers

As solution approaches the constraint the penaltyincreases

Example:

min f(x) => min f(x) + t/(3x - x2

)s.t. x2 3x

as x2 approaches 3x, penalty increases rapidly



62/63

Relatively reliable tools for

Quadratic objective

Linear constraints

Continuous variables



63/63

SummaryEasy Problems

Convex MinimizationConcave Maximization


Local gradient informationConstrained problems

Tricks for reducing to unconstrained or simply

constrained problemsNLP tools practical only for smaller problems

Documents

13 Nonlinear