Embed Size (px)
DESCRIPTION
opt101a
Citation preview
Presentation OverviewPresentation Overview Mechanical systems and modelingMechanical systems and modeling Fitness Fitness – what is it and what are the possibilities– what is it and what are the possibilities Back-fittingBack-fitting as a loaded gun – what are the issues and pitfalls as a loaded gun – what are the issues and pitfalls Near-optimal solutionsNear-optimal solutions and a high level overview of how its and a high level overview of how its
donedone Walk-forwardWalk-forward methodology methodology --- BREAK ------ BREAK --- Mechanics of the IO optimizerMechanics of the IO optimizer Three functionsThree functions
Detailed example of Detailed example of back-fitted optimizationback-fitted optimization Sensitivity analysisSensitivity analysis and how it predicts robustness - with example and how it predicts robustness - with example Detailed example of Detailed example of walk-forward optimizationwalk-forward optimization, and use of , and use of
sensitivity analysissensitivity analysis How to use it in your trading – some guidelinesHow to use it in your trading – some guidelines Q&AQ&A Preview of Optimization 201Preview of Optimization 201
Presentation Goal -Presentation Goal -
System vs. discretionary System vs. discretionary tradingtrading““ Dad – are all stops signs like mandatory, or are Dad – are all stops signs like mandatory, or are
some just like recommended?”some just like recommended?”
There are successful mechanical traders and There are successful mechanical traders and successful discretionary traderssuccessful discretionary traders
Both back-test, although differently, and over Both back-test, although differently, and over different time periodsdifferent time periods
Discretionary traders use the “computer Discretionary traders use the “computer between their ears”between their ears”
Discretionary traders tend to factor in more Discretionary traders tend to factor in more complex patterns, but don’t take all signalscomplex patterns, but don’t take all signals
Mechanical traders must take all signals to Mechanical traders must take all signals to achieve expected results – but most don’t take achieve expected results – but most don’t take all signalsall signals
Discretionary tradersDiscretionary traders
Discretionary traders process –Discretionary traders process – Patterns they articulatePatterns they articulate Patterns that they do not, OR can not Patterns that they do not, OR can not
articulatearticulate Patterns may involve price action Patterns may involve price action
alone, or other information such as alone, or other information such as markets, sectors, age of move, etc.markets, sectors, age of move, etc.
Fall into two categoriesFall into two categories SetupsSetups Projections of future effectsProjections of future effects
Mechanical system Mechanical system traderstraders RationaleRationale
Mechanical system can be back-testedMechanical system can be back-tested In theory, it removes emotions from tradingIn theory, it removes emotions from trading
Modern T/A programs have made Modern T/A programs have made development much easierdevelopment much easier
There are a number of traps waiting for a There are a number of traps waiting for a developer of trading systemsdeveloper of trading systems Back-fitting is one of the worst !Back-fitting is one of the worst !
Most system traders will exercise discretion Most system traders will exercise discretion thinking that they can improve on resultsthinking that they can improve on results
Why do we model, backtest, and Why do we model, backtest, and
optimizeoptimize - - ModelModel
To define an “edge” in timing, selection, money To define an “edge” in timing, selection, money managementmanagement
A model approximates historyA model approximates history Are we modeling an underlying crowd behavior, or a Are we modeling an underlying crowd behavior, or a
statistical relationship with no cause and effect ?statistical relationship with no cause and effect ? Storks nesting in post-WWII Copenhagen -> birth rateStorks nesting in post-WWII Copenhagen -> birth rate
Accuracy is required, but not exactnessAccuracy is required, but not exactness Back-testBack-test
To verify that edge, know the odd’sTo verify that edge, know the odd’s To test across different marketsTo test across different markets To provide confidence going forwardTo provide confidence going forward
OptimizeOptimize To maximize or minimize those factors that are most To maximize or minimize those factors that are most
importantimportant CAR, MDD, Sharpe, UPI, etc.CAR, MDD, Sharpe, UPI, etc. The combination is called “fitness”The combination is called “fitness”
What is fitness?What is fitness? We have to decide on what we are going to We have to decide on what we are going to
optimize onoptimize on It can be a single measure, or a weighted It can be a single measure, or a weighted
combination of measurescombination of measures Some typical candidatesSome typical candidates
CAR – compounded annual returnCAR – compounded annual return CAR / MDDCAR / MDD UPI - Ulcer Performance Index – penalize net. UPI - Ulcer Performance Index – penalize net.
volatility onlyvolatility only Sharpe ratio – neg. volatility will follow pos. volatilitySharpe ratio – neg. volatility will follow pos. volatility Combination measuresCombination measures
Fitness = CAR / Trades ^ 0.5 / MDD ^ 2 / UI ^ 0.5Fitness = CAR / Trades ^ 0.5 / MDD ^ 2 / UI ^ 0.5 Risk adjusted measures will often need Risk adjusted measures will often need
minimum hurdles and maximum limits - goalsminimum hurdles and maximum limits - goals Examples - minimum CAR, maximum number of tradeExamples - minimum CAR, maximum number of trade
Trading System Mining Trading System Mining Pitfalls-Pitfalls-
““ A fossicker is one who A fossicker is one who searches searches (unsystematically) for any (unsystematically) for any object by which to make object by which to make gain - to dig - to hunt - to gain - to dig - to hunt - to bustle about - or to bustle about - or to rummage...rummage... “ “
Back-fitting to history Back-fitting to history usually yields negative usually yields negative surprisessurprises
Back-testingBack-testing
Big QuestionBig Question – What percentage of the – What percentage of the back-test return, MDD, etc. should you back-test return, MDD, etc. should you expect going forward ?expect going forward ?
TODAYTODAY2003200320012001
BACKTEST AND BACKTEST AND OPTIMIZEOPTIMIZE
??????
The back-fitting sensitivity The back-fitting sensitivity problem -problem -
Other issues -Other issues -
Back-testingBack-testing Different market action vs. recent market Different market action vs. recent market
actionaction Test over bull, bear, sideways marketsTest over bull, bear, sideways markets
Number of signal samples - statistical Number of signal samples - statistical significance significance Issue for intermediate term systemsIssue for intermediate term systems
Survivorship biasSurvivorship bias Bad performing stocks fold, or mergeBad performing stocks fold, or merge Good funds’ record is boughtGood funds’ record is bought Bad funds record is buriedBad funds record is buried
Selection biasSelection bias Testing from current watch listsTesting from current watch lists
AI optimization -AI optimization - AI optimization offers many new capabilitiesAI optimization offers many new capabilities To date we have fitted as much data as To date we have fitted as much data as
possible and “hoped”possible and “hoped” Mechanization is one of the most important Mechanization is one of the most important
capabilitiescapabilities Eliminates biases - selection, event knowledge, etc.Eliminates biases - selection, event knowledge, etc.
Less sensitive solutions can be found for a Less sensitive solutions can be found for a point in timepoint in time
We can “time travel” to points in the past and We can “time travel” to points in the past and look at walk forward performancelook at walk forward performance
A “spliced” equity curve can be assembled to A “spliced” equity curve can be assembled to look at what would have happened if the look at what would have happened if the system had been traded forward OOS.system had been traded forward OOS.
Keywords –Keywords –
Near optimalNear optimal RobustRobust Out of sample, walk-forwardOut of sample, walk-forward Many of these concepts were Many of these concepts were
pioneered by Rober Pardopioneered by Rober Pardo
Near optimality -Near optimality -
Robustness -Robustness -
Rolling walk forward Rolling walk forward testingtesting
TODAYTODAY2002200219981998 20002000
Green segments represent equity that Green segments represent equity that COULD have been achieved !COULD have been achieved !
Anchored walk forward Anchored walk forward testingtesting
TODAYTODAY2002200219981998 20002000
Green segments represent equity Green segments represent equity that COULD have been achieved !that COULD have been achieved !
Walk forward issues Walk forward issues
Rolling walk forward –Rolling walk forward – How far back to back-test and optimize How far back to back-test and optimize
overover Attractive to look over recent historyAttractive to look over recent history Markets change when you least expectMarkets change when you least expect
Anchored walk forward –Anchored walk forward – Encompasses different marketsEncompasses different markets Compromises over those different Compromises over those different
environmentsenvironments
Walk forward efficiencyWalk forward efficiency Several approaches to walk forward efficiency including Several approaches to walk forward efficiency including
Pardo’sPardo’s One approach is compound annual return (CAR) of walk One approach is compound annual return (CAR) of walk
forward periods / CAR of solution optimized over entire forward periods / CAR of solution optimized over entire periodperiod
The following alternative, though, performs The following alternative, though, performs measurements in the same period as a kind of “apples to measurements in the same period as a kind of “apples to apples” approachapples” approach
Efficiency (WFE) -Efficiency (WFE) - Walk forward period / value for walk forward period when Walk forward period / value for walk forward period when
optimized through end of walk forward periodoptimized through end of walk forward period Example –Example –
LB = Optimize 1/2/1998 – 1/2/2002LB = Optimize 1/2/1998 – 1/2/2002 WF = Walk forward 1/2/2002 – 1/2/2003WF = Walk forward 1/2/2002 – 1/2/2003 TOT = Optimize 1/2/1998 – 1/2/2003TOT = Optimize 1/2/1998 – 1/2/2003 IDEAL = Segment of TOT for 1/2/2002 – 1/2/2003IDEAL = Segment of TOT for 1/2/2002 – 1/2/2003 WFE = WF / IDEALWFE = WF / IDEAL
Why Pardo -Why Pardo -
Design, Testing, and Optimization Design, Testing, and Optimization of Trading Systemsof Trading Systems - Robert Pardo - Robert Pardo
Originally written in 1992Originally written in 1992 His book is a good set of guidelinesHis book is a good set of guidelines Other authors such as Perry Other authors such as Perry
Kaufman have also written about Kaufman have also written about this topicthis topic
Pardo Cliff Notes -Pardo Cliff Notes - First – Pardo is a guideFirst – Pardo is a guide
Difference between necessary and sufficientDifference between necessary and sufficient Not A -> Not BNot A -> Not B A -> BA -> B
Most of Pardo’s guidelines are sufficientMost of Pardo’s guidelines are sufficient In other words, if you have them, you should In other words, if you have them, you should
have a robust system, BUT, is you don’t, you have a robust system, BUT, is you don’t, you may still have a robust system.may still have a robust system.
““Overfitting occurs when excessive attention is Overfitting occurs when excessive attention is paid to creating a curve or testing a trading model paid to creating a curve or testing a trading model with past data and not enough attention is paid to with past data and not enough attention is paid to judging the predictive value of the curve or judging the predictive value of the curve or trading model.”trading model.”
Much of Pardo’s work deals with over-fitting, how Much of Pardo’s work deals with over-fitting, how to detect it, and how to avoid itto detect it, and how to avoid it
Pardo’s operational Pardo’s operational recommendations -recommendations -
Know the point at which to abandon a systemKnow the point at which to abandon a system Models have a “shelf life”Models have a “shelf life”
Re-optimization can be too frequent Re-optimization can be too frequent OROR too too infrequentlyinfrequently
Systems must have a statistically significant Systems must have a statistically significant sample sizesample size Intermediate systems – as much data as is availableIntermediate systems – as much data as is available Swing systems – multiple markets vs. recent market Swing systems – multiple markets vs. recent market
debatedebate Monitor the equity curveMonitor the equity curve Trade the equity curve – trend line, MA, MDD, Trade the equity curve – trend line, MA, MDD,
etc.etc. Look for changes in trading consistency – Look for changes in trading consistency –
frequency, runs, distribution of profits and losses, frequency, runs, distribution of profits and losses, etc.etc.
What is the problem to What is the problem to solve?solve? Today we optimize to today, fit, and hope !Today we optimize to today, fit, and hope ! Many/most system models are too complex Many/most system models are too complex
for standard optimization techniquesfor standard optimization techniques We do not know how a model would have We do not know how a model would have
performed in the past as it was optimized performed in the past as it was optimized and traded forward.and traded forward.
““In theory there is no difference between In theory there is no difference between theory and practice. In practice there is.” - theory and practice. In practice there is.” - Yogi BerraYogi Berra
System off-roading -System off-roading -
A Solution -A Solution - What if we had ways to find a near optimal, What if we had ways to find a near optimal,
reasonably robust solution (if one exists) to reasonably robust solution (if one exists) to any system, at any point in time?any system, at any point in time?
And, we could GO BACK IN TIME see how And, we could GO BACK IN TIME see how such a system would do OOS in a walk forward such a system would do OOS in a walk forward test.test.
We could piece together those OOS tests over We could piece together those OOS tests over time to get a result that time to get a result that could have beencould have been achieved.achieved.
We would no longer be optimizing and hoping.We would no longer be optimizing and hoping. We could see how we might expect the system We could see how we might expect the system
to perform in the future.to perform in the future.
BREAKBREAK
Artifical Intelligence Artifical Intelligence OptimizationsOptimizations - - The idea is to use AI based search techniques that model The idea is to use AI based search techniques that model
techniques in nature to find a near-optimal, robust techniques in nature to find a near-optimal, robust solution.solution.
Solutions are expected to be the best that can be Solutions are expected to be the best that can be detected by intelligent searches in a reasonable detected by intelligent searches in a reasonable timeframe.timeframe.
Timeframe for full search is impractical ! (NP Complete)Timeframe for full search is impractical ! (NP Complete) RUTTR – 14 variables (9 without 4 levels and 1 lookback)RUTTR – 14 variables (9 without 4 levels and 1 lookback) 51 x 51 x 41 x 21 x 21 x 51 x 101 x 15 x 21 x 21 x 18 x 100 x 100 51 x 51 x 41 x 21 x 21 x 51 x 101 x 15 x 21 x 21 x 18 x 100 x 100
x 5 = x 5 = 1,442,204,034,769,858,500,000 combinations1,442,204,034,769,858,500,000 combinations 1,483,131,035,700,000 combinations without levels and 1,483,131,035,700,000 combinations without levels and lookbacklookback
2,351,4892,351,489 years at 1,728,000 per day years at 1,728,000 per day AI optimization targets 10-15 minute timeframeAI optimization targets 10-15 minute timeframe
AB and IO -AB and IO - AB lends itself to this approach because –AB lends itself to this approach because –
Has very fast backtesterHas very fast backtester Allows user functionsAllows user functions Support COM automation by controlling scriptSupport COM automation by controlling script
IO – product written by Fred Tonetti as AB add-onIO – product written by Fred Tonetti as AB add-on IO uses particle swarm and differential evolutionIO uses particle swarm and differential evolution Manipulates the Optimize() statements in the Manipulates the Optimize() statements in the
source and “feeds in” potential solution pointssource and “feeds in” potential solution points Exports performance results and iteratesExports performance results and iterates Animates the ongoing equity charts and produces Animates the ongoing equity charts and produces
statistical data and charts in browserstatistical data and charts in browser Also produces numerous CSV files of resultsAlso produces numerous CSV files of results Will “dim the lights”Will “dim the lights”
Setup for background IO Setup for background IO run -run -
AB tool used to run IO -AB tool used to run IO -
Goals and penalties -Goals and penalties - Goal assigns a minimum desired value to a Goal assigns a minimum desired value to a
fitness field name (AB optimization result fitness field name (AB optimization result column)column)
Values below the goal result in an adjustment Values below the goal result in an adjustment factor that is applied to fitness.factor that is applied to fitness.
Two types of scaling – multiplier and Two types of scaling – multiplier and exponentialexponential
Example of multiplierExample of multiplier Fitness is UPIFitness is UPI UPI of solution point = 2UPI of solution point = 2 CAR = 15, CAR goal = 20CAR = 15, CAR goal = 20 Fitness adjustment is .75 * 2 = 1.5Fitness adjustment is .75 * 2 = 1.5
SensitivitySensitivity
Definition - change in fitness with Definition - change in fitness with respect to “distance” from solutionrespect to “distance” from solution
Rationale – since change in inputs Rationale – since change in inputs can’t be accurately predicted, can’t be accurately predicted, approach is to measure change in approach is to measure change in output based on change in parametersoutput based on change in parameters
An average percentage is determined An average percentage is determined based on samplesbased on samples
The solution point fitness is adjusted The solution point fitness is adjusted by a sensitivity goalby a sensitivity goal
The sensitivity adjusted The sensitivity adjusted fitness -fitness -
IO directives for traditional IO directives for traditional optimization optimization
//IO: StatusWindow: N//IO: StatusWindow: N//IO: SaveCancelled: Y//IO: SaveCancelled: Y
// Remove in final "production" run// Remove in final "production" run//xIO: FitnessTime: 300//xIO: FitnessTime: 300//xIO: SenFinalTests: 1000//xIO: SenFinalTests: 1000
//IO: SenOptGoalPct: 85//IO: SenOptGoalPct: 85//IO: SenOptTests: 10//IO: SenOptTests: 10//IO: Fitness: UPI//IO: Fitness: UPI//IO: Goal: CAR: 15//IO: Goal: CAR: 15
//xIO: BegISDate: 01/01/1995//xIO: BegISDate: 01/01/1995//xIO: EndISDate: 01/01/2001//xIO: EndISDate: 01/01/2001//xIO: EndOSDate: 9/15/2005//xIO: EndOSDate: 9/15/2005//xIO: WFAuto: Anchored: Every: Year//xIO: WFAuto: Anchored: Every: Year
5 minute limit1000 samples for sensitivity
Disable the directive
Penalize result < 85% sensitivity10 samples to determine sensitiviyCAR goal to prevent high UPI, low CAR
Walk-forward directives disabled
Timing signal logic -Timing signal logic -BuyPriceBuyPrice = SellPrice = ShortPrice = CoverPrice = Close;= SellPrice = ShortPrice = CoverPrice = Close;SetTradeDelays( 0, 0, 0, 0 );SetTradeDelays( 0, 0, 0, 0 );
emalenemalen = Optimize( "emalen", 4, 2, 100, 1 );= Optimize( "emalen", 4, 2, 100, 1 );longfiltlongfilt = Optimize( "longfilt", 0.014, 0.001, 0.050, = Optimize( "longfilt", 0.014, 0.001, 0.050,
0.001 );0.001 );shortfiltshortfilt = Optimize( "shortfilt", 0.016, 0.001, 0.050, = Optimize( "shortfilt", 0.016, 0.001, 0.050,
0.001 );0.001 );
fundfund = Close;= Close;
fundemafundema = EMA( fund, emalen );= EMA( fund, emalen );
BuyBuy = Cross( fund, fundema * ( 1 + longfilt ) );= Cross( fund, fundema * ( 1 + longfilt ) );SellSell = Cross( fundema * ( 1 - shortfilt ), fund = Cross( fundema * ( 1 - shortfilt ), fund
););
BuyStateBuyState = Flip( Buy, Sell );= Flip( Buy, Sell );SellStateSellState = NOT BuyState;= NOT BuyState;
BuyBuy = ExRem( BuyState, SellState );= ExRem( BuyState, SellState );SellSell = ExRem( SellState, BuyState );= ExRem( SellState, BuyState );
ShortShort = 0; //Sell;= 0; //Sell;CoverCover = 0; //Buy;= 0; //Buy;
SetOption( "ReverseSignalForcesExit", True );SetOption( "ReverseSignalForcesExit", True );
No needed in this case – but always safe
Turns off short trading
Graphical signal logic -Graphical signal logic -
Note that area between green and red requires state logic
State and boolean signal State and boolean signal logic -logic - Area between red and green can’t be Area between red and green can’t be
described by Boolean logicdescribed by Boolean logic Fund is in that area AFTER BUY and AFTER SELLFund is in that area AFTER BUY and AFTER SELL
Impulse signal occurs when fund crossed up Impulse signal occurs when fund crossed up thru green or down thru red EMA’sthru green or down thru red EMA’s
Design patternDesign pattern These impulses are switched into state by the Flip() These impulses are switched into state by the Flip()
functionfunction Flip() also resolves combinations of signalsFlip() also resolves combinations of signals AB’s trading engine want impulse form which is AB’s trading engine want impulse form which is
obtained with the ExRem() functionobtained with the ExRem() function
IO Startup Screen -IO Startup Screen -
Next – IO Optimizer in Next – IO Optimizer in detaildetail
Traditional optimization Traditional optimization --
Portfolio indicator -Portfolio indicator -
Current drawdown
Days in drawdown
Regression line
Reference ticker
Equity & signals
Parameter sensitivity Parameter sensitivity distribution -distribution -
Report of traditional Report of traditional backtest -backtest -
1 year walk forward 1 year walk forward optimization -optimization -
Walk forward vs. traditional Walk forward vs. traditional from 2000 -from 2000 -
Walk forward vs. traditional Walk forward vs. traditional from 1996 -from 1996 -
Comparison of walk-forward to Comparison of walk-forward to traditional -traditional -
Primary difference is that the 2003 run-Primary difference is that the 2003 run-up was not captured by the walk-up was not captured by the walk-forwardforward
Walk-forward actually did better in Walk-forward actually did better in 2000, and 1997-19982000, and 1997-1998
The traditional back-fit must make The traditional back-fit must make compromises to handle all periodscompromises to handle all periods
But the walk-forward is more sensitive But the walk-forward is more sensitive to changes in the early yearsto changes in the early years
Note differences in number of tradesNote differences in number of trades
Additional IO Additional IO functionality -functionality - Perform sensitivity testing on a given systemPerform sensitivity testing on a given system Invoke IO from the command line and run Invoke IO from the command line and run
batch directive filesbatch directive files Process user defined metricsProcess user defined metrics Turn off sensitivity testing of individual Turn off sensitivity testing of individual
variablesvariables Optimization of groups of variablesOptimization of groups of variables Walk forward on every signalWalk forward on every signal Many more options that are documented in the Many more options that are documented in the
IO.DOC fileIO.DOC file
How to use in your How to use in your trading -trading - Model your trading ideasModel your trading ideas
Don’t incorporate hidden fixed assumptions/filtersDon’t incorporate hidden fixed assumptions/filters FilteredEMA example is a stop and reverse systemFilteredEMA example is a stop and reverse system A system that went flat might allow for better filters and signals A system that went flat might allow for better filters and signals
with less retracementwith less retracement Utilize Optimize() statements for all system parameters – Utilize Optimize() statements for all system parameters –
not just indicator parametersnot just indicator parameters For example - PositionSize, WorstRankHeld, ApplyStops, etc.For example - PositionSize, WorstRankHeld, ApplyStops, etc.
Perform IS (in sample) over several types of marketsPerform IS (in sample) over several types of markets Perform a walk-forward analysis Perform a walk-forward analysis
Now, there is no reason not to – we no longer have to fit and Now, there is no reason not to – we no longer have to fit and hopehope
Use a problematic period – for example, 2001-2002Use a problematic period – for example, 2001-2002 Use common sense – Use common sense –
Don’t re-optimize too frequentlyDon’t re-optimize too frequently Don’t have short look-back periodsDon’t have short look-back periods Expect to iterate your modelExpect to iterate your model
Look at it as improving your odds for successLook at it as improving your odds for success
Conclusions -Conclusions - The example used was a teaching example, not a The example used was a teaching example, not a
practical systempractical system IO allows optimization of previously intractable IO allows optimization of previously intractable
systemssystems But, it’s value is in allowing walk forward But, it’s value is in allowing walk forward
optimization of any systemoptimization of any system This allows you to see how a system would have This allows you to see how a system would have
performed in the pastperformed in the past It allows you to see how it may perform in the It allows you to see how it may perform in the
future and to assess its stabilityfuture and to assess its stability You may find results you didn’t want to findYou may find results you didn’t want to find
Some high performance systems may not walk forward wellSome high performance systems may not walk forward well But, others willBut, others will
“ “ The future ain't what it used to be “ – Yogi BerraThe future ain't what it used to be “ – Yogi Berra Remember, it is like going …Remember, it is like going …
IO - IO -
Questions and AnswersQuestions and Answers
Preview of Optimization Preview of Optimization 201 -201 - Knowledge of AFL programming will be neededKnowledge of AFL programming will be needed Construction and optimization of timing signalsConstruction and optimization of timing signals
Level vs. stateLevel vs. state Using IO to turn signal components on and offUsing IO to turn signal components on and off
Explore which components add valueExplore which components add value Optimization of ticker selection and weightsOptimization of ticker selection and weights
Assess portfolio mixAssess portfolio mix IO optimization of rotational systemsIO optimization of rotational systems
Special case of rotational system = walk-forward systemSpecial case of rotational system = walk-forward system Selecting look-back and walk-forward periodsSelecting look-back and walk-forward periods Fine-tuning the directives and optionsFine-tuning the directives and options Minimizing run timeMinimizing run time Design patternsDesign patterns Why system design is partly a science and partly an Why system design is partly a science and partly an
artart
![[WEBINAR] Outbrain Campaign Optimization 101](https://img.pdfslide.us/doc/110x75/587de2301a28abaf6b8b5a79/webinar-outbrain-campaign-optimization-101.jpg)
