Neural Networks: Reduction to Practice R obert J. M arks II Baylor University CIA L ab School of Engineering

Neural Neural Networks:Networks:

Reduction to Reduction to PracticePractice RRobert obert JJ. . MMarks arks IIII

Baylor UniversityBaylor University

CIA LCIA Lab ab School of EngineeringSchool of Engineering

In 1995, DAILY rates wereIn 1995, DAILY rates were• 6 Evolutionary Computation 6 Evolutionary Computation Papers Papers Per Day,Per Day,• 11 Fuzzy Papers Per Day,11 Fuzzy Papers Per Day,• 20 A.I. Papers Per Day,20 A.I. Papers Per Day,• 25 Neural Networks Papers 25 Neural Networks Papers Per Day &Per Day &• 34 Computational 34 Computational Intelligence Papers Intelligence Papers Per DayPer Day

Few are perishing ...Few are perishing ...

““The image which is portrayed is of the ability to perform The image which is portrayed is of the ability to perform

magically well by the incorporation of `new age’ technologiesmagically well by the incorporation of `new age’ technologies

Professor Bob Bitmead,Professor Bob Bitmead, IEEE Control Systems MagazineIEEE Control Systems Magazine, ,

June 1993, p.7.June 1993, p.7.<[email protected]> <[email protected]>

< http://keating.anu.edu.au/~bob/>< http://keating.anu.edu.au/~bob/>

Continued Continued ControversyControversy

of fuzzy logic, neural networks,of fuzzy logic, neural networks,

... approximate reasoning, and ... approximate reasoning, and

self-organization in the face of self-organization in the face of

dismal failure of traditional dismal failure of traditional

methods. This is pure unsupported methods. This is pure unsupported

claptrap which is pretentious claptrap which is pretentious

and idolatrous in the extreme, and idolatrous in the extreme,

and has no place in scientific and has no place in scientific

literature.”literature.”

The Wisdom of The Wisdom of Experience ... ???Experience ... ???

“(Fuzzy theory’s) delayed exploitation outside Japan teaches several lessons. ...(One is) the traditional intellectualism in engineering research in general and the cult of analyticity within control system engineering research in particular.” E.H. Mamdami, 1975 father of fuzzy control (1993).

"All progress means war with society." George Bernard Shaw

"If we knew what it was "If we knew what it was we were doing, it would we were doing, it would not be called research, not be called research,

would it?"would it?"

A.EinsteinA.Einstein

““In theory, In theory, theory and theory and

reality are the reality are the same. same.

In reality, they In reality, they are not.”are not.”

Better Than Average Better Than Average (?)(?)

U.S. News & World ReportU.S. News & World Report says that a poll of university says that a poll of university professors found that 94% professors found that 94% of the respondents thought of the respondents thought

that they were better at that they were better at their jobs than their their jobs than their average colleague. average colleague.

U.S. News & World Report 16 U.S. News & World Report 16 Dec 96 p26Dec 96 p26

Steel Industry: Steel Industry: Cold Rolling Mill ProcessCold Rolling Mill Process

Cold rolling mill flattens a steel strip Cold rolling mill flattens a steel strip to a desired thicknessto a desired thickness

Cho, Cho & Yoon, “ReliableCho, Cho & Yoon, “ReliableRoll Force Prediction in Cold MillRoll Force Prediction in Cold Mill

Using Multiple Neural Metworks”,Using Multiple Neural Metworks”,IEEE TNN, July 1997.IEEE TNN, July 1997.

Cold Rolling Mill...Cold Rolling Mill... Neural Network predicts the roll force 30% to

50% better. (Pohang Iron & Steel Co., Korea)

Cho, Cho & Yoon, “Reliable Roll Force Prediction in Cold MillCho, Cho & Yoon, “Reliable Roll Force Prediction in Cold MillUsing Multiple Neural Metworks”, IEEE TNN, July 1997.Using Multiple Neural Metworks”, IEEE TNN, July 1997.

Intelligent Control for a Steel PlantIntelligent Control for a Steel Plant Coating Process Control Controlling Alloying Thermal Cycle Induction Furnace Control

Sollac & CRAN (Research Centre for Automatic Control of Nancy)Bloch, Sirou & Fatrez, “Neural Intelligent Control for a Steel Plant”, IEEE TNN, July, 1997.

Estimation of induction temperature

Robert Novak syndicated columnRobert Novak syndicated columnWashington, February 18, 1996Washington, February 18, 1996

UNDECIDED BOWLERSUNDECIDED BOWLERS

““President Clinton’s pollsters have identified the voters President Clinton’s pollsters have identified the voters who will determine whether he will be elected to a second who will determine whether he will be elected to a second term: two-parent families whose members bowl for term: two-parent families whose members bowl for recreation.”recreation.”

““Using a technique they call the ‘neural network,’ Clinton Using a technique they call the ‘neural network,’ Clinton advisors contend that these family bowlers are the advisors contend that these family bowlers are the quintessential undecided voters. Therefore, these are the quintessential undecided voters. Therefore, these are the people who must be targeted by the president.”people who must be targeted by the president.”

A Political ApplicationA Political Application

““A footnote: Two decades ago, A footnote: Two decades ago, Illinois Democratic Gov. Dan Illinois Democratic Gov. Dan

Walker campaigned heavily in Walker campaigned heavily in bowling alleys in the belief he bowling alleys in the belief he

would find swing voters there. would find swing voters there. Walker had national political Walker had national political

ambitions but ended up in federal ambitions but ended up in federal prison.”prison.”

Robert Novak syndicated columnRobert Novak syndicated columnWashington, February 18, 1996Washington, February 18, 1996(continued)(continued)

Plasma Etch Process ControlPlasma Etch Process Control

Controls dielectric etcher drift > 50 time variant variables

DEC & NeuralWare

Card, Sniderman & Klimasaukas, “Dynamic Neural Control

for a Plasma Etch Process”, IEEE TNN, July 1997.

Modelling the EnvironmentModelling the EnvironmentNeural networks are used to monitor the interactions

between ozone pollution, climate & crop ozone sensitivity

United Nations Economic Commission for Europe

Roadnight, Balls, Mills & Palmer-Brown, “Modelling Complex Environmental Data”, IEEE TNN, July 1997.

Liu & Sin, “Fuzzy Neural Networks for Machine Maintenance in MassLiu & Sin, “Fuzzy Neural Networks for Machine Maintenance in MassTransit Railway System, IEEE TNN, July 1997.Transit Railway System, IEEE TNN, July 1997.

Train Ticket Machine Train Ticket Machine Maintenance in Hong KongMaintenance in Hong Kong

Why?Why? 2.45 million people daily. 2.45 million people daily. Increase 1% each yearIncrease 1% each year Machine breakdowns cause delays in Machine breakdowns cause delays in

serviceservice

Hong Kong Mass Transit Railway Hong Kong Mass Transit Railway Corporation SystemCorporation System

Liu & Sin, “Fuzzy Neural Networks for Machine Maintenance in MassLiu & Sin, “Fuzzy Neural Networks for Machine Maintenance in MassTransit Railway System, IEEE TNN, July 1997.Transit Railway System, IEEE TNN, July 1997.

Neural Networks in Neural Networks in Telecommunication SoftwareTelecommunication Software

Assesses reliability telecommunications software. > 13 million code lines.

Failure prone software modules identified.

Khoshgoftaar, Allen, Hudepohl & Aud, “Application of Neural Networks to Software Quality Modeling of a Very Large

Telecommunications System”, IEEE TNN, July, 1997

Nortel & Bell CanadaFlorida Atlantic U

AvionicsAvionics

ADALINE is used to optimize the engine control of Concord.ADALINE is used to optimize the engine control of Concord.

Who: Alan Gerber, University of London (QueenWho: Alan Gerber, University of London (QueenMary College) late 60’s. Mary College) late 60’s.

Source: Professor Igor AleksanderSource: Professor Igor AleksanderHead of Neural Systems Engineering (EEE Department)Head of Neural Systems Engineering (EEE Department)

Pro-Rector (External)Pro-Rector (External)Imperial College, LondonImperial College, London

Pattern Recognition in AerospacePattern Recognition in Aerospace

The The Boeing Airplane CompanyBoeing Airplane Company uses an ART-1 neural uses an ART-1 neural network system (NIRS) for the identification and network system (NIRS) for the identification and retrieval of 2-D and 3-D representations of engineering retrieval of 2-D and 3-D representations of engineering designs.designs.

Avoids redesign of existing parts and toolsAvoids redesign of existing parts and tools Production solid model data base > 55,000 entriesProduction solid model data base > 55,000 entries 2-D data base > 95,000 entries2-D data base > 95,000 entries

S.D.G. Smith, R. Escobedo, Michael Anderson, T.P. Caudell, “A Deployed Engineering S.D.G. Smith, R. Escobedo, Michael Anderson, T.P. Caudell, “A Deployed Engineering Design Retrieval System Using Neural Networks”, IEEE TNN, July 1997.Design Retrieval System Using Neural Networks”, IEEE TNN, July 1997.

Neural Networks for Police Classification

The Democrats held their 1996 presidential The Democrats held their 1996 presidential convention in Chicago. convention in Chicago.

Twenty six years before, in 1968, the Chicago Twenty six years before, in 1968, the Chicago presidential convention was marred by violent presidential convention was marred by violent clashes between demonstrators and police. A clashes between demonstrators and police. A presidential commission called the conflict a presidential commission called the conflict a “police riot”. “police riot”.

In 1968, Richard J. Daley was mayor of Chicago. In 1968, Richard J. Daley was mayor of Chicago. In 1996, his son, Richard M. Daley, was mayor. In 1996, his son, Richard M. Daley, was mayor.

In order to belay a repeat incident at the 1996 In order to belay a repeat incident at the 1996 convention, the Internal Affairs Department of the convention, the Internal Affairs Department of the Chicago Police Department used a Chicago Police Department used a neural neural networknetwork to classify `bad cops’ who might provoke to classify `bad cops’ who might provoke conflictconflict..

Scientific American , December 1994, (p.44).Scientific American , December 1994, (p.44).

““THE (NEURAL NETWORK) PROGRAM THE (NEURAL NETWORK) PROGRAM FORECASTS WHETHER EACH OF THE FORECASTS WHETHER EACH OF THE 12,500 OFFICERS ON THE FORCE IS 12,500 OFFICERS ON THE FORCE IS LIKELY TO BEHAVE IN A MANNER LIKELY TO BEHAVE IN A MANNER

SIMILAR TO NEARLY 200 COLLEAGUES SIMILAR TO NEARLY 200 COLLEAGUES WHO WERE DISMISSED OR RESIGNED WHO WERE DISMISSED OR RESIGNED UNDER INVESTIGATION DURING THE UNDER INVESTIGATION DURING THE

LAST FIVE YEARS FOR ACTIONS LAST FIVE YEARS FOR ACTIONS RANGING FROM INSUBORDINATION RANGING FROM INSUBORDINATION

CRIMINAL MISCONDUCT.”CRIMINAL MISCONDUCT.”

A total of 91 officers were identified. They A total of 91 officers were identified. They

were to enroll in a counseling program. The were to enroll in a counseling program. The

neural network results, though, were chal-neural network results, though, were chal-

lenged by the police union. lenged by the police union.

The neural network, as a The neural network, as a

““black box”, contained no black box”, contained no

causal mechanism to causal mechanism to

specify the reason or specify the reason or

reasons the classification reasons the classification

of potential ‘bad cop’ was of potential ‘bad cop’ was

made. The neural network made. The neural network

lacked an lacked an explanation facilityexplanation facility..

Power System Dynamic Power System Dynamic Security AssessmentSecurity Assessment

Problem: In a power system, Problem: In a power system, what contingencies may causewhat contingencies may causepower system violations andpower system violations andstability? System operator aid.stability? System operator aid.

i.e.i.e. brownoutsbrownouts & & blackoutsblackouts!!

The describing coupled set ofThe describing coupled set ofnonlinear differential equations nonlinear differential equations are computationally intensive.are computationally intensive.

Neural Net SolutionNeural Net Solution ... ...

Train a neural network to emulate the complex equations.

Development & Use: B.C. Hydro &

1293 buses

Hydro Quebec 963 buses

Reference: Mansour, Vaahedi & El-Sharkawi, IEEE TNN, July 1997

VAR FlowVAR Flow

A neural network was used A neural network was used to control the VAR flow to to control the VAR flow to

Vancouver Island,Vancouver Island,

CanadaCanada

Done by: Geoff NeilyDone by: Geoff Neily

Short Term Load Short Term Load ForecastingForecasting

•Problem: Forecast the power demand of a givenProblem: Forecast the power demand of a givengeographical area.geographical area.

�If under - expensive power must If under - expensive power must � be purchased elsewherebe purchased elsewhere�If over - must sell power in an uncertain marketIf over - must sell power in an uncertain market

•Features: current temperature, forecaster temperatureFeatures: current temperature, forecaster temperatureday of week (holiday?), current load, humidityday of week (holiday?), current load, humidity

ANNSTLF:ANNSTLF:Artificial Neural Network Short-Term Load ForecasterArtificial Neural Network Short-Term Load Forecaster

Actual (solid) vs. forecast (dashed line) forActual (solid) vs. forecast (dashed line) fora seven day interval.a seven day interval.

(X 10(X 1044 MW vs. hours) MW vs. hours)

Khotanzad, Afkhami-Rohani, Abaye & Matatukulam, “ANNSTLF - A Neural Network Based Electric Load Forecasting System” IEEE TNN, July 1997.

1.1. Alabama Electric CooperativeAlabama Electric Cooperative

2.2. Allegheny Power SystemAllegheny Power System

3. BC Hydro (Canada)3. BC Hydro (Canada)

4.4. Bonneville Power Administration (WA)Bonneville Power Administration (WA)

5.5. Buckeye Power Inc. Buckeye Power Inc.

6.6. Central & Southwest Corp.Central & Southwest Corp.

7.7. City Public Service of San AntonioCity Public Service of San Antonio

8.8. Detroit EdisonDetroit Edison

9. Entergy (Lousiana)9. Entergy (Lousiana)

10. Houston Lighting and Power Company10. Houston Lighting and Power Company

11. Idaho Power 11. Idaho Power

12. Illinois Power Company12. Illinois Power Company

Who is forecasting loads using Who is forecasting loads using ANNSTLFANNSTLF??

Khotanzad, Afkhami-Rohani, Abaye & Matatukulam, “ANNSTLF - A Neural Network Based Electric Load Forecasting System” IEEE TNN, July 1997.

13. Kansas City Power & Light 13. Kansas City Power & Light

14. Kentucky Utilities Company14. Kentucky Utilities Company

15. Madison Gas & Electric 15. Madison Gas & Electric

16. Metropolitan Edison16. Metropolitan Edison

17. Nevada Power Company17. Nevada Power Company

18. New England Power Exchange18. New England Power Exchange

19. North East Utilities19. North East Utilities

20. Northern Indiana Public Service Company20. Northern Indiana Public Service Company

21. Ottertail Power Company21. Ottertail Power Company

22. PECO Energy (PA)22. PECO Energy (PA)

23. Pennsylvania Power & Light Company23. Pennsylvania Power & Light Company

24. Potomac Electric Power Company (WA DC)24. Potomac Electric Power Company (WA DC)

25. PJM Interconnection (PA)25. PJM Interconnection (PA)

26. Public Service Electric & Gas (New Jersey)26. Public Service Electric & Gas (New Jersey)

27. Rochester Gas & Electric27. Rochester Gas & Electric

28. Salt River Project (Arizona)28. Salt River Project (Arizona)

29. San Diego Gas & Electric29. San Diego Gas & Electric

30. Southern California Edison30. Southern California Edison

31. Southern Company Services (Alabama)31. Southern Company Services (Alabama)

32. Tennessee Valley Authority32. Tennessee Valley Authority

33. Texas Utilities Electric33. Texas Utilities Electric

34. Western Area Power Administration (CA)34. Western Area Power Administration (CA)

35. Wisconsin Power & Light35. Wisconsin Power & Light

Wine Cork ClassificationWine Cork Classification

88thth Class: holes Class: holes cracks cracks bugs bugs

First ClassFirst Class 33rdrd Class Class 5 5thth Class Class

Chang, Han, Valverde, Griswald, Duque-Carrillo & Sanchez-Sinencio, “Cork Quality Classification System Using a Unified Image Processing Fuzzy-Neural Methodology”, IEEE TNN, July, 1997.

Cork Classification Steps...Cork Classification Steps... Morphological filtering Contour extraction & following Fuzzy-neural network classifier Result: 6.7% rejection vs. 40% traditional

Chang, Han, Valverde, Griswald, Duque-Carrillo & Sanchez-Sinencio, “Cork Quality Classification System Using a Unified Image Processing Fuzzy-Neural Methodology”, IEEE TNN, July, 1997.

Cork image and corresponding contours

A Random Walk A Random Walk Down Wall StreetDown Wall Street

W.W. Norton & Co, NY 1973W.W. Norton & Co, NY 1973Sixth Edition, 1996Sixth Edition, 1996

Finance, Neural Nets & Finance, Neural Nets & Dart ThrowingDart Throwing

Burton G. Malkiel, PrincetonBurton G. Malkiel, Princeton

Advice ...Advice ..."Stocks have reached what looks like a "Stocks have reached what looks like a

permanently high plateau." permanently high plateau." Irving Fisher, Professor of Economics, Yale Irving Fisher, Professor of Economics, Yale

University, 1929.University, 1929.

““October. This is one of the peculiarly dangerous October. This is one of the peculiarly dangerous months to speculate in stocks. The others are months to speculate in stocks. The others are

July, January, September, April, November, May, July, January, September, April, November, May, March, June, December, August, and February,” March, June, December, August, and February,”

Mark TwainMark Twain

Factor Models for Factor Models for

Tactical Asset AllocationTactical Asset Allocation� Factor models are widely used in portfolio Factor models are widely used in portfolio management. Performance differentials management. Performance differentials between the main asset classes (bonds vs between the main asset classes (bonds vs equities) can be explained in terms of changes in equities) can be explained in terms of changes in fundamental economic fundamental economic and financial variables.and financial variables.� This project uses neural This project uses neural networks instead of regression networks instead of regression analysis to model relative analysis to model relative performance between the mainperformance between the main asset classes on the basis of asset classes on the basis of their exposure to a set of (17) their exposure to a set of (17) economic and financial factors.economic and financial factors. Dr. Apostolos-Paul Refenes, London Dr. Apostolos-Paul Refenes, London

Business SchoolBusiness School

Factor Models for Tactical Asset Allocation...Factor Models for Tactical Asset Allocation...

� The neural models significantly The neural models significantly outperform multiple linear regression in outperform multiple linear regression in terms of forecasting accuracy. terms of forecasting accuracy. � Initially for the UK markets and at a later Initially for the UK markets and at a later stage at the international and global stage at the international and global level, portfolios are reset on a monthly level, portfolios are reset on a monthly basis. basis. � Non linear methods for variable Non linear methods for variable selection, and for analysing the selection, and for analysing the sensitivity of differential returns to sensitivity of differential returns to changes in the independent variables changes in the independent variables have been developed for thishave been developed for this..

In use, since 1995, by Postel (Hermes) In use, since 1995, by Postel (Hermes) Investment Management FOR THE Investment Management FOR THE CHIEF ECONOMISTS OFFICE AT HERMESCHIEF ECONOMISTS OFFICE AT HERMES

Dr. Apostolos-Paul Refenes, London Dr. Apostolos-Paul Refenes, London

Business SchoolBusiness School

Arbitrage models are Arbitrage models are finding increasing use in finding increasing use in tactical asset allocation tactical asset allocation as an alternative to as an alternative to factor models. factor models.

The basic idea is to The basic idea is to exploit short-term pricing exploit short-term pricing anomalies between the anomalies between the different asset classes.different asset classes.

Dr. Apostolos-Paul Refenes, London Business SchoolDr. Apostolos-Paul Refenes, London Business School

Arbitrage Models for Arbitrage Models for Tactical Asset Tactical Asset

AllocationAllocation

This is analogous to a hedging This is analogous to a hedging strategy designed to exploit short-strategy designed to exploit short-term over reactions to external term over reactions to external events. A model for the UK, events. A model for the UK, exploiting daily pricing anomalies exploiting daily pricing anomalies between equities and gilts was between equities and gilts was completed on January 1995. completed on January 1995.

A simple long/short trading strategy A simple long/short trading strategy is used to achieve excess returns of is used to achieve excess returns of 30% better than buy-and-hold and up 30% better than buy-and-hold and up to 100% better when adjusted for to 100% better when adjusted for risk.risk.

Societe Generale. IN USE SINCE 1995 - INITIALLY AT Societe Generale. IN USE SINCE 1995 - INITIALLY AT SOCGEN NOW AT T. K. HOARE BROKERS. T. K. SOCGEN NOW AT T. K. HOARE BROKERS. T. K.

HOARE BROKERSHOARE BROKERS

Arbitrage Models for Tactical Asset Arbitrage Models for Tactical Asset Allocation...Allocation...


Nonlinear Cointegration in Nonlinear Cointegration in European Equity FuturesEuropean Equity Futures

A nonlinear co-integration model of the A nonlinear co-integration model of the FTSE with a basket of European indices was FTSE with a basket of European indices was developed on daily data. developed on daily data.

The residuals of the cointegration are The residuals of the cointegration are modelled as a nonlinear function of modelled as a nonlinear function of exogenous variables (exogenous variables (e.g.e.g. interest rate interest rate volatility, oil price changes, etc) selected via volatility, oil price changes, etc) selected via ANOVA and neural network analysis.ANOVA and neural network analysis.


Trading the mispricing on a weekly basis with Trading the mispricing on a weekly basis with simple long/short strategies yields annualised simple long/short strategies yields annualised returns of over 20% net of transaction costs. returns of over 20% net of transaction costs.

Robust neural network models are used to obtain Robust neural network models are used to obtain smooth equity curves. The methodology is being smooth equity curves. The methodology is being extended to other markets and asset classes.extended to other markets and asset classes.

With: CitiBank., IN USE SINE 1996 - FOR PROPRIETARY TRADING AT EUROPEAN EQUITY DERIVATIVES. LATER MODELS ARE ALSO IN USE AT DEUCHE-MORGAN GRENFELL SINCE JAN. 1997.

Nonlinear Cointegration in Nonlinear Cointegration in European Equity Futures..European Equity Futures....

Forecasting Intra-day Forecasting Intra-day Volatility for Option PricingVolatility for Option Pricing

Multivariate neural models produce Multivariate neural models produce

estimates of implied volatility for estimates of implied volatility for

option pricing for futures contracts.option pricing for futures contracts. High frequency tick-data is used. High frequency tick-data is used. The neural networks give significant The neural networks give significant

performance improvements in termsperformance improvements in terms

of forecasting accuracy over time-series models & regression. of forecasting accuracy over time-series models & regression. Sensitivity analysis is used to verify the plausibility of the neural models, and to provide closed form representations of the pricing formulae.Sensitivity analysis is used to verify the plausibility of the neural models, and to provide closed form representations of the pricing formulae.


Modelling and Trading UK Modelling and Trading UK Gilts vs. Equities.Gilts vs. Equities.

This project models differential returns between UK equities and Gilts as a bivariate time series problem.

The approach is purely technical. The input variables being technical indicators on the differential (normalized) levels.

The resultant network architecture uses a trend following indicator and an oscillator as inputs and attempts to switch between the two, thus exploring changes in dynamics.


Modelling and Trading UK Gilts vs. Equities...Modelling and Trading UK Gilts vs. Equities...

The "hybrid" neural network system outperforms both indicators showing a profit of approximately 60 percentage points (net of transaction costs) over 3 years in out-of-sample data against significantly lower figures for the trend following systems with a higher Sharpe ratio and a smaller drawdown.


Tactical Intra-day Currency Tactical Intra-day Currency TradingTrading

Neural networks are used to generate Neural networks are used to generate buy/sell signals by switching between buy/sell signals by switching between trend following and reversal-based trend following and reversal-based indicators which model the two indicators which model the two primary market states. primary market states.

Tick-data is first transformed into Tick-data is first transformed into "variable time" (e.g. periods of 100 "variable time" (e.g. periods of 100 ticks) before additional indicators are ticks) before additional indicators are computed both within and between computed both within and between each period.each period.


““Money is better than poverty, if only for financial reasons.” Money is better than poverty, if only for financial reasons.” Woody AllenWoody Allen

Tactical Intra-day Currency Trading...Tactical Intra-day Currency Trading... For an institution such as a bank with relatively For an institution such as a bank with relatively

low transaction costs the results show significant low transaction costs the results show significant profitability. profitability.

Software and algorithms have been installed Software and algorithms have been installed including including software for checking high-frequency data software for checking high-frequency data

integrity, integrity, transformations to variable time, transformations to variable time, construction of technical indicators, construction of technical indicators, bayesian neural networks, andbayesian neural networks, and network/trading rule evaluationnetwork/trading rule evaluation

(Customers: Proprietary)


A portfolio of non-linear and A portfolio of non-linear and linear models are used to linear models are used to predict 6 and 12 hour deviations predict 6 and 12 hour deviations from a 12 hour running mean.from a 12 hour running mean.

With the use of 12 hour changes With the use of 12 hour changes

from the current (12 hour) trend from the current (12 hour) trend positive returns are obtained positive returns are obtained with linear models and simple with linear models and simple trading strategies.trading strategies.

Currency Portfolio Currency Portfolio ModelsModels

A family of trend-following and A family of trend-following and mean-reverting indicators are mean-reverting indicators are modelled as "states" in a Hidden modelled as "states" in a Hidden Markov process which switches Markov process which switches between the two on the basis of between the two on the basis of performance deterioration performance deterioration differentials.differentials.

A portfolio approach is taken with A portfolio approach is taken with up to 20 technicals which is "tilted" up to 20 technicals which is "tilted" towards good performing models. towards good performing models.

The results produce much smoother The results produce much smoother equity curves than either family equity curves than either family alone.alone.

(Customers: Proprietary)Dr. Apostolos-Paul Refenes, London Business SchoolDr. Apostolos-Paul Refenes, London Business School

Currency Portfolio Models...Currency Portfolio Models...

Term-structure Models Term-structure Models of of

Eurodollar FuturesEurodollar FuturesNeural networks are used to model Neural networks are used to model

the "volatility factor" in the term-the "volatility factor" in the term-structure of Eurodollar futures. The structure of Eurodollar futures. The

"volatility factor" is the third "volatility factor" is the third principle component. It represents principle component. It represents

a flexing of the yield curve on a a flexing of the yield curve on a portfolio of short, medium and long portfolio of short, medium and long

maturity contracts.maturity contracts.


Term-structure Models of Term-structure Models of Eurodollar Futures...Eurodollar Futures...

This component is shown to be This component is shown to be mean-reverting and it is linked mean-reverting and it is linked to volatility among other to volatility among other factors. factors.

The neural network model The neural network model estimates variations in this estimates variations in this component which are then component which are then used as signals to reset the used as signals to reset the portfolioportfolio..

IN USE SINCE 1996. NOW BEEING REDEVELOPED FOR DRESDNER BANK AND BANQUE NATIONALE DE PARIS.


Portfolio Replication for Risk ManagementPortfolio Replication for Risk Management

“We took risks. We knew we took them. Things have come out against us. We have no cause for complaint.” Robert F. Scott - found in his diary

after the party froze in Antarctica

With modern derivative portfolios, Monte Carlo simulation and bootstrapping is often required to measure risk exposure.

For large portfolios of over 1,000 derivative and synthetic instruments the computational requirements for the simulations are often unrealistic.

Dr. Apostolos-Paul Refenes, Senior Research Fellow, London Business SchoolDr. Apostolos-Paul Refenes, Senior Research Fellow, London Business School

In this project neural networks are used to replicate large portfolios with much smaller ones but which exhibit the same sensitivities and risk profiles. The replication procedure is not always analytically tractable but neural networks provide an efficient way to reduce computation to manageable levels.

THIS IS A PROJECT STILL UNDER DEVELOPMENT WITH

IN USE BY HUGHES FINANCIAL ANALYTICS.

Portfolio Replication for Risk Management...Portfolio Replication for Risk Management...


Modelling Currency Modelling Currency Flows Through Flows Through Network BranchesNetwork Branches Intra- and inter-day currency flows from branches into a Intra- and inter-day currency flows from branches into a

bank's group treasury exhibit significant patterns of bank's group treasury exhibit significant patterns of regularity, and seasonality.regularity, and seasonality.

Typically for small amounts, these positions and crosses are Typically for small amounts, these positions and crosses are often netted locally (at some arbitrary limit) before being often netted locally (at some arbitrary limit) before being passed onto market makers alongside larger positions for passed onto market makers alongside larger positions for global risk management.global risk management.

Neural network models are used to analyse Neural network models are used to analyse

currency flows through branches and to currency flows through branches and to set adaptive netting limits.set adaptive netting limits.

Barclays PLCBarclays PLCDr. Apostolos-Paul Refenes, London Business SchoolDr. Apostolos-Paul Refenes, London Business School

APT Models for APT Models for Equity Equity

InvestmentInvestment The Arbitrage Pricing Theory is The Arbitrage Pricing Theory is

widely used in portfolio widely used in portfolio management .management .

The main proposition of APT The main proposition of APT

models has been that stock models has been that stock returns can be explained in returns can be explained in terms of a set of factors.terms of a set of factors.

Dr. Apostolos-Paul Refenes, London Business School

APT Models for Equity APT Models for Equity Investment...Investment...

Traditionally it has been assumed Traditionally it has been assumed that the return is a linear that the return is a linear combination of each stock's combination of each stock's exposure to these factors.exposure to these factors.

This project weakens the This project weakens the

assumptions on linearity and uses assumptions on linearity and uses neural networks instead of multiple neural networks instead of multiple linear regression to model stock linear regression to model stock returns achieving better model returns achieving better model fitness (by a factor of 2) in- and fitness (by a factor of 2) in- and out-of-sample over a period of ten out-of-sample over a period of ten years on a universe of 150 UK years on a universe of 150 UK stocks.stocks.

With: County Natwest. IN USE SINCE 1993.With: County Natwest. IN USE SINCE 1993.

Dr. Apostolos-Paul Refenes, London Business School

Neural Networks are used to Neural Networks are used to

perform quantitative asset perform quantitative asset

allocation between global allocation between global

bond markets and US cash. bond markets and US cash. Assets are allocated monthly Assets are allocated monthly

in seven markets (USA, Japan, UK, Germany, Canada, in seven markets (USA, Japan, UK, Germany, Canada, France and Australia) chosen on the basis of France and Australia) chosen on the basis of outperformance. outperformance.

The system consists of several neural models (one for The system consists of several neural models (one for each market) which are integrated into a global each market) which are integrated into a global portfolio management system which allocates the fund portfolio management system which allocates the fund proportionately to predicted returns but subject to proportionately to predicted returns but subject to certain constraints on the allocation to minimise risk. certain constraints on the allocation to minimise risk.

Since November 1992, the portfolio has consistently Since November 1992, the portfolio has consistently outperformed industry benchmarks by a factor of 2.outperformed industry benchmarks by a factor of 2.

With: Econostat Ltd. IN USE SINCE 1992- MANAGING 100 With: Econostat Ltd. IN USE SINCE 1992- MANAGING 100 MILLION.MILLION.

http://www.clients.globalweb.co.uk/nctt/newsletter/09/part008/05.html http://www.clients.globalweb.co.uk/nctt/newsletter/09/part008/05.html

Global Bonds Global Bonds PortfolioPortfolio


Space Robot CalibrationSpace Robot Calibration

Self-adapts robot whose performance is degraded by wear & damage.

Uses SOM’s to learn inverse kinematics. Installed on robot at the space station

mock-up at Daimler- Benz Aerospace.

V.C. de Angulo and C. Torras, “Self-Calibration of a V.C. de Angulo and C. Torras, “Self-Calibration of a

Space Robot”, IEEE TNN, July 1997.Space Robot”, IEEE TNN, July 1997.

Process Control in Process Control in Petroleum Refining Petroleum Refining

A control system using an adaptiveA control system using an adaptive neural network for target and path neural network for target and path optimization for a multivariable, optimization for a multivariable, nonlinear process. nonlinear process.

The developed neural network is an The developed neural network is an adaptive, multivariable system for adaptive, multivariable system for controlling, stabilizing, and optimizing controlling, stabilizing, and optimizing complex, industrial processes.complex, industrial processes.

NeuralWare /TexacoNeuralWare /Texacohttp://www.neuralware.com/http://www.neuralware.com/

Process Control in Petroleum Refining...Process Control in Petroleum Refining...

Target processes for the neural Target processes for the neural network include petroleum network include petroleum

refining, chemical, steel, utility, refining, chemical, steel, utility, pharmaceutical, food processing, pharmaceutical, food processing,

and pulp and paper operations. The and pulp and paper operations. The developers claim the NN “can developers claim the NN “can

minimize process variations, reduce minimize process variations, reduce costs by cutting utility and raw costs by cutting utility and raw

material usage, increase safety by material usage, increase safety by maintaining stable and efficient maintaining stable and efficient process operation, and decrease process operation, and decrease

environmental impact by reducing environmental impact by reducing emissions.”emissions.”

NeuralWare /TexacoNeuralWare /Texacohttp://www.neuralware.com/http://www.neuralware.com/

Neural Networks are Neural Networks are Used to Grade Pig Used to Grade Pig

Carcases in Danish Carcases in Danish Slaughter HousesSlaughter HousesDr. Hans Henrik ThodbergDr. Hans Henrik Thodberg

The Danish Meat Research InstituteThe Danish Meat Research InstituteMaglegaardsvej 2Maglegaardsvej 2DK 4000 RoskildeDK 4000 Roskilde

DanmarkDanmark

Credit Card Fraud Credit Card Fraud DetectionDetection

The probability of fraud is The probability of fraud is calculated with a neural network with calculated with a neural network with

each card transaction. When the each card transaction. When the probability of fraud reaches a critical probability of fraud reaches a critical threshold, the case is sent to one of threshold, the case is sent to one of

the retailer's fraud analysts for action.the retailer's fraud analysts for action.

Sears, Roebuck and Co. has annual revenue of Sears, Roebuck and Co. has annual revenue of more than $38 billion. Sears Credit is one of the more than $38 billion. Sears Credit is one of the

nation's largest credit card businesses, with nation's largest credit card businesses, with receivables of nearly $24 billion and more than 30 receivables of nearly $24 billion and more than 30

million active accounts.million active accounts.

http://www.hncs.com/http://www.hncs.com/

“Money is a poor man’s credit card.” Marshall McLuhan

Cancer DiagnosisCancer Diagnosis PAPNET Testing System is designed for the screening

of Papanicolau (Pap) smear slides for the presence of cervical cancer and precancerous abnormalities.

During its clinical trials, average accuracy rates of 97% above

average 70% accuracy

rate of current system.

Neuromedical Systems Inc.http://www.ebml.com/ebpapn.htm

Cancer Diagnosis...Cancer Diagnosis...

Competition: Fuzzy Decision Tree vs. Neural Network

AutoPap vs NetPap Who’s winning?

Neuromedical Systems Neopath Neopath

Examines only the historyExamines only the historyof the card user.of the card user.

Uses a nonlinear Uses a nonlinear generalization of Fisher’sgeneralization of Fisher’s

discriminant analysis.discriminant analysis.

Used since 1996 by SEMP.Used since 1996 by SEMP.Receives 60% of the VISAReceives 60% of the VISA

traffic in Spain.traffic in Spain.

Dorronsoro, Ginel, Sanchez &Dorronsoro, Ginel, Sanchez &Cruz, “Neural Fraud DetectionCruz, “Neural Fraud Detection

on Credit Card Operations”on Credit Card Operations”IEEE TNN, July 1997.IEEE TNN, July 1997.

training pdf’straining pdf’s(Jan -Sept 1994)(Jan -Sept 1994)

test pdf’stest pdf’s(Oct 94 to April 1995)(Oct 94 to April 1995)

Credit Card FraudCredit Card Fraud

Merchant Merchant Fraud !Fraud !

"Absence of evidence is not evidence of absence."Carl Sagan

Neural networks and a rule engine automate a Neural networks and a rule engine automate a merchant portfolio review process to minimize the merchant portfolio review process to minimize the risk of unpaid chargebacks for acquiring banks.risk of unpaid chargebacks for acquiring banks.

Merchant Fraud ...Merchant Fraud ... Performing audits and investigating Performing audits and investigating

chargebacks by traditional, paper-based chargebacks by traditional, paper-based means is time-consuming and labor-means is time-consuming and labor-intensive. Much of the merchant intensive. Much of the merchant analysts' time is spent sorting through analysts' time is spent sorting through information to determine which information to determine which accounts to review. Such process accounts to review. Such process inefficiencies mean time consuming inefficiencies mean time consuming work for the analysts to ensure that work for the analysts to ensure that problems are not discovered too late to problems are not discovered too late to take remedial action. take remedial action.

Barclays Merchant Services is the UK's Barclays Merchant Services is the UK's largest acquirer and processor of plastic largest acquirer and processor of plastic card transactions.card transactions.

www.hncs.comwww.hncs.com

Neural Models for Neural Models for House Prices Arrears House Prices Arrears and Repossessionsand Repossessions

The design and pricing of products in the The design and pricing of products in the housing loans market is critically housing loans market is critically dependent on the risk profiles of dependent on the risk profiles of segments within the population. segments within the population.

a better understanding of the sources of a better understanding of the sources of default risk allow the likely consequences default risk allow the likely consequences of marketing decisions to be predicted of marketing decisions to be predicted more accurately. more accurately.

The current policy of many lenders is to The current policy of many lenders is to decline loan applications which fail to decline loan applications which fail to pass a set of policy rules, these pass a set of policy rules, these applicants are referred to as "out-of-applicants are referred to as "out-of-policy" and are deemed to present an policy" and are deemed to present an undesirable level of risk.undesirable level of risk.

Neural Models for House Neural Models for House Prices Arrears and Prices Arrears and

Repossessions...Repossessions... Policy rules are often the result of linear

statistical analysis of the mortgage book (cf scorecards) which fails to capture non monotonic relationships between some factors (e.g. age and loan-size) and arrears rates.

A neural network is used to model the relationship between loan application details and subsequent arrears rates; using over 20,000 observations over three years of data, the neural network is consistently twice as accurate as a "scorecard" system.

This allows 100% more new business to be accepted at the same level of risk.

(Customers: Proprietary) Dr. Apostolos-Paul Refenes, London Business Dr. Apostolos-Paul Refenes, London Business SchoolSchool

Insurance Fraud DetectionInsurance Fraud Detection Most insurance fraud goes undetected. Health care insurers discovered one percent of their Most insurance fraud goes undetected. Health care insurers discovered one percent of their

fraud losses; life/disability detected about 10 percent, & property / casualty (including workers' fraud losses; life/disability detected about 10 percent, & property / casualty (including workers' compensation), about 20%.compensation), about 20%.

Further, insurance fraud is estimated to cost cost the industry $120 billion in 1995, a 30 percent Further, insurance fraud is estimated to cost cost the industry $120 billion in 1995, a 30 percent increase from the $90 billion in losses estimated in 1990. The health care sector is cited as the increase from the $90 billion in losses estimated in 1990. The health care sector is cited as the most victimized with $95 billion in fraud losses, followed by property/casualty at $20 billion.most victimized with $95 billion in fraud losses, followed by property/casualty at $20 billion.

““Insurance. An ingenious modern game of chance in which the player is Insurance. An ingenious modern game of chance in which the player is

permitted to enjoy the comfortable conviction that he is beating the man permitted to enjoy the comfortable conviction that he is beating the man

who keeps the table,who keeps the table,” ” Ambrose BierceAmbrose Bierce

...Insurance Fraud Detection...Insurance Fraud Detection Compare workers' compensation insurance claims Compare workers' compensation insurance claims

against historical patterns of fraudulent claims. against historical patterns of fraudulent claims. Enhances investigators' ability to identify suspicious Enhances investigators' ability to identify suspicious claims early and pin-point claims with a high probability claims early and pin-point claims with a high probability of fraud.of fraud.

Risk Data CorporationRisk Data Corporation for Workers' Compensation Fund of Utah for Workers' Compensation Fund of Utah: <www.riskdata.com>www.riskdata.com>

NN Forecast of nonagricultural employment

State Economic ForecastingState Economic ForecastingForecast tax revenues.Forecast tax revenues.

Used by The Office of the Legislative Fiscal Analyst (Utah)Used by The Office of the Legislative Fiscal Analyst (Utah)

J. V. Hansen, “Neural Networks and Traditional Time Series J. V. Hansen, “Neural Networks and Traditional Time Series

Methods: A Synergistic Combination in State Economic Methods: A Synergistic Combination in State Economic

Forecasts”, IEEE TNN, July 1997.Forecasts”, IEEE TNN, July 1997.

Arguing with success ...Arguing with success ... "The abdomen, the chest, and the brain will forever be shut from the intrusion of the "The abdomen, the chest, and the brain will forever be shut from the intrusion of the

wise and humane surgeon", Sir John Eric Ericksen, British surgeon, appointed wise and humane surgeon", Sir John Eric Ericksen, British surgeon, appointed Surgeon-Extraordinary to Queen Victoria 1873.Surgeon-Extraordinary to Queen Victoria 1873.

"Who the hell wants to hear actors talk?" H.M. Warner, Warner Brothers, 1927."Who the hell wants to hear actors talk?" H.M. Warner, Warner Brothers, 1927. "We don't like their sound, and guitar music is on the way out." Decca Recording Co. "We don't like their sound, and guitar music is on the way out." Decca Recording Co.

rejecting the Beatles, 1962.rejecting the Beatles, 1962. "Everything that can be invented has been invented." Charles H. Duell, "Everything that can be invented has been invented." Charles H. Duell,

Commissioner, U.S. Office of Patents, 1899.Commissioner, U.S. Office of Patents, 1899. "Stocks have reached what looks like a permanently high plateau." Irving Fisher, "Stocks have reached what looks like a permanently high plateau." Irving Fisher,

Professor of Economics, Yale University, 1929.Professor of Economics, Yale University, 1929. ““The image … of ‘new age’ technologies of fuzzy logic, neural The image … of ‘new age’ technologies of fuzzy logic, neural

networks,... approximate reasoning, and self-organization in the networks,... approximate reasoning, and self-organization in the face of dismal failure of traditional methods … is pure face of dismal failure of traditional methods … is pure unsupported claptrap which is pretentious and idolatrous in the unsupported claptrap which is pretentious and idolatrous in the extreme, and has no place in scientific literature.” Professor extreme, and has no place in scientific literature.” Professor Bob Bitmead,Bob Bitmead, June 1993.June 1993.

““The best way The best way to predict the to predict the

future is to future is to invent it”invent it”

The The EndEnd ““Never express yourself more clearly than you think.” Neils Bohr (1885-1962)Never express yourself more clearly than you think.” Neils Bohr (1885-1962)

http://cialab.ee.washington.edu/Marks.htmlhttp://cialab.ee.washington.edu/Marks.html

[email protected]@ieee.org

Robert J. Marks IIRobert J. Marks II

Many of the logos used in this presentation are registered Many of the logos used in this presentation are registered trademarks of the companies they characterizetrademarks of the companies they characterize

17 is the correct answer!17 is the correct answer!

Appendix... Appendix...

INSPEC & CASSISINSPEC & CASSIS “Information Service for

Physics & Engineering Communities”4000+ journals 2 1/4 Million + entries since 1989Compiled by IEE & IEEE

“Classification for Search Support Information Systems”

Search Words Neural Networks Neural Net(s) Neural Network(s) Neurocomputing

Artificial Intelligence Expert System(s) Machine Intelligence Intelligent Systems

Computational Intelligence Neural Networks search words Fuzzy Genetic Algorithm(s) Evolutionary Programming Artificial Life

INSPEC DATA

19891991

19931995EC

FuzzyExpert

NN'sCI

AI0

2000

4000

6000

8000

10000

12000

14000

EC Fuzzy Expert NN's CI AI

1986 1988 1990 1992 1994

FuzzyAI

NN's

0

50

100

150

1986 1988 1990 1992 1994

FuzzyAI

NN's

CASSIS Data: Patents

Documents

Neural Networks: Reduction to Practice R obert J. M arks II Baylor University CIA L ab School of Engineering