AlgoTraderDocumentation Overall Documentation of the Framework algotrader as well as instructions on how to build a strategy with it Featured Updated Sep 3, 2012 by andyflury Introduction Architecture Operating Systems Services / Processes Platform requirements Eclipse Projects UML Mode and Code Generation UML Model Code Generation Java Development Java Packages Java Libraries Coding Standards Esper Engine Trading Framework Modules Strategy Modules Database & Transaction Handling Entity Relationship Model DB Tables Transaction Handling Installation Subversion Checkout Generate the code Compile Build Errors Install Start a Demo Strategy Strategy Development MovServiceImpl .java module-mov-main.epl conf-mov.properties esper-mov.cfg.xml DB table strategy DB table watch_list_item Operation Modes Back Testing / Simulation Mode Live Trading Mode Broker Interfaces Order / Execution Market Data Miscellaneous Topics Tick Data Format Configuration Strategy vs. Strategy Family ServiceLocators , BeanFactory & ApplicationContext Security Remoting AspectJ LoggingIntroductionThe Trading Framework AlgoTrader facilitates a minimum effort development of trading strategies based on Esper component for complex event processing (CEP). The system is developed by a Model Driven Architecture. All artifacts of the system are modeled in UML and generated into code by the Framework AndroMDA. For Database Persistence the Framework Hibernate is used. Currently the system uses MySql as a Database. Through the use of Hibernate, other database providers can used by changing configuration. The system is based on the Spring Framework, which enables a clean Service Oriented Architecture. The system does not provide a GUI Frontend. Instead it uses Java Management Extension (JMX) for management at runtime. Currently, the system provides the following main features: Automate Trading Strategies based on Complex Trading Rules Simulate and run several strategies in parallel Backtest strategies based on historical Data Portfolio tracking & Performance Measurement Provide different Numerical and Statistical Libraries To work with AlgoTrader it is recommended, that you have a basic level of knowledge on following topics: Java Standard Edition Eclipse IDE XML Spring & Hibernate MySql ArchitectureThe System is composed of the following two layers: The Trading Framework layer is the base for all strategies running on top of it. It is responsible for persistence to the database. It holds the connection to the Broker Interfaces to retrieve live market data as well as to place orders. The Trading Framework is also responsible for Risk Management, Portfolio Management & Performance Measurment. For many of its tasks the Trading Framework uses a contained Esper Engine Instance. On top of the Trading Framework any number of Strategies can be installed. In Live Trading mode each strategy runs in its own process. Each Strategy contains its own Esper Engine Instance. A Strategy can use any type and number of Technical Indicators and custom trading rules to evaluate trend and to retrieve market signals. Orders are delegated to the underlying Trading Framework for execution.

Operating SystemsThe entire system is developped in Java. Therefor the system is portable to any environment. The System has been developed on a Windows 7 machine and has been in production on FreeBSD as well as Linux Servers. Services / ProcessesThe following Services / Process are used by the system: Service / Process Description

Strategies In Live Trading Mode each strategy runs in its on JVM (Java process). In simulation mode, the strategies run within the same process as the trading framework

Trading Framework This is the main process that should be kept running all the times in order to record market data without gaps

MySql Database process

Because the framework and the strategies are running within separate processes, individual strategies can be stopped / altered / restarted independent of each other and the framework. Platform requirementsFor development of new Strategies the following Platform requirements exist: Software URL Tested Version

Java 1.6 http://www.oracle.com/technetwork/java/javase/downloads/index.html 1.6.0_22

Eclipse IDE http://www.eclipse.org 3.7.0

MagicDraw UML http://nomagic.be/files95 (use MD_UML_95_sp1_CE... for your plattform) 9.5 SP1

Subversive SVN Client http://community.polarion.com/projects/subversive/download/eclipse/2.0/indigo-site/

M2E http://download.eclipse.org/releases/indigo

Maven SCM Handler for Subversive http://repository.tesla.io:8081/nexus/content/sites/m2e.extras/m2eclipse-subversive/0.13.0/N/0.13.0.201207010112/

MySql Database http://www.mysql.com 5.1

A database management tool (i.e. TOAD for MySql) http://www.quest.com/toad-for-mysql

Hibernate Tools (optional) http://download.jboss.org/jbosstools/updates/development/indigo

Veloeclipse (optional) http://veloeclipse.googlecode.com

Eclipse CheckStyle (optional) http://eclipse-cs.sourceforge.net/

Eclipse ProjectsThe Framework AlgoTrader consists of the following Eclipse Projects: algotraderlight (the main project) algotraderlight-code (all java code) algotraderlight-mda (UML Model and code generator) algotraderlight-crud (DB management application) algotrader-strat (parent project for all strategies) algotraderlight-xxx (sample strategies) algotraderlight has the following structure: Directory Description

code/src/main manually created source files

code/src/test JUnit test

code/target generated source files

code/results files which are created (i.e. copy of retrieved HTML code) or imported (i.e. Tick Data) into the system

code/bin Launch configuration and shell start scripts

code/lib jar files not available through maven

code/log log files

code/sql SQL scripts

mda/conf Contains configuration files for the generator

mda/src Contains UML Model and AndroMDA configuration / customization files

The strategy projects algotrader-xxx have the following structure: Directory Description

src manually created source files

lib jar files not available through maven

bin Launch configuration and shell start scripts

log log files

results Contains mainly Market Data Files

UML Mode and Code GenerationThe entire system is developed by means of Model Driven Architecture (MDA) using the framework AndroMDA (www.andromda.org). To get familiar with the Code Generator AndroMDA the following links and tutorials give a good overview: A Bird's Eye view of AndroMDA Getting started with AndroMDA or Prepare the Development Environment Getting Started Java - long winded 'TimeTrackingService' example Spring Cartridge example - short and focused on buisness tier of 3-tier application architecture AndroMDA's forum where you can get FREE support UML ModelThe UML Model in use by the system is shown here: UML Model Code GenerationMuch of the system is modelled in UML. It is recommended to use MagicDraw for modelling (see Platform_requirements) The actual UML Model is defined in the following xmi file: /algotraderlight-mda/src/main/uml/algotrader.xmi The model is transferred into code my means of AndroMDA which uses Maven artefacts and Velocity templates for code generation. Generated code is placed under the directory /target, whereas implementation classes are generated into the directory /src with only the method signatures. Configuration of the code generator is placed into the /mda directory. It is also this directory that contains template modifications as well as mappings from UML artifacts to Java / Database objects. Notes: When opening the file algotrader.xmi for the first time, the location of the Maven 2 Repository has to be defined. This is usualy in user-home-directory/.m2/repository. In addition the path to the UML-Standard-Profile has to be defined. This file lies in the profiles sub directory of the MagicDraw Installation directory (e.g., ..\MagicDrawUml95CE\profiles\UML_Standard_Profile.xml) Java DevelopmentJava PackagesThe following table gives an overview of the provided java packages: Package Description

com.algoTrader ServiceLocators

com.algoTrader.entity Entities and corresponding Hibernate DataAccessObjects

com.algoTrader.service Contains all Spring Services

com.algoTrader.starter Startup-Classes as well as a generic ServiceInvoker

com.algoTrader.listener Esper Statement Listeners

com.algoTrader.subscriber Esper Statement Subscribers

com.algoTrader.util General Utility methods

com.algoTrader.util.io CSV Adapters for market data as well as market data conversion functions

Java LibrariesThe following table shows an overview of used java libraries: Java Library Description

Spring Frameworkhttp://www.springsource.org spring-aop-3.0.5.RELEASE.jarspring-asm-3.0.5.RELEASE.jarspring-beans-3.0.5.RELEASE.jarspring-context-3.0.5.RELEASE.jarspring-context-support-3.0.5.RELEASE.jarspring-core-3.0.5.RELEASE.jarspring-expression-3.0.5.RELEASE.jarspring-jdbc-3.0.5.RELEASE.jarspring-orm-3.0.5.RELEASE.jarspring-tx-3.0.5.RELEASE.jarspringtemplate-3.2.jar

Hibernatehttp://www.hibernate.org aopalliance-1.0.jarantlr-2.7.6.jarcglib-2.2.jardom4j-1.6.1.jarehcache-2.3.3.jarehcache-core-2.3.3.jarehcache-terracotta-2.3.3.jarhibernate-core-3.5.6.jarhibernate-jmx-3.5.6.jarjta-1.1.jarslf4j-api-1.6.1.jarslf4j-log4j12-1.6.1.jar

Esperhttp://esper.codehaus.org esper-4.2.0.jaresper-csv-4.2.0.jaresper-socket-4.2.0.jarcglib-nodep-2.2.jarantlr-runtime-3.2.jar

XMLhttp://xml.apache.org serializer-2.7.1.jarxalan-2.7.1.jarxml-apis-1.3.04.jar

Mail / SMS activation-1.1.jarmail-1.4.3.jar

AspectJhttp://www.eclipse.org/aspectj aspectjrt-1.5.4.jaraspectjweaver-1.5.4.jar

Apache Commons http://commons.apache.org collections-generic-4.01.jarcommons-beanutils-1.8.3.jarcommons-beanutils-core-1.7.0.jarcommons-cli-1.0.jarcommons-collections-3.2.1.jarcommons-configuration-1.6.jarcommons-digester-1.8.jarcommons-lang-2.6.jarcommons-logging-1.1.1.jarcommons-math-2.2.jargoogle-collections-0.9.jar

Miscellaneous jtidy-r8-21122004.jar (HTML to XHTML)mysql-connector-java-5.1.15-bin.jar (MySql Client)ibclient-9.6.3.jar (InteractiveBroker API client) log4j-1.2.16.jar (Log4J)agent.jar (JMX Agent)ant-1.7.0.jar (Apache Ant)ant-launcher-1.7.0.jar (Apache Ant)checkstyle-5.0.jar (Checkstyle)checkstyle-optional-4.4.jar (Checkstyle)javassist-3.12.1.GA.jar (Javassist)junit-4.8.2.jar (JUnit)ta-lib-0.4.0.jar (TA-Lib)SuperCSV-1.52.jar (SuperCVS)

Coding StandardsThe project uses checkstyle to verify java coding standards. Checkstyle is enabled by default in the Maven built. In addition the Plugin Eclipse-CS can be used to enable Checkstyle inside Eclipse. To use Eclipse-CS a new Global Check Configuration has to be added in Eclipse / Window / Preferences / Checkstyle pointing to: /algotraderlight-code/checkstyle.xml

Also, in the java project properties, Checkstyle has to be enabled and pointing to the AlgoTrader (Global) configuration.

In addition to using Checkstyle, Eclipse has two handy features to automatically format Java Code Clean Up: This feature is responsible to do a general clean up when saving a modified File (i.e. remove unused imports) Formatter: This feature will automatically format source code according to definable standards (can be invoked through right click / source / format) A default set of policies targeted to automatically conform with defined checkstyle rules is available. These are available in the source code of the project. These configurations have to be imported them into Eclipse: Eclipse / Preferences / Java / Code Style / Clean Up: AlgoTraderLight-code/AlgoTrader-CleanUp.xml Eclipse / Preferences / Java / Code Style / Formatter: AlgoTraderLight-code/AlgoTrader-Formatter.xml

Esper EngineThe system is built on CEP (Complex Event Processing) using the open source framework Esper. Individual Esper service engines are used inside the trading framework as well as within the strategies. Each of these engines can contain several modules. Modules specified with in the column MODULES of the Table strategy, are loaded automatically on startup. Each strategy also contains its own Esper configuration file named esper-xxx.cfg.xml. This configuration file defines settings like: Event Types Auto Import Classes & Packages Custom Aggregation Functions Variables General Engine Settings Esper Listeners and Subscribers are configured inside the modules using the following syntax: @Name('REBALANCE_PORTFOLIO')@Tag(name='listeners', value='com.algoTrader.listener.RebalancePortfolioListener')select * from Listeners or Subscribers which correspond to a particular Service are commonly implemented as an inner class of the Service. Trading Framework ModulesThe Trading Framework contains only one module called module-base.epl with Esper Statements to fulfill the following tasks: Trigger persistence of MarketData Evaluation and Logging of the Portfolio Values at certain intervals Position handling (i.e. close position when exit level is reached, etc.) Evaluation of Performance Parameters (i.e. Sharpe Ratio, Calmar Ratio, Volatility, Draw-Down, etc.) Margining Process Cash Transactions (Subscriptions / Redemptions) Rebalancing between the strategies to keep strategy allocation at a constant level Strategy ModulesStrategies are completely free in the definition of their Esper Statements. Examples of Statements used by strategies are: Creation of technical indicators (i.e. Moving Average, Stochastic, MACD, etc.) Creation of trade signals Trend evaluation Open / Close / Increase / Reduce Positions Set Exit Values Roll Positions (i.e. for Options and Futures) Pattern recognition Orderbook analysis Database & Transaction HandlingThe database MySql is used by the system, even though by using Hibernate, any supported database can be used. The directory algotraderlight-code/sql contains a script algotrader.sql to create the database. Entity Relationship Model

DB TablesThe following table gives a description of the most important tables: Table Description

strategy Each record represents a strategy within the system

security, stock, equity_index, stock_option, future, intrest_rate & forex) Security is the base table for all securities in the system. The related tables carry specific information of individual asset clases

tick, bar, bid, ask & trade All relevant market data is stored in these tables

transaction Each Fill is stored as a transaction in the database. In addition the table transaction also carries transactions like intrest, debit, credit & fees

position For each opening transaction a position is created which also carries exit values and maintenance margin

security_family Common information about an entire family of securities (i.e. all general information about options on S&P500 are stored using this class). The table provides fields like market, currency, market opening hour & market closing hour

Transaction HandlingUsing AndroMDA and Spring Transaction Boundaries can be declared inside the UML model using the tagged value @andromda.spring.transaction.type. Transaction Boundaries will be generated into the file applicationContext.xml and handled by the org.springframework.transaction.interceptor.TransactionInterceptor. InstallationSubversion CheckoutIt is recommended to use the Plugin M2Eclipse for the initial checkout (Import / Maven / Check out Maven Projects from SCM). Note: AndroMDA does not allow any special characters (i.e. spaces or exclamation marks) in the project path. Select https://algo-trader.googlecode.com/svn/trunk/AlgoTraderLight. This will create the following 3 projects in Eclipse: algotraderlight algotraderlight-code (name of the subfolder under algotraderlight: code) algotraderlight-crud (name of the subfolder under algotraderlight: crud) algotraderlight-mda (name of the subfolder under algotraderlight: mda) Run the import again and select https://algo-trader.googlecode.com/svn/trunk/AlgoTraderLightStrat. This will create the following additional projects in Eclipse: algotrader-mov algotrader-max algotrader-periodic Generate the codeRight click on the project algotraderlight-mda / Run As / Maven install. This will generate all necessary code into algotrader-code target directory. CompileNow refresh all projects. Eclipse will compile all java code automatically. Before this is done, please check the Java Build Path of algotraderlight-code project. There are two sources configured. The source folder target/src/main/java might have an exclusion defined (Star-Star). If there is such exclusion, please remove it (the exclusion not the source folder!) Build ErrorsBuilding the MOV Example for the first time may result in errors due the 'Optional' condition put on ibclient-9.6.3.jar, quickfixj-core-1.5.2.jar, quickfixj-msg-fix42-1.5.2.jar and ta-lib-0.4.0.jar in algotraderlight-code project. This can be easily corrected by opening the algotraderlight-code pom.xml file, navigate to the 'Dependencies' tab, right click on each of the jar files mentioned and uncheck the Optional box. Install MySqlInstall latest MySql Community Server run the create-script: sql/algotrader.sql configure the properties dataSource.user & dataSource.password inside ../algotraderlight/code/src/main/java/conf-base.properties: The default settings are dataSource.user=root and dataSource.password=password. Your DBA will probably frown on using root as the user so here is a simple set of sql commands to create a new user account, password, and privileges limited to the algotrader schema (this can be added to the bottom of the algotrader.sql file if you want to run everything from one file): CREATE USER 'algouser'@'localhost' IDENTIFIED BY 'algopwd';

GRANT ALL PRIVILEGES ON AlgoTrader.* TO 'algouser'@'localhost'; Start a Demo StrategyBefore running a strategy check the database table strategy. Only the records "BASE" and the record corresponding to strategy should be set to AUTO_ACTIVATE Mov Example StrategyThis sample strategy is based on a moving average indicator. It's purpose is to demonstrate AlgoTrader in Back Testing / Simulation mode. To start it, use the provided eclipse lunch configuration: /algotraderlight-mov/bin/SimulationStarter-simulateWCP-Mov.launch Periodic Example StrategyThe purpose of this strategy is to demonstrate AlgoTrader in Live Trading mode. It currently contains the following examples: periodic buy/sell trades every 10 seconds order cancellation if order is not executed within 2 seconds order modification every 2 seconds until order is fully executed example Execution Algo SteppingLimitOrder To enable / disable any of these examples uncomment the relevant section in /algotraderlight-periodic/src/main/java/module-periodic-main.epl To run any of these examples InteractiveBrokers TWS has to be running with the following configuration under API/Settings: Enable ActiveX and Socket Clients Socket Port: 4001 In live trading mode the trading framework and the strategy run in separate JVM's and have to be started separately. To start the trading framwork, use the provided eclipse lunch configuration: /algotraderlight-code/bin/MarketDataStarter.launch To start the periodic example strategy, use the provided eclipse lunch configuration : /algotraderlight-periodic/bin/PeriodicStarter.launch Strategy DevelopmentThe following paragraph will give a short example based on a simple moving average strategy (with the Short Name MOV). After backtesting, newly developed strategies can be tested in a live environment using Paper Trading Accounts. At the end of a thorough test procedure, the new strategy can be put into production. The following diagram shows the general procedure for developing new strategies:

A strategy is based on the following minimum artifacts: Artifact Description

/src Source-File Directory containing necessary Java-Classes for the Strategy

/lib Jar Files not available through maven

/src/main/java/module-mov.epl Esper Module containing the necessary Esper Statements

/src/main/java/conf-mov.properties Contains parameters used by the strategy (i.e. Moving average durations etc.)

/src/main/java/esper-mov.cfg.xml Contains event-types, imports, variables and general Esper settings

/src/main/java/applicationContext-client.xml Application Context File for the strategy, mainly containing autowire instructions

/log log files

/results/bardata/xxx Directory containing Bar Data Files needed for Back Testing

/results/tickdata/xxx Directory containing Tick Data Files needed for Back Testing

DB table strategy A record for the strategy

MovServiceImpl.javaThis is the main Java-class containing Subscriber inner classes and Business Logic First we need references to services provided by the base framework: private PositionService positionService;private LookupService lookupService;private TransactionService transactionService;

public MovServiceImpl(PositionService positionService, LookupService lookupService, TransactionService transactionService) {

this.positionService = positionService; this.lookupService = lookupService; this.transactionService = transactionService;}These values will be auto injected on startup by the Springframework (based on applicationContext-mov.xml) Next we need an Esper Subscriber to react onto market signals. This can be coded as a separate class or as an inner class within MovServiceImpl.java public static class OpenPositionSubscriber { public void update(String strategyName, int securityId, BigDecimal currentValue) {

MovServiceImpl movService = ((MovServiceImpl) ServiceLocator.commonInstance().getService("movService"));

movService.openPosition(strategyName, securityId, currentValue); }}The ServiceLocator will give as a reference to the movService (Spring Bean) Finally we need a business method to instruct the base framework to open a position: public void openPosition(String strategyName, int securityId, BigDecimal currentValue) {

Strategy strategy = this.lookupService.getStrategyByNameFetched(strategyName); Security security = this.lookupService.getSecurity(securityId); int qty = (int) (strategy.getAvailableFundsDouble() / currentValue.doubleValue()); if (qty indicator=Indicator)]where indicator.value > 0and prior(1, indicator.value)