View
1.426
Download
0
Embed Size (px)
Citation preview
What I Learned Writing a Trillion $
Trading System
Nick Zeeb & Antonio Molina @NickZeeb @ajmolinacom
23rd October 2015
$$
Challenges
{ FX Market }
Source: http://www.fxkeys.com, April 29th, 2015 by Chris Pottorff
{ $1.9 Trillion / year }{ $7.3 Billion / day }{ $5.0 Million / minute }
Challenges
{ spikes of 250,000 transactions / second }
7
Challenges
{ Innovation }{ Risk of Failure }
{ simpler is better }
Plain Old Java(JavaSE)
{ integrate integrate integrate }
12
{ feedback }
{ embargo }
Nick and Antonio at GeeCon Prague!
{ invest }- Virtualized dedicated system- Acceptance Tests : 664 cores, 4.9TB RAM, 5.4
TB SSD Storage, 3.3TB SAS Storage, 10.1TB NL-SAS Storage.
- Commit build: 80 cores, 128GB RAM, 1.2TB Storage.
- Staging: 240 cores, 2.3TB RAM, 4.8TB SSD Storage, 24TB NL-SAS.
- Native 10gig networking throughout CI.
Classification: Public
{ everyone is a tester }
Go Crazy on Tests
Acceptance Tests Integration tests
Unit tests
Software unit
Unit tests
Software unit
Unit tests
Software unit
{ test like crazy }
Client
{ acceptance tests }
{ acceptance tests }
@Testpublic void shouldNotLoginWithInvalidCredentials(){ tradingUI.login("user: demo", "password: NotCorrect1.",
"failureMessage: BAD_CREDENTIALS");
logsAPI.checkCustomerAuditLogContains("user: demo", "expression: Failed log on: BAD_CREDENTIALS");
}
{ clean logs }
20
{ reliability }
@Testpublic void shouldFailOverToSecondaryMatchingEngine() {
//when coordinator.killService("primary matching engine") tradingGateway.placeOrder("id: myOrder",
"side: buy", "price: 10",
"quantity: 10"); //then tradingGateway.waitForOrder("id: myOrder",
"expectStatus: OK");}
{ reliability }
{ performance }
{ what does NOT work }
O(x^2)
{ be a scientist }
{ performance tests }
Microbenchmarks
(class, algo level)
Macrobenchmarks
(whole system)
shorter time to feedback
longer time to feedback
less realistic more realistic
Mesobenchmarks
(business operation
level)
{ microbenchmarks }
{ microbenchmarks }public BigDecimal myDivision(long quantity){ BigDecimal bd = BigDecimal.valueOf(quantity)
.setScale(SCALE, UNNECESSARY);
return bd.divide(DIVISOR, UNNECESSARY);}
public BigDecimal myDivision(long quantity){ BigDecimal bd = BigDecimal.valueOf(quantity); return bd.divide(DIVISOR);}
{ microbenchmarks }
{ macrobenchmarks }- Representative of production.- We need load generators:
- Statistical models.- Spiky scenarios.- Maintained load (red line).
- Real usage from journals.
{ releases }
Revision X
Revision X+1
{ releases }
Software changesdata migration
{ compatibility }download & import data -> migrate -> run test
@Testpublic void shouldHaveASecurityRoleForEveryRoleIdInUse() {
Collection<Integer> validIds = SecurityRole.getIds();
roleDao.getAllRolIdInUse().stream().forEach(roleId -> {
assertThat("Unexpected role ID on the DB: " + roleId, validIds.contains(roleId), is(true));
} );}
{ compatibility }
{ staging release }
Deploy the current production revision to staging
Load production data (sanitized)
Migrate staging to the new revision that we are going to
deploy
START and CHECK system is working as expected
Host1
Host3 Host2
Host4
Real Exchange(with $$$)
Testing in Live Exchange
{ testing in live }physical view of the system logical view of the system
Testing in Live(set of representative
Acceptance Test)
{ other tests... }{ static analysis }{ invariant test }
{ configuration test }{ disaster recovery test }
{ culture }
Have Standup Meetings
{ standup }
Swarm on Stories
{ swarm }
Collectively Own the Code
{ everyone owns the code }
Pair Program
{ pair }
Don’t be a Jerk
{ be nice }
{ rotate }
Take Time to Sharpen Your Tools
{ sharpen your tools }
Share Your Knowledge
{ share your knowledge }
Share Your Code { share your code }
{ share your process }
@NickZeeb@ajmolinacom
Be Pragmatic
{ be pragmatic }
{ last responsible moment }
Avoid Irreversible Decisions
{ avoid irreversible decisions }
It’s OK to Change Your Mind
{ it’s OK to change your mind }
The Business Must Trust You{ trust }
The Most Important Thing You’ll Do
{ find the right people }
55
{ thanks! ;) }
@ajmolinacom@NickZeeb
Images thanks to
Classification: Public
• Icons and predesigned images: Pixabay http://www.pixabay.com• Rafael Matsunaga https://www.flickr.com/photos/rednuht/ • Philippe Lewicki https://www.flickr.com/photos/lewicki/ • Martin Fisch https://www.flickr.com/photos/marfis75/ • Talus https://www.flickr.com/photos/nocallerid_man/ • Kevin Jarrett https://www.flickr.com/photos/kjarrett/ • Jeffrey Beall https://www.flickr.com/photos/denverjeffrey/ • Improve It https://www.flickr.com/photos/improveit/ • Where is wally image https://upload.wikimedia.org/wikipedia/commons/2/27/Where%E2%80%
99s_Wally_World_Record_(5846121513).jpg• Vajrapani666
https://commons.wikimedia.org/w/index.php?title=User:Vajrapani666&action=edit&redlink=1 • MECU https://commons.wikimedia.org/wiki/User:MECU • Ehsan Lebatihem https://www.flickr.com/photos/lebatihem/ • Massimo Valiani https://www.flickr.com/photos/leader_maximo/ • Skip Steuart https://www.flickr.com/photos/omnidirectional/ • Karl-Ludwig Poggemann https://www.flickr.com/photos/hinkelstone/ • Open Clipart https://openclipart.org • Miguel Librero https://www.flickr.com/photos/dark_imp666/ • Police Cars of the Ohio Valley & the Great Lakes https://www.flickr.com/groups/1597741@N24/pool/38396373@N08/• Gaming4JC at English Wikipedia http://commons.wikimedia.org/wiki/File:3DCrystal_ball.png• some guy called Darren https://www.flickr.com/photos/59239214@N08/ • JD Hancock https://www.flickr.com/photos/jdhancock/