Upload
chris-tankersley
View
134
Download
0
Embed Size (px)
Citation preview
Madison PHP Conference, September-October 2016 1
Oh Crap, My Code is SlowChris Tankersley@dragonmantankMadison PHP Conference, September-October 2016
Madison PHP Conference, September-October 2016 2
1.79 MHz 8-bit Processor128K RAM
640x192 max resolution64 color palette
RS-232 Serial PortCartridge Bay
2 Joystick Ports
Disk Extended Color Basic 2.1
Madison PHP Conference, September-October 2016 3
520 Mhz Apple S1512MB RAM
390x312 resolution (~303 ppi density)16 million colors
WatchOS
Madison PHP Conference, September-October 2016 4
32 Cores512GB RAM
1-10Gbps NICs10 Terabyte FusionIO Cards
Madison PHP Conference, September-October 2016 5
Madison PHP Conference, September-October 2016 6
“premature optimization is the root of all evil.”
Donald Knuth, “Structured Programming With Go To Statements”
Madison PHP Conference, September-October 2016 7www.phpbench.com
Madison PHP Conference, September-October 2016 8
It Doesn’t Matter
Madison PHP Conference, September-October 2016 9
We Are the 3%
Madison PHP Conference, September-October 2016 10
Who is at fault?• 3rd Party Connections• I/O Performance• Database
Madison PHP Conference, September-October 2016 11
Madison PHP Conference, September-October 2016 12
Madison PHP Conference, September-October 2016 13
Madison PHP Conference, September-October 2016 14
4 Hours• Cache Ad Campaign Data• Cache Analytics Data• Run Numbers
• Sync Products
Madison PHP Conference, September-October 2016 15
4 Hours• Cache Ad Campaign Data• Cache Analytics Data• Run Numbers
• Sync Products
Madison PHP Conference, September-October 2016 16
The Problems
Madison PHP Conference, September-October 2016 17
Running Numbers was heavy• Primary server would spike under load• Secondary servers would get out of sync and go into “rollback”
Madison PHP Conference, September-October 2016 18
5:20am 5:25am 5:30am 5:35am 5:40am 5:45am0
500
1000
1500
2000
2500
3000Replication Lag
Lag in Seconds
Madison PHP Conference, September-October 2016 19
5:20am 5:25am 5:30am 5:35am 5:40am 5:45am0
1000
2000
3000
4000
5000
6000Replication Lag
Lag in ms
Madison PHP Conference, September-October 2016 20
Product Sync was Slow• Just took hours to run
Madison PHP Conference, September-October 2016 21
Madison PHP Conference, September-October 2016 22
Product Sync
Madison PHP Conference, September-October 2016 23
Just Start Logging• Add DEBUG log messages with timestamps• Where is it slow?
Madison PHP Conference, September-October 2016 24
Seldaek/monologuse Monolog\Logger;use Monolog\Handler\StreamHandler;
// create a log channel$log = new Logger(‘job_debug');$log->pushHandler(new StreamHandler('path/to/your.log', Logger::DEBUG));
// add records to the log$log->debug(date(‘Y-m-d H:i:s’) . ‘ – Contacted API’);// Do our business logic$log->debug(date(‘Y-m-d H:i:s’) . ‘ – Finished with page’);
CulpritsMadison PHP Conference, September-October 2016 25
Madison PHP Conference, September-October 2016 26
What did we find?• All calls to Product API had to be full sets, couldn’t subset• Calls to Product API were slow, but not horrid• Generating and inserting the Products were slow due to business logic• Blocked Operations:
• Getting next page from API• Processing products
Madison PHP Conference, September-October 2016 27
Our Workflow// Original WorkflowGet Page X from API
For Each Product:Extract Data from XMLTransmogrify the Data into a Product ObjectSave Object to DBIf No Next Page:BreakElse:Page++Continue
Madison PHP Conference, September-October 2016 28
Solution – Out of Band Processing// Original WorkflowGet Page X from API
For Each Product:Extract Data from XMLTransmogrify the Data into a Product ObjectSave Object to DBIf No Next Page:BreakElse:Page++Continue
Madison PHP Conference, September-October 2016 29
Solution – Out of Band Processing// Job 1 - Cache Product API CallsGet Page X…X+10 from API
Cache XML to DatabaseIf No Next Page:
BreakElse:
Page++Continue
Call Job 2Respawn Job
Madison PHP Conference, September-October 2016 30
Solution – Out of Band Processing// Job 2 – Insert ProductsGet Page X…X+10 from DB
For Each Product:Extract Data from XMLTransmogrify the Data into a Product ObjectSave Object to DBIf No Next Page:BreakElse:Page++Continue
Madison PHP Conference, September-October 2016 31
Run Totals
Madison PHP Conference, September-October 2016 32
Madison PHP Conference, September-October 2016 33
Background• Was originally PHP• Turned into a MongoDB Script because it was too slow
Madison PHP Conference, September-October 2016 34
5:20am 5:25am 5:30am 5:35am 5:40am 5:45am0
500
1000
1500
2000
2500
3000Replication Lag
Lag in ms
Madison PHP Conference, September-October 2016 35
Madison PHP Conference, September-October 2016 36
What did we find?
Madison PHP Conference, September-October 2016 37
Check the Server Metricshttps://aws.amazon.com/blogs/aws/new-cloudwatch-metrics-for-amazon-ebs-volumes/
SuspectMadison PHP Conference, September-October 2016 38
Madison PHP Conference, September-October 2016 39
Our Solution – Throw Hardware At It
Madison PHP Conference, September-October 2016 40
Our Solution – Throw Hardware At It• Increased IOPs on the SSD’s• Larger Instances on AWS
Madison PHP Conference, September-October 2016 41
Our Solution – Move out of MongoDB• Rewrite the script back into PHP• Run in our worker system
Madison PHP Conference, September-October 2016 42
The Result
Madison PHP Conference, September-October 2016 43
The New Bug
It took 5 hours to run
SuspectMadison PHP Conference, September-October 2016 44
Madison PHP Conference, September-October 2016 45
Madison PHP Conference, September-October 2016 46
Madison PHP Conference, September-October 2016 47
What did we find?
Madison PHP Conference, September-October 2016 48
Code Profiling
Madison PHP Conference, September-October 2016 49
xhprof• Low-cost dynamic analysis for PHP• PHP Extension• Store results in a DB• Has a pretty good GUI• https://www.digitalocean.com/community/tutorials/how-to-set-up-x
hprof-and-xhgui-for-profiling-php-applications-on-ubuntu-14-04
Madison PHP Conference, September-October 2016 50
Pretty Graphs
Madison PHP Conference, September-October 2016 51
Useful Metrics
Madison PHP Conference, September-October 2016 52
What we find?• Hydrating objects was expensive• We were doing deep hydration, resulting in extra DB and hydration
calls• We had authentication checking happening in a loop, due to bad
logging code
Madison PHP Conference, September-October 2016 53
The result?
It brought it down to around 3.5 hours
Madison PHP Conference, September-October 2016 54
Valgrind• General programming tool for checking memory debugging, memory
leaks, and code profiling• Supported by xdebug• KCacheGrind/QCacheGrind to view output
Madison PHP Conference, September-October 2016 55
Enable it in xdebugzend_extension=/usr/lib/php/20151012/xdebug.soxdebug.profiler_enable=1xdebug.profiler_output_dir=/var/www/tests/xdebug
Madison PHP Conference, September-October 2016 56
Function Calls and Code Flow
Madison PHP Conference, September-October 2016 57
What we find?• We were looping a lot• We were looping big loops inside small loops• We were looping through a lot of the same data multiple times
Madison PHP Conference, September-October 2016 58
The Result – Reduce the Looping
Runtime was reduced to 30 minutes
Madison PHP Conference, September-October 2016 59
Tips for Slow Code• Use Monolog to add Debugging messages• Use xhprof to profile “live” code• Use xdebug and Valgrind to get deeper profiling
Madison PHP Conference, September-October 2016 60
Thank You!• https://github.com/dragonmantank• Author of “Docker for Developers”
• https://leanpub.com/dockerfordevs
• http://ctankersley.com• [email protected]• @dragonmantank
Madison PHP Conference, September-October 2016 61
Credits• Slide 13 – Andrei.D40 – Stacks of Books, Flickr• Slide 34 – Upper Snake River Valley Historical Society – 3339 loggin,
Flickr