Perfomance tuning on Go 2.0

© 2010 ThoughtWorks, Inc. All rights reserved.

Performance Tuning on Go 2.0http://www.thoughtworks-studios.com/go/

Yogi Kulkarni


GoContinuous integration Release management



Stage-2Unit Test

Run Unit Test Suite - 1


Stage-3Acceptance Test

Accetance Test Suite - 1

Acceptance Test Suite – 2

Stage-1Compile

CompileJob

Pipeline



Pipeline Dashboard



Pipeline Dependencies


The Problem


Slow page loads

Problem Investigation Setup Solution


Jobs taking long to get assigned to agents



Jobs getting rescheduled



Artifact uploads timing out



Server VM freezing



Lots of concurrent activity


What was happening under the hood?Profiler revealed the culprits



Thread blocks


Threads blocked on the database connection pool


So we introduced asynchronicity



Big mistake!



Find the root cause




Problems

Too many queries

Slow queries

Coarse-grained synchronized blocks

JRuby and Rails issues


Problems Solutions

Too many queries Caching

Slow queries Query tuning

Coarse-grained synchronized Reduce lock granularity

JRuby and Rails issues Hack and pray!


ObjectiveNo request should take > 100 ms

Most should be under 50 ms



Setup & Tools



Conservative hardware



“Performance Server”Dell 620 laptop

2 Cores, 2GB RAM


Large Dataset5GB merged database

Large config file with 100 pipelinesReal-world data



Extreme load70 agents

Each Job generating lots of console outputEach Job uploading 10 MB artifacts

Apache Bench / Httperf



Automation for fast feedbackCapistrano script

Feedback cycle down from hours to 10 min



Caching


Caching Locks Query Tuning Jruby & Rails


Single application-wide cacheDomain objects

DTOsRails view fragments





EHCacheIn-memory only

Least-Recently-Used eviction policy




Transaction rollbacks == Incorrect cached dataIgnore cache puts in transactions

Invalidate caches on transaction commit only




Spring to the rescue


Fast deep-cloning of cached objectsJava Deep Cloning Library: http://robust-it.co.uk/clone/index.php



http://robust-it.co.uk/clone/index.php


Locks




Be suspicious of synchronized methods




Lock on interned stringsFine granularityProfiler support

Also used as cache keys





Use ReadWriteLocks for highly contended code




Use “volatile” fields for lockless reads




Slow


Not synchronized


Acquire locks before starting transactionsTo enforce aggregate invariants






Stage(Building)

Job – 1(Completed)

Job – 2(Building)

Stage(Building)

Job – 1(Building)


Thread 1

Thread 2

Stage(Building)

Job – 1(Building)

Job – 2(Building)

Stage(Building)


Job – 2(Cmpleted)





Query Tuning




Use a large databaseUse the query analyzer to find missing indexes




Views are great… sometimes10x-20x faster for “TOP n” queries

Depends on database implementation




Query selectivity is important200 ms

Stages(70,000 rows)

Pipelines(20,000 rows)

BuildStateTransitions(4,000,000 rows)


2 ms

Introduce an n+1 query


JRuby on Rails Hacks




Single JRuby runtime + Rails multi-threaded mode

Much lower memory footprintBut completely uncharted territory




Severe thread blocks in JRubyReported to JRuby community

Fixed immediately in 1.5.0 & 1.5.1




Rails nested partials Mysterious lock contention

10 concurrent requests take 90 secs to completeStill a mystery!




Desperate situations, desperate measuresStatically inline partials at build time!




Routes subsystem is slowCache url_for




LoggingReplace Rails BufferedLogger with Java Log4jLogger

Turn off Rails logging in production




Fragment cachingGenerate a cache-key that incorprates all fields that are needed to

uniquely identify that object’s current stateLRU cache will evict it on state change




“pipelineDashboardFragment|dev|2[compile,1,passed][test,1,building]”

Key-identifier Pipeline-name Stage detail Stage detail


“pipelineDashboardFragment|dev|1[compile,1,passed][test,1,building]”

“pipelineDashboardFragment|dev|2[compile,1,passed][test,1,passed]”

Key-identifier Pipeline-name Stage detail Stage detail


Lessons LearnedFind the root cause

Use a profilerMeasure before and after each fix



How we got from that to thisINSERT IMAGE

Threads after performance tuning


http://www.thoughtworks-studios.com/go/

Technology

Perfomance tuning on Go 2.0