Best Practices for Optimizing Blackboard Learn

Steve Feldman, sfeldman@blackboard.com

optimizing* The process of increasing speed (performance) and capacity (scalability) of a system and/or program.

What We’ll Cover

•  Deploy for performance from the start.

•  Optimizing the platform components.

•  Continuous measurements are absolutely critical.

Scalable Deployments

Flexible and Scalable Application Deployment

Deployment: Large Address Space •  As of Blackboard Learn™ Release 9.1 all supported/

certified configurations include a 64-bit option.

•  Pushing more processing to client and DB over the last few releases, but major memory management technique is to use more application caches. –  Memory stays persistent longer –  Less wasteful from a creation/destruction perspective, but puts

greater demands on larger spaces. •  Most of our application testing focused on 4GB and 8GB

JVM deployments on 6GB and 10GB OS spaces. –  Limited testing at 16GB and 32GB

Deployment: Resource Utilization via Virtualization

•  Moore’s law is in full effect –  CPUs are getting faster with more cores –  Memory is in abundance and cheap –  Storage is grossly abundant

•  Massive systems can be obtained at low cost, but cannot be saturated in stand-alone configurations.

•  Virtualization offers this opportunity… –  Deploy with availability in mind –  Saturate system resources –  Save money and data center space

Deployment: Improving Page Responsiveness

•  Gzip…Gzip…Gzip… –  All of our supported browsers handle gzip? –  Reduces payload

•  Improves lower latency connections like Cable, DSL and Dial-up –  Minor overhead on the application layer (~2% to ~5%)

•  Have the option to perform at the load-balancer layer –  Most Bb deployments do not enable Gzip at all

•  Even when enabled, some proxies and software packages mess-up the Accept Encoding Header

•  Optimize your images –  Page size really does matter –  Reduce the size without reducing the quality –  Smush.it, PunnyPng, OptiPng, RIOT and ImageOptim

Example Page Waterfall

Tools for Page Performance •  Analysis Tools

–  Independent tools: Fiddler, HTTPWatch, WebPageTest.org, Web Inspector (Mac), IE Developer Tools

–  Browser Plugins: Firebug/YSlow, Google Page Speed, WebPageTest Client

–  JavaScript tools: dynatrace, IE Developer Tools, •  Optimization Tools

–  Image optimization to reduced overall size and metadata: SmushIt, PNGOptimize, PNGCrusher, OptiPng and graphicsoptimization.com

•  http://yuiblog.com/blog/2008/10/29/imageopt-1/ –  CSS optimization to reduce unused elements and compress

CSS of unnecessary white spaces: CSS Optimizer, CSS Compressor and CleanCSS

Optimizing the Web Server •  The web server in the BbLearn configuration is nothing

more than a gateway to the application container. –  When clusters were more relevant, the web server acted as a

pseudo load-balancer. •  Not many opportunities for optimization other than

–  KeepAlives –  Interfaces –  Compression

•  It can become a bottleneck if not properly optimized –  Better to have high ceilings from an interface perspective

Optimizing the JVM •  Java hotspot offers standard –X and non-standard –XX

options for performance and behavior. –  -X options are always guaranteed across releases and patches of

Java. –  -XX options must be used with caution as they are subject to

change with any release of Java. •  -XX options should be tested and measured using the

production safe arguments. –  Be careful of leveraging options you see on ListServes and

Google •  Read the release notes of Java for “performance” updates

–  http://java.sun.com/javase/6/webnotes/ReleaseNotes.html

Regions of the JVM

Optimizing the JVM •  Cross-platform recommendation for using Concurrent

Mark Sweep Collector –  Best optimized for 64-bit address –  Combine –XX:+UseConcMarkSweepGCwith –XX:+UseParNewGC

•  Manually size New Space using –XX:NewSize and –XX:MaxNewSize options (1/4 to 1/3 total heap). –  Consider Survivor Space ratios 4 or lower.

•  Be careful about sharing –XX non-standard options across customers. –  If you don’t understand what the option does and it’s not

recommended by Blackboard, best choice is to not use it. •  PermSpace and B2s

–  Every B2 you add, make sure to measure PermSpace usage via tools like JStat and VerboseGC

High Level Old Space Collection

Prototyping JVM Arguments •  Wrapper.conf exists on both

UNIX and Windows configurations for handling JVM arguments.

•  Options can be tested out without a PushConfigUpdates by simply adding an additional line with –  wrapper.java.additional.<insert

numeric sequence>= –  Simply issue a “./

ServiceController services.appserver.restart” to restart the JVM without arguments

PrintGCStats

Tools for Java Instrumentation •  GC Log Analysis Tools: PrintGCStats, Jstat,

GCViewer,

•  Thread Analysis Tools: Samurai, JConsole, JavaVisualVM,

•  Memory Analysis Tools: Eclipse Memory Analyzer, Jhat,

•  Robust Commercial Tools: Quest Foglight, Oracle OEM, Dynatrace for Java, CA IntroScope

•  Profiling Tools: Jprobe, YourKit, JProfiler

Optimizing the Database: SQL Server

•  # of data files makes no difference on SQL Server for Data and Transaction

•  Allow the data/transaction files to grow as big as they want within reason. –  What’s reason: 64GB –  http://msdn.microsoft.com/en-us/library/ms143432(sql.90).aspx

•  TempDB is completely different story –  # of files = # of DB Threads –  Set first X files to a uniform size, set last file to same size with

auto-extension ON –  Determine size need over time

•  Separate volume for paging file

Optimizing the Database: SQL Server

•  Be aware of MDOP: Max Degree of Parallelism –  Setting to unlimited can have a negative affect on query

performance unintentionally. •  AWE can and does work on 64-bit systems

•  Configure READ_COMMITTED_SNAPSHOT

•  Two nuggets of information: –  Learn How to Use SQL DMVs –  Study SQL Server Wait Events and Tuning

Optimizing the Database: Oracle

•  Balance I/Os across multiple data files (~2 to 8GB per file).

•  REDO is critical to performance a session/query level. –  Be aware of how much REDO is being used over time. –  NOLOGGING will disable, be we rarely use NOLOGGING

•  TEMP is very complex and used for managing transient data. –  One TEMP file is adequate –  If latency exists on TEMP, consider introducing TEMP file groups

•  SGA is important, but PGA can be your best friend or your worst enemy with high concurrency.

Optimizing the Database: Oracle

•  Oracle CBO can be your friend –  Must understand optimizer behavior –  Willingness to read Cost Execution Plans

•  Using Wait Events and Cost Execution Plans for tuning initiatives –  Wait events are at a system, session and query level

•  Importance of Statistics and Histograms –  CBO is just guessing without properly set statistics and

histograms. –  CBO is dependent on your data.

Approach to Understanding DB Performance

•  Capture statements using some pre-identified tool –  Oracle AWR and ASH (Session Level) –  SQL Server Performance Dashboard and Performance

Warehouse •  Review Instance Level Information to gain a cursory view

of “system” performance.

•  Deep dive into SQL Analysis –  Focus on Logical I/Os as starting point.

•  Physical I/Os are important, but more often logical –  Drill into CPU consumers followed by executions –  Isolate physical and structural for over usage and possible

locking conditions –  Question why executions, but 0 rows processed –  Importance of both: Histograms and Statistics

•  Know your data better than your DBAs

Please provide feedback for this session by emailing BbWorldFeedback@blackboard.com.

The subject of the email should be title of this session:

Best Practices for Optimizing Blackboard Learn