1

Delivering High Performance Plone

Internet Development Group (ILRT)Dominic Hiles

2

Outline

• Measuring performance and requirements

• Profiling code• Native caching in Zope and ZEO• Using Zope’s RAMCache Manager• Caching externally e.g. Squid• Scaling it all up -> load balancing and

redundancy• What’s on our horizon?

3

Measuring performance

• Decide how much performance you actually need– Easier when migrating an existing system, just analyse the

web logs and/or use monitoring tools– Allow plenty of headroom

• Use benchmarking tools, e.g. Apache Bench to assess performance before and after changes– Test different content types in isolation e.g. web pages, file

objects, images, CSS– Consider authenticated vs. anonymous access– Use varying numbers of requests and request concurrency– Use a reasonable sample size (for example all Web pages

on the site)

4

Profiling code

• Identify the “slow” content types with benchmarking

• Profile the code using appropriate tools, e.g. ZopeProfiler, PTProfiler

• Analyse the slowest parts of the code in more detail and optimise, but don’t necessarily aim for perfection!

• Re-test when you’re done to confirm your diagnosis

5

Hassle-free caching• Zope provides caching “out-the-box”• The ZODB cache size controls the number of

objects loaded from the ZODB and held in a RAM-based cache– If you’ve got lots of memory, make this bigger (e.g. 5,000 –

20,000 objects)

• ZEO also provides a disk-based ZEO cache, which holds objects loaded from the ZEO server– You’re likely to want to make this bigger than the default,

e.g. 100MB

• Performance of both can be assessed using standard Zope tools

6

More complex caching

• Zope provides a RAMCacheManager– Stores the results of e.g. a Python Script or Zope Page

Template in RAM. Results should therefore be lightweight and/or not too numerous.

– Useful for:• objects that change rarely e.g. a “Not Found” (404) response• objects that change according to a limited number of

parameters e.g. month of year• other specialised objects that you may have written to take

advantage of it– In a ZEO setup, the cache exists (and may be different) on

each ZEO client• This can complicate the process of programmatically removing

objects from the cache when stale

7

HTTP Caching• Zope/Plone can set a series of headers on the content it

serves. These determine how long the content should be considered “fresh” for

• Upstream caches (either your own or external ones) and users’ browsers inspect these headers and hold the content for the specified time, checking back for updated content when necessary

• Cache strategies are complex to implement and depend on your specific configuration

• CacheFu (a Plone product) provides some common strategies and some good documentation. The Cacheability Engine is useful for testing and also provides some good documentation.

• Once you have the correct headers, just implement your own cache…

8

Implementing an HTTP Cache

• Caches serve content much more quickly than Plone can, as they don’t need to generate the content, just serve it– Say, 2 requests/second vs. 100s – 1000s requests/second

• A good caching strategy should see most of your content (~90% in our case) being served from this cache, minimising the load on your Zope servers

• Use whatever suits your environment e.g. Squid, Apache 2.2 + mod_cache

9

Scalability and redundancy• ZEO

– Client/server architecture– ZEO Clients do the “hard” processing work and can be

spread across different servers– This provides redundancy and scalability– Bottleneck may just move to ZEO server instead!

• Keeping a “warm” copy of the ZEO server as well, gives a fully redundant Zope system

• Multiple ZEO clients need something to distribute the traffic between them -> a load balancer e.g. Pound

• Ensure that this redundancy is carried through the rest of the architecture

10

Bristol CMS Architecture

ZEO server

ZEO server ZEO client

ZEO client

ZEO client

ZEO client

ZEO client

ZEO client

Pound load balancer

Squid cache and redirectors

Pound load balancer

Squid cache and redirectors

Apache and onward links to other parts of

the UoB Web

Failover solution

implemented along lines of

Linux HA project, using IP takeover

Apache and onward links to other parts of

the UoB Web

Manual failover

12

Summary

• Measure performance before you start• Profile code and fix any bottlenecks• Use the “free” Zope caching• Use Zope’s RAMCache Manager• Implement an HTTP cache• Use ZEO and load balancing to create a

redundant and scalable system

13

Other ideas we’re looking at• memcached – a generic cache that could be

shared across all the ZEO clients (even on different servers)– May use this in preference to Zope’s RAMCacheManager– Would allow cached SQL queries to be shared across all

servers

• Splitting out the Data.fs, mounting different sites and/or the portal_catalog separately

• Increasing RAM • Using multiple “throw away” servers for the

ZEO clients• Improving the specification of the ZEO server