Mistakes Made and Lessons Learnt Scaling Plone post-Launch

Matt HamiltonMatt Sital-Singh

Plone Conference 2011San Francisco

Matt Hamilton&

Matt Sital-Singh



Background0



• Learning Management-type system

• A lot of content

• Users ‘editing’ content, but not necessarily aware of it

• Lots of ‘clustering’ of load on resources

Use Case



Scale

• Plone 3

• User accounts: 30,000

• Messages: 160,000

• Course questions: 260,000

• Other: 60,000



Don’t Create Ghost Content1



• User perception of speed

• If they expect a ‘read’ they want a ‘read’- user signups- course starting

• Will accept a delay if you are actually processing something



Use Optimization Products2



• experimental.catalogqueryplan

• experimental.contentcreation

• experimental.daterangeindexoptimisations

• experimental.aggressiveopaquespeedup

• collective.indexing



One Plone site or Multiple?3



• One Plone site per ‘site’

• Multiple ‘sites’ per Plone site

• Common content?



Large Catalogs are Slow4



• We had 260,000 questions

• each had 2-4 answers

• = 520,000 - 1,400,000 answers

All in portal_catalog!



Optimize your Catalogs5



• Pre-load Catalog Queryplan

• Balance your BTrees



1319817170

1319817198

1319730857

1319126057

1308585257

effective date index:



1308585257

1319126057

1319730857

1319817170

1319817198



• Used catalog multiplexer to assign ‘Answer’ content type to dummy catalog

• Moved messages to separate message_catalog

• Monkey-patch items out of the catalog?



If You Know Where Something Is - Don’t

Search for it6



• Got into bad habit of using the catalog ALL the time

• Don’t just blindly use catalog when unrestrictedTraverse / contentValues will do

• RAM is cheap, user patience is NOT



Use unrestrictedSearchResults

if possible7



• unrestrictedSearchResults does not check:

• effective index

• allowed_roles_and_users index

➡ A lot less work for the catalog to do



Is Plone the Right Tool?8



• “When you have a hammer, everything looks like a nail...”

• A lot of this data is relational...

• Not much of it is CMSish

• Maybe should have used Pyramid?



Use Event Driven Workflow

(Don’t recalculate)9



• User rating / Course ranking

• Use workflow event subscribers, don’t re-calculate on view every time



Some things don’t need to be immediate10



• “You have 10 new messages”

• Not everything has to be updated all the time in every location on the site



Thanks!

Technology

Mistakes Made and Lessons Learnt Scaling Plone post-Launch