Upload
dlieberman
View
8.177
Download
7
Tags:
Embed Size (px)
DESCRIPTION
Slides from the workshop "Scaling the LAMP Stack" at the Future of Web Apps on October 5, 2007
Citation preview
Scaling the LAMP StackFuture of Web AppsOctober 5, 2007
Introductions
Specific Problems, Challenges and Issues
About this workshop
• This is a broad topic
• Theory and application
• Real-world focus
• Interactive (please!)
About web apps and scaling
Some different ways of looking at the problem…
Things to think about• Multi-server: locking and
concurrency• Running many: keep in mind
what’s expensive, sloppy or risky• Code quality• The law of truly large numbers
Elements of Scaling• Split up different tasks• Use more hardware (intelligently)• Partition• Replicate• Cache• Optimize (code and hardware)• Identify and fix weaknesses• Manage
Tools and Components
• Apache + PHP
• MySQL
• File System (local)
• Networked File System
• Load Balancers
• memcached
Contemplating Scaling
• Understand what your app does (and how much)
• Identify the bottlenecks
• Solve near-term problems
• Design well, but don’tover-design
Web apps do lots of things
Different operations have different scaling issues.
What does your app do?
List the high level elements of what your application does. Separate out different functions that will have different scaling issues.
Common things that web apps do
• Manage connections/protocols
• Deliver static content
• Manage sessions
• Manage user data
• Render dynamic pages
• Access external APIs
• Process media
Update the list of things your app does
• Add anything you missed
• Note which items you do in quantity
Easy vs. Difficult Scaling
What happens when you add hardware?
•Does it work?
•Does more hardware = more performance?
Things that break when you scale
• State that isn’t properly shared (especially sessions)
• Updates/refreshes (caching and replication issues)
Things that don’t improve when you add more servers
• Unpartitioned databases
• Anything that locks/blocks
• Inefficient code, especially big queries
Scaling Each Element
• (do easy separations first)
Managing Connections/Protocols
• No problem putting on multiple servers
• Apache is goodo Not too far away out of the boxo Moderately tunable
• Linux tuningo TCP stack (tune to handle unusual
networking needs)
Key Apache Configuration Issues
• MaxClients (and ServerLimit, ThreadLimit and ThreadsPerChild)
• Avoid using PHP (or other) handler unnecessarily
• Use the worker MPM
• Maybe MaxRequestsPerChild
Delivering Static Content
• Don’t process it unnecessarilyo Either cache or use no Apache
handlerso Caching can let you treat semi-static
content as static
• Multiple servers complicates updates, but is otherwise easy
General Discussion:Multi-server, state and sessions
Rethinking state for multi-server environments
• What is state?
• Short-term state (sessions)
• Long-term state (application data)
• Managing state is usually the hardest part of scaling
What happens with state
• Written (created/destroyed/changed)
• Read
• Stored
Requirements for managing state
• Depend on what it is and how it is used
• Perfect coherence
• Performance of different operations
Ways of scaling state
• Replication: make more copies
• Partitioning: split up the work
• Caching
Should make different choices for different state/data elements
About Load Balancers
• What load balancers doo Spread loado Detect server failureso Stickiness/persistenceo Acceleration (especially SSL)
• Fancy features (including good stickiness) are expensive
Why sticky sessions are not usually good in practice
• Servers fail
• Corner cases exist
Managing Sessions
Where session data can be stored
• Browser cookies
• Web server temporary files(not scalable)
• App server state
• Database
• Cache
PHP session management
• Default (files) method is not multi-server friendly, and thus not scalable (unless sticky)
• Can implement a different back-end easily
Designing a session back-end
• Requirements
• Data storage optionso Cookies only (re-auth, let the browser
take care of the logout – but less secure)o Full-featured involves a combination of
cookies and database and cache
(discussion of session details)
Managing small user data
• Databases are more efficient, flexible and sharable than small files
• Frequently-read data should be cached
Managing large user data
• NFS has flaws but is almost inevitable
• Locking is usually not important, but can be
• Performance degradation can be sudden
About NFS• NFS is usually transparent to your
app• NFS is easy to implement gives you
multiple-write access• NFS locking is not to be trusted• The Linux NFS client is slow for writes
and can do bad things under stress
User data and locking
• Names based on hashes often mean no locking is needed
• Databases do locking better than file systems do
• Locking requires housekeeping
Disk Storage Hardware
• Disk performance can degrade suddenly
• If the ratio of access to storage is low, then even slow disk is usually fine
• Think about seek times and spindles
Rendering dynamic pages• Depends heavily on application
specifics (query, search, process, etc.)
• Watch out for:o Onerous queries (create and watch slow query log)o Locking of resources and/or incoherence if state changeso Heavy CPU and memory usage
• Cache both elements and complete pages
Processing media
• CPU intensive
• May be memory intensive
• Might be spiky
• Might need its own server pool
Hardware
• Start simple
• Observe performance and respond accordingly
• Get lots of memory
Hardware-driven behaviors
• Sudden degradation because demand exceeds supply (usually relieved unhappily)
• Get behind due to a spike, and recover
• Not enough resources for normal optimization
Specific hardware issues
• Not enough memoryo Severe: paging/swappingo Mild: poor automatic caching; slowness
due to fragmentation
• Disk seek (very common)
• CPU (but might really be memory)
• Disk throughput (rare for web apps)
Hardware decisions
• SCSI/SAS vs. SATA
• Resource ratios
• Combining vs. splitting functions
• Big vs. little boxes
Techniques
• Caching
• Partitioning
• Replication
• Data management middleware
• Queuing
Caching• Turn expensive operations in to
cheap ones• Reduce:
o Database readso Object and page calculation/rendering
operations
• Cache objects and subobjects• Add memory
Apache Caching
• Can be done with zero application modifications
• Complete pages/HTTP requests only
• Must use Apache 2.2
• Cache is not shared between servers
memcached
• Extremely useful
• Distributed caching system
• Requires new thinking and new coding
• Straightforward API
memcached URLs
• Home: http://www.danga.com/memcached/
• Intro: http://www.majordojo.com/2007/03/memcached-howto.php
• PHP documentation: http://www.php.net/memcache
Partitioning• Mostly for data management
• Split load on to separate servers/pools
• Partition algorithm/mechanism must be lightweight
• Partition algorithm must anticipate the future
File Storage Partitioning
• Index/database gives the most flexibility
• Hash-based is simplest
Database partitioning
• You will need to do this, but perhaps later than you think
• Index vs. hash-based
Replication
• Used where data is read far more than written
• Consider caching first
• Also used for failure recovery
Types of Replication
• Replication: sync vs. asynco Synchronous is not usually scalableo Asynchronous only works with certain
kinds of data and use cases, because of coherence issues
Database Replication
• Simple but finicky
• Asynchronous (but not by much)
• Allows big queries and backups to be moved to separate servers
File System Replication
• Slow and very asynchronous
• Mostly for disaster recovery
Data Management Middleware• Mostly for databases• Can handle partitioning and
replication, and do it well• Big investment in coding to the
API• Sometimes easier to add
functionality to app
Queuing
• Save work for later
• Useful for less urgent operations, especially messaging
• Can be used to wait for a pause, or to separate hardware
Dealing with lots of hardware(operations)
• Automation
• Process
Imaging/Provisioning
• Be consistent
• Use your distro’s automation (Kickstart, AutoYaST, etc.)
• Use boring, meaningful hostnames
• Make re-imaging easy
Deployment Systems• Content and code replication• Coherence/atomic updates• Managing pieces and processes• Simple scripts are fine• Create audit trail• Include back-out• Think 3AM• Do it!
Monitoring systems
• A pain, but a lifesaver• Start with built-in basics• Add custom checks, especially end-
to-end and communication between pieces
• Eliminate false alarms (ongoing)• Nagios, usually
Coping with hardware failure• Have extra servers/capacity• Load balancers handle stateless
layers• Replication prepares you to handle
data layers manually• Use middleware or app-level multiple
writes to get true data layer redundancy
Change management
• Part automation, part process
• Use version control on everything
• Stage changes with realistic data
• Know how to back out
• Consult the right people (internal and/or external)
Efficiency
• Access the smallest amount (DB, FS, etc.)
• Don’t do complex stuff when simple will suffice
Using the database efficiently• Keep it simple• Know what queries you do• Index every query key• Cache to reduce demand• Check slow query log• Replicate if you need big queries
The messy real world
Security and abuse
• Mostly same issues, just magnified
• You will be a target
• Spam (coming and going)
• Abuse of file storage
Corner Cases
• Murphy’s law enforcement
• Watch out for how different user activities relate
• Lock data, not functions
• Housekeeping
Performance and tuning• Observation and responsiveness is
more important than pre-optimizing• Redesign as needed• Collect the data to be able to
analyze (both resource utilization and end-user performance)
Miscellaneous Warnings
Files and directories
• Most default file system configurations get really slow with lots of files in one directory
• Numerical limits on files and subdirectories
• Some programs don’t like files over 2GB
AJAX
• Sequential round trips
• Make preloading invisible
• UI that waits for too many things
Other topics
• Multiple sites
• CDNs
Scaling the LAMP StackFuture of Web AppsOctober, 2007
Daniel Lieberman [email protected]