Care and Feeding of Large Web Applications

Care and FeedingCare and Feeding of of Large Web ApplicationsLarge Web Applications

Perrin HarkinsPerrin Harkins

Arcos, a project at Plus Three

About 2.5 years of continuous development

2 - 5 developers on the team during that time

~79,000 lines of Perl code

~4900 lines of SQL

(Computed with David Wheeler's SLOCCount program)

Plenty of HTML, CSS, and JavaScript too

~14000 automated tests in 269 files

169 CPAN modules

Arcos, cont'd

CMS with modern AJAX UI

E-commerce

Data warehouse with AJAX query builder GUI

E-mail campaign management

Asynchronous job queue system

Comprehensive reporting

Arcos, cont'd

Object-oriented

MVC-ish

Controller classes

Database classes

Templates

Core modules:

CGI::Application

Class::DBI

HTML::Template

Deployment

Hard to generalize

.tar.gz files

Always release full builds, not individual files

Consistent state for production system

QA installs are accurate copies

The CPAN problem

“Just use the CPAN shell”

Fails too often

Installs whatever the latest version is

CPAN installer requirements:

Install specific versions

Install from local media

Allow locally modified versions

Fully automated

“Erase your hard drive? [n]”

Install in a local directory

Skip the tests

Cluster install

Some will never work

Our solution

Finds all modules in src/ directory of package

(Including ones we hacked)

Uses Expect to answer questions

Builds non-CPAN stuff too

Apache

mod_perl

SWISH-E search engine

Why bundle dependencies?

Avoid troubleshooting local problems

If possible, specify Perl, MySQL, OS, etc.

Sometimes reality intervenes

Hardware support

Office politics

Next up: binary distributions

Compile takes too long

Still just tar?

RPMs?

PAR?

See build system in Krang: http://krancgcms.com/

OH HAI!OH HAI!

I HELP WIF URI HELP WIF URINDENTATION!INDENTATION!

Upgrades

Need full automation

Got the database part

Schema version number in database

Run all upgrade scripts with appropriate names

e.g. 2.0 --> 3.0 means run upgrade/V2_1.pm and upgrade/V3_0.pm

Hand-written SQL scripts

Test mode that upgrades from old schema first

LVM for rollback in QA

Configuration

● Highly configurable systems must be highly configured

● Started with httpd.conf style– Config::ApacheFormat

– Basic scoping and inheritance

Observations on Configuration

● People ignore options they don't understand– If the server starts, it must be ok!

● Some comments in a config file == weak documentation

● Things you rarely change shouldn't be in the shipped config file

I CAN HASI CAN HASMAINTENANCEMAINTENANCE

BRANCH?BRANCH?

Version Control

Many tool choices now

svn, svk, git...

Fight it out and let me know who wins

Most projects need at least two branches at all times

development

stable

Very simple version control

Main branch is development

Must build and pass tests

When making a release from a branch, tag it

Make a “2.0” tag when you release it

Very simple version control, cntd

Maintenance branch from that tag for bug fixes

“2.x” branch

On maintenance release, merge all changes on maintenance branch to main branch

Tag “2.1” on 2.x branch

Merge 2.0 --> 2.1 onto main branch

Too simple?

Not everyone is done at the same time

Hard to keep dev branch stable for major changes

Feature branches

Possible integration problems

Beware of complex merges

This is not the Linux kernel

Could be a people problem

An alternate point of view

http://utsl.gen.nz/talks/git-svn/intro.html

HALP!HALP!TESTS R TESTS R FAYLIN!FAYLIN!

Testing

● You all know the drill● Local library for common testing tasks

– Log in mech object

– Run test SMTP server● Test data setup and teardown

– Keep stack, delete in reverse order

– Probably easier to just wipe the db

Testing tools

● Simple Test::More scripts are easy– But scoping becomes an issue

– Test::Class

– Test::Builder● Improves test failure messages

● Test::WWW::Mechanize– Great for testing SSL, Apache config, mod_rewrite

– Works fine on JavaScript pages● But doesn't test the JavaScript● Selenium does

KTHXBYE!KTHXBYE!