Condor In Flight at The Hartford2006 Transformations

Condor Week 2007

Bob Nordlund

About The Hartford…

• Headquartered in Hartford, CT• Founded in 1810• Fortune 100• 31,000 employees worldwide• $26.5 billion revenues• $2.9 billion core earnings• $377.6 billion assets under management

The Hartford’s Businesses

• Property & Casualty• Auto, home, marine, workers compensation, etc.

• Retail Investment Products• Variable and fixed annuities, mutual funds, 529 college

savings plans

• Retirement Plans• 401(k), 403(b), 457

• Institutional Financial Solutions• Individual Life Insurance• Group Benefits• International

A Brief History…

• Exponential growth in risk modeling activity exceeded existing computing capabilities.

• Grid technology was identified as a possible solution.

• A pilot program with a commercial grid provider was started.

• In parallel, a ‘skunkworks’ Condor implementation was evaluated as an alternate approach.

• Condor was selected over the commercial provider.• Windows Support• Simple, Scalable, and Flexible• Active Community• Free

2006 Transformations

• GRAPE• Condor 6.8 Upgrade• Accounting and Usage Reporting• GridMC• User Community

GRAPE Challenge(Global Risk Analysis Projection Enhancement)

• Prepare comprehensive risk analysis report for all products for investment analysts.

• Run three orders of magnitude more projections through the system.

• Perform stochastic on stochastic on stochastic analysis.

• 2.4 billion individual simulations.

GRAPE IT Response

• Infrastructure• 500 dual-core, dual-CPU AMD servers

• 20 Racks

• Dedicated GigE switches

• Condor pool with HAD

• 4 filers with 8TB of storage

• Purchased, installed and configured in 6 weeks

• Application• Java-based liability model

• 75x performance gain

• Meta-scheduling application

• In-memory market scenario generation

Condor 6.8 Upgrade

• Dual version Condor MSI installation• Centralized configuration management• Multiple schedulers

• 6.8 shadow process overhead

• Load balancing scripts

• Scheduler coordination

• Accounting Groups• SOAP based submission• Mixed Linux Scheduler/Windows Execute• High Availability Central Managers• 1000+ desktops, notebooks, and workstations

Accounting and Usage Reporting

GridMC (Grid Management Console)

• Web-based view of the grid• Administrative tools

• Configuration management

• Policy management

• Condor Daemon Management

• User-priority / Group Quota Management

• User tools• Job submission

• Job control

• Prioritization

• Job Monitoring

• Grid utilization

• Usage Reporting

User Community

• Weekly user group meetings to discuss issues, planned usage, allocation trading.

• Users own the grid, IT provides care and feeding

• Promote ‘good’ behavior by highlighting ‘bad’ behavior.• Users who don’t want to share can’t take advantage of

others who do – and are prominently displayed on usage graphs as ‘exclusive’.

• Shame and ridicule can be effective tools to manage a grid.

• Avoid draconian policies by providing transparency to users and letting a cooperative community evolve.

Our Condor Environment…

• In production since 2004• Three pools (GRAPE, GA, Test)• Dedicated and non-dedicated execute nodes

• ~1000 Two-socket, dual-core HP x86 servers• ~1000 desktops, notebooks

• Linux central managers with HAD• Linux and Windows schedulers• Windows execute nodes

Conclusion

• The GRAPE project won The Hartford’s 2006 Chairman’s Award

• Condor has been a transformational technology providing users access to capabilities they wouldn’t have otherwise had – or can now live without.

• Grid computing is an integral part of our business and places The Hartford in a leadership position with respect to risk analysis in our industry.

• Condor has proven to be an invaluable asset and has time and again handled whatever challenge we’ve thrown at it.

Thank you Condor Team!