Global Business Services
© 2007 IBM Corporation
Considerations for Developing a Performance Measurement System
MTAC Workgroup #114April 13, 2007
© 2007 IBM Corporation2 Performance Measurement Methodology 4/13/2007
Agenda
Introduction
Performance Measurement Methodology
General Study Considerations
Sample Design
Mail Fabrication
Induction
Receipt
Data Analysis
Reporting
Summary
© 2007 IBM Corporation3 Performance Measurement Methodology 4/13/2007
Introduction The USPS requested that we present today based on the
requirement from the Postal Accountability and Enhancement Act to implement performance measurement for all Market Dominant Products
IBM’s Background in External Performance Measurement– Have conducted USPS transit-time measurement studies since 1990
– Mail measurement experience in more than 35 countries on various domestic and international initiatives
– Methodology developed specifically for this work• Simple in concept
• Complexity is in all the details
© 2007 IBM Corporation4 Performance Measurement Methodology 4/13/2007
Performance Measurement Methodology
Quality Assurance
StatisticalDesign
Test MailFabrication
Test MailInduction
Test MailReceipt
DataAnalysis
Reporting
Panel Management
© 2007 IBM Corporation5 Performance Measurement Methodology 4/13/2007
General Study Considerations
What objectives need to be met?
Who are the various stakeholders?– What are their interests and priorities?
Standards and best practices– What are other countries doing?
© 2007 IBM Corporation6 Performance Measurement Methodology 4/13/2007
Sample Design
Representative of real mail flows – Base the design on real data (e.g. ODIS)
Need a defined measurement period and precision level– Monthly, quarterly, annual estimates to +/- X%
What are you interested in measuring?– Usually on-time rate– Could also consider measuring:
• Percent of mail delivered within a delivery window• Percent early, on time and late• Percent delivered within J+1 days, J+2 days…
© 2007 IBM Corporation7 Performance Measurement Methodology 4/13/2007
Provide for continuous measurement – Measure across the period you are interested in – Include all days, weeks, months in the period
Follow live mailing patterns to the extent possible– Across days of the week– Across weeks of the quarter/year
Define the field of study that you want to measure– Performance of a mailing– Performance of letters, cards, flats– Performance of all Standard Mail
Sample Design (cont)
© 2007 IBM Corporation8 Performance Measurement Methodology 4/13/2007
Meet desired stratification and geographic coverage by spreading inductions and receipts accordingly
– Could use available ODIS data:• Origin/destination ZIP Code pairs• Address type (handwritten, machine-addressed) • Barcoded• Shape or weight• Postage type• Day of week mailed/received
– Could also use other characteristics:• Physical appearance (color, size, window)• Presort level• Sender type • Method of acceptance/collection• Receiver type
Sample Design (cont)
© 2007 IBM Corporation9 Performance Measurement Methodology 4/13/2007
Provide for a geographic stratification to ensure coverage across all areas covered by the product
– Some systems place o/d links into sample groups and sample links from the groups to represent the population• Major metro to major metro• Major metro to rural areas of size X• Rural areas to rural areas
– Some systems seek to estimate performance for:• All mail originating from an area with
certain precision • All mail destinating to an area • Origin/destination combination is more complex
Sample Design (cont)
© 2007 IBM Corporation10 Performance Measurement Methodology 4/13/2007
Sample size required is largely determined by:– Number of sample groups to be measured– Desired level of precision for more detailed cuts of the data– Time period for which precision is needed– Variability of the population
Larger sample sizes can improve the precision– A simple random sample size of 1,067 achieves precision of +/-3% – A sample size of 4 times that size achieves +/-1.5%– Sample must represent the field of study well to be accurate
Sample Design (cont)
© 2007 IBM Corporation11 Performance Measurement Methodology 4/13/2007
What else drives sample size?
– Variability of the results • Closer to 50% on time - more sample needed
• Closer to 100% on time - less sample needed
– Number of and correlation between test pieces mailed at the same time • The more pieces are correlated, the less information
you have
– Number of usable pieces out of your sample (after business rules are applied)
Sample Design (cont)
© 2007 IBM Corporation12 Performance Measurement Methodology 4/13/2007
How are test pieces created / chosen– Sampling– Created by mailers in different locations– Centrally by a third party– Will test pieces need to be shipped for induction
Matching test pieces to participants– Seeding receiver names into mailings– Timing and coordination
What addressing standards are needed– Quality of address:
• Avoiding mail forwarding and other delays– What checks are done up front versus later
in the process
Mail Fabrication
© 2007 IBM Corporation13 Performance Measurement Methodology 4/13/2007
How will you manage senders / mailers– Where are they needed based on design
– Recruiting, training, and performance monitoring
What types of mail acceptance are needed – Collection boxes
– Post Office
– DDU, SCF, ADC, AADC, BMC or BMEU
How will the Start-the-Clock data be validated– What will the induction rules be
– When will induction data be removed
Induction
© 2007 IBM Corporation14 Performance Measurement Methodology 4/13/2007
Are receivers needed and how many– What types of delivery addresses are needed
– Recruiting, training, and monitoring
– Handling performance issues
How many pieces can be delivered to the same address: – Without compromising the confidentiality of the study
– Without overburdening receivers
How will the Stop-the-Clock data be validated– What will the rules be
– When will receipt data be removed
Receipt
© 2007 IBM Corporation15 Performance Measurement Methodology 4/13/2007
Processes will be needed to validate the statistical assumptions and results– Design will rely on having data about the characteristics of the population to
ensure it is representative
– Weighting methodology for metrics will need to be validated
– Must develop a methodology for estimation that is agreed to and understood
Business rules about what data are suspicious and how to identify, investigate and decide whether or not to keep data are needed– Pieces missing valid start-the-clock or stop-the-clock information
– Pieces that were forwarded, returned to sender, etc
– A receiver’s data when there is a pattern of “bad” reporting
– Extreme circumstances (e.g. hurricanes, blizzards, blackouts)
Data Analysis
© 2007 IBM Corporation16 Performance Measurement Methodology 4/13/2007
What deliverables will be needed– Reports– Data files
Who are the recipients– USPS– Mailers– PRC / Congress
What level of reporting is needed– National– Mailer-level– By sample group
How frequently should deliverables be released– Preliminary versus final results
Reporting
© 2007 IBM Corporation17 Performance Measurement Methodology 4/13/2007
Credibility of the measurement system depends on quality processes
Must be able to withstand scrutiny:
– Mailer and USPS stakeholders
– GAO
– Congress
Ability to identify and respondto issues
Quality
© 2007 IBM Corporation18 Performance Measurement Methodology 4/13/2007
Summary
There are numerous considerations in external measurement that vary depending on:– Objectives and priorities of your stakeholders
– Which product you are measuring
– What level of detail you require
Quality Assurance
StatisticalDesign
Test MailFabrication
Test MailInduction
Test MailReceipt
DataAnalysis
Reporting