Response to a Self-IdentifiedData Quality Issue

Presented by Steve Vanderboom and

Bruce Warden

Bad News Arrives• Sr. Quality Officer:

“Do you have a few minutes to discuss an issue?”• Charlotte, NC lab data quality issue• Identified by Pace Quality Manager • Improper manual integration• Semi – volatile organics lab• Two analysts interviewed and fired• Investigating scope (2 or 3 months)

North Carolina Laboratory

• Around 65 employees• Market leader• Strong performer• Fast and reliable service• Quality reputation• No prior problems• Good profitability and growth

Charlotte Laboratory

The Risk• Fired semi-voa staff, lost capability• Outsourced work to Pace labs / others• Service slowed• Quality issue must be corrected• Pace must:

– Define scope and customer impact– Communicate with customers and regulators proactively

• Cannot communicate before problem defined• Cannot allow a perception of a cover up

Defining Scope and Communications

• Traced back 18 months to find beginning• 12,000 samples• Close to a million data points• Over 300 customers• Six states• Seven outside audits plus Pace Corp and

local audits did not catch the problem

Communication• No communication until we understood situation• Need to plan communication to:

– Employees– Regulators– Customers– Competitors

• Risk of early communication significant– Misperception by regulators– Confusion and anger by customers

• “There’s a problem, but I can’t tell you the impact.”

External Communication Customer Communications Team• Information Package for Each Customer

– Letter for customer– Data impact summary per customer– Summary from CEO

• Initial Communication Strategy – Contact customers within 30 days– Contact agencies concurrently– CEO to inform Laboratory community

• Focused on competitors of Charlotte lab

• Ongoing Communication Strategy– Customer Response Team at Charlotte lab to address

continued concerns– Team access to Target & LIMS server & databases

Timing• Define problem and impact

– Sept through January– 17 analysts - ~ 2 man-years– Bruce will review process

• Communicate:1. Internal staff – Mon, Feb 12. Regulators – Mon/Tues, Feb 1/23. Customers – Tues Feb 2 (three weeks)

• Prioritized by size with officers visiting first week4. Competitors

• Share with outside – place on table for them to pick up after

Importance of Documentation• Risk of perceived cover up

– (4+) months to assess problem and produce summary reports for each customer

– Documented: • Internal plans for reporting and backing up data• Impact on each customer sample• Communication plans• Corrective actions

Investigation Process

• Initial Discovery • Scope of impact • IT systems set up • Data Review • Oversight and direction of the

investigation• Communications during investigation• Documentation of investigation

Initial Discovery• Manual integration problems• Verified behavior • Questioned employees involved• Terminated employees

What did we see?• SVOA GCMS analyses

– Total Ion Chromatogram with 75-170 peaks

• Manipulated Manual Integrations– Peak area shaving– Peak area enhancement

Chromatogram for 8270 Semivolatile Analyses

Peak Shaving Through Baseline Elevation

• Improperly raised baseline to remove area. – Excludes large area of

original peak.– Distorts the normal shape.

Peak Shaving- Removing Area

• Improperly integrated to remove area. – Excludes area of original

peak.– Peak shape not preserved.

Gross Peak Shaving

• Improperly integrated to meet QC limits.– Desired area obtained by

manipulating integration.– Includes peak shaving and

elevating the baseline.

Improper Manual Integrations

Lab Follow-up Activity after Discovery

• Determine what GCMS-SVOA work was in house and was reportable

• Sub-out all samples requiring these analyses to accredited labs

• Perform root cause analyses• Carry out corrective actions identified

– Instruments optimized and qualified– New analysts hired and trained– Review process strengthened

Scope of Impact

• Corporate Quality involved• Reviewed SVOA GCMS manual

integrations• Determined timeframe

– 5 calendar days/instrument/month• Query of LIMS to ID all

customers/projects/samples requesting SVOA GCMS analyses

Scope of Impact (cont.)

• Additional testing at the Charlotte laboratory– Random manual integration review

for other organics methods• Other Pace laboratories

– Mintminer: identified Target files containing manual integrations

– Reviewed manual integrations for compliance

IT Systems to Facilitate Investigation Process

• Corp server set up to house data involved– Copied data from Target and LIMS

databases for time period involved– Backed up original data & re-processed data

• Setup 10 computers for remote data processing

• Integrated data evaluation with customer information

Data Review Process• Review Team formed

– Guidance document provided on review process

– Training for all team members– Assignments given by Corp Quality Manager

• Communicated Q&A through shared email folder

• Daily meetings to ensure consistent interpretation and resolve issues

• Data review by primary and secondary analyst(s)

• Revised results compared to original

Oversight of Investigation

• Oversight Team formed – Corporate QA, Corporate officers, and sr. GM’s– Reviewed information regularly– Provided direction; short & long term goals.– Ensured resources were available– Decided on when enough information was

available to define impact and inform customers – Strategized internal & external communication

• Review of Decisions made

Internal Communication Limited internal communication during the investigation:• Local staff informed as issues confirmed• Corporate QA coordinated team member

communication• Review Team members kept communication

within team• Review Team & Oversight Team communicated

regularly– Summary provided ongoing– Confidentiality maintained

• Memo to All Pace Staff from CEO before external communication

Documentation of Investigation

• Databases backed up at Corporate– Target database: original and revised – LIMS database: before/after results, footnotes, etc.– Access database: relating ICAL & CCV QC criteria

to customers/samples/analytes• Customer call log maintained at Charlotte lab• Customer correspondences filed at Charlotte lab

– Electronic and hardcopy

Final Outcome

• Corrective Actions company wide– Retraining on manual integration policy for all– 3 or more analysts in secondary review pool– Require before/after pictures of manual integrations– Mintminer scans and review performed qrtly– Raw data audits of each method per quarter– Corporate audits to include manual integration review

• Conclusion: “Should not happen again…but it could.”

Summary

• Carolina alone would not have been able to survive this:

– Cost– Resource

Overall Impact

• Direct and indirect costs ~ $1,000,000– Investigation time– Re-analysis of existing data– Outsourcing – Database setup and management– Loss of business due to slow service– Re-staffing and training– Re-sampling and analysis– Compensating customers for time

Outcome

• Regulator confidence increased• Customer confidence stable or increased• Preserved all major customer relationships• Increased confidence of NC team• Improved quality of operation• Preserved a valuable business and our

reputation• Never want this to happen again!