Data Integrity Charlie Appleby, U.S. EPA Region 4 SESD Quality Assurance Section

Data Integrity

Charlie Appleby,

U.S. EPA Region 4

SESD

Quality Assurance Section

Integrity

• “There can be no friendship without confidence [trust], and no confidence [trust] without integrity.” – Samuel Johnson

• “Transparency is the key to trust.” – Steven Hill

• “Real integrity is doing the right thing, knowing that nobody’s going to know whether you did it or not.” – Oprah Winfrey

Data with Integrity

• Data of known and documented quality

• Representative, Comparable and Complete

• Defensible and Usable for its intended purpose, the first time.

– Best Practices for the Detection and Deterrence of Laboratory Fraud, 1997

Data Integrity Requirements

• Careful planning prior to sample collection

• Coordination between stakeholders

• Communication lines

Planning For Success

• DQO Process

• Decisions

• Data Needs

• Data Quality Requirements

Planning For Success

• Take time to plan

• Define the the project boundaries

• Include all stakeholders in the entire process

• Establish lines of communication providing essential information to all personnel involved in the project

Data QualityNo Task for a One-Man Band

• Project Leader• Project QA• Contractor Support• Management• RSCC• Laboratory

• Peer Reviewer• QA Manager

Coordination For Success

• Between– Branches– Divisions– Agencies

Planning for Quality

• Quality Management Plan

• Quality Assurance Project Plan

It Comes Down to People and Communication

Corporate Policies Reflect Values

• Hiring practices - checking references

• Ethics training

• Data integrity training

• Complete technical training

• A quality system

In the future, employees will either be superstars or perspiration wipers. Those who aren’t qualified to do either will become managers. – Scott Adams

Ethics Policy

– Conduct all business with integrity and in an ethical manner

– Responsibility of each staff member and manager to hold to the highest ethical standard of professional conduct in the performance of all duties

Data Integrity Policy

– To ensure work is of highest integrity– Employees responsible and accountable

for the integrity and validity of their own work

– Employees to respect and adhere to the principles of ethical conduct

– Fabrication or falsification of work results are direct assaults on the integrity of the laboratory and will not be tolerated

Documenting the Quality System,The QA Manual

• The Corporate Mission, Values & Vision• Organizational structure and

responsibilities• Procedures for documenting lab ops

– Sample receipt– Stds/reagent prep– Completing Training

• Document control• Corrective action

The QA Manual (continued)

• Data verification, approval, and reporting

• Facility/data security• Emergency procedures• Corrective action policies and

procedures• Index of SOPs• Reports to management

Cracks in the Quality System

• Entropy – Newton?

Cracks in the Quality System

• Lax documentation in sample receiving,

• Poor hiring decisions,

• Failure to complete or document training,

• Lack of cross-training,

• Missed SOP updates, internal audits,

Cracks in the Quality System

• Lax peer review,

• Poor document control,

• Poor housekeeping,

• Turnover,

• Drop in data quality

Whither the Quality SystemVulnerabilities

• Inappropriate practices

• Lost business/revenue

• Excessive turnover

• Fraudulent activities

What is Laboratory Fraud?

• Intentional misrepresentation of lab data to hide known or potential problems

• Make data look better than it really is

Dr. Bruce Woods

Potential Areas of Lab Fraud

• Procedural Deceptions:– Not following critical steps of

methodology– Short-cutting sample prep,

calibration, analysis

• Measurement Deceptions:– Directly altering results– Time and date, conditions of

experiment

Examples of Procedural Fraud

• Leaving out hydrolysis step in herbicide analysis to avoid hassle.

• Not prepping the PE sample before analysis (direct injection).

• Not digesting metal samples for analysis due to heavy workload

More Examples

• Selectively background subtracting spectra from other peaks to make tuning criteria pass in GC/MS analysis.

• Using calibration procedures that are not allowed by the required method.

Examples of Measurement Fraud

• Deceptive GC Peak Integration

• Time Travel

• Dry Labbing

Preventing QA System PitfallsWhat can the lab do?

• Independent QA Officer,

• Ethics Policy,

• Internal audits,

• Certifications,

• Managers who keep the vision fresh

Detecting QA System ProblemsWhat can EPA do?

• Independent data validation,

• Monitoring PT sample performance,

• Data tape audits,

• On-site laboratory audits– Announced– Unannounced

• Follow-up audits

Preventing QA System Weaknesses

• Contract language,– Clear QA/QC requirements,– Incentives / Disincentives

• Pre-award audits,

• Past performance assessment,

• Performance Testing

Case Studies

Let’s Test your Knowledge

Challenges for EPA

Why do we need a vision for data integrity?

“Though leaders in the middle may not always be the inventors of the vision, they are almost always its interpreters.” – John C. Maxwell

We are what we repeatedly do. Excellence then is not an act, it is a habit. – Aristotle

Vision Statement Creation

• First, identify the mission– Protect Human Health and the Environment

• Identify values– science-based policies and programs – adherence to the rule of law – overwhelming transparency

• Distill and refine

Elements of vision

• Clarity• Connection of Past, Present, and

Future• Purpose• Goals• Challenge• Stories• Passion

Vision Exercise

• Imagine the ideal state• Identify needed

skills/competencies• Evaluate strategy• Clarify the forces you will face• Be realistic

Data Integrity Starter Quiz

Are you DI Savvy?

Questions borrowed from SHOQ Quality Assurance Manuals Inc. ISO 17025 Culture Quiz

Management and technical personnel should have the

necessary:

A. PersonnelB. Authority and resourcesC. Instrumentation

The laboratory’s quality system policies and objectives should

be defined in a:

A. Quality Policy StatementB. Quality ManualC. Standard Operating Procedure

Document control means:

A. Ways to reduce paperB. Documents are identified, authorized, reviewedC. Give all documentation to supervisor

A laboratory is not responsible to the client for the work of a

subcontractor

A. TrueB. False

Records should be maintained of all client complaints and of:

A. How angry the client wasB. The investigations and corrective actions taken by the lab C. How loud the client complainedD. To CYA in court

The procedure for corrective action must start with:

A. Finger PointingB. Risk Assessment C. An investigation to determine the root cause(s) of the problemD. A judicious application of CYA

Controlled records should be:

A. Legible, readily retrievable, and in a suitable environmentB. Designed for auditorsC. Controlled by IT personnel

Internal audits are conducted to verify:

A. Compliance of operations with quality system B. We won’t get caught again!C. The cost vs benefit of each test offeredD. Compliance of operations with EPA requirements

Management reviews determine:

A. Continuing suitability and effectiveness of the quality systemB. That there will always be another Dilbert cartoonC. Employee requirements are met through 365 degree feedback

Training records are essential to:

A. Writing Job Descriptions B. Accrediting the analystC. Ensure competence and authorize personnel

Methods must be sufficiently validated as well as:

A. Maximize profits

B. Meet the needs of the client and appropriate for the testC. Easily implemented by the lab

Primary measurement standards must be traceable by means of an unbroken chain of

calibrations or comparisons linking them to:

A. The last standard entered in the log

B. NTISC. Check samples

Sampling generally happens prior to reception at a lab, and

therefore has little affect on final results:

A. True

B. False

Three levels of data/peer review are necessary to:

A. Keep the analysts feeling insecure

B. Give the manager something to doC. Ensure the data are accurate, defensible, and meet the clients’ needs

Case Studies

Let’s Test your Knowledge

Case Study 1• An analyst is preparing a method blank associated

with a batch of samples which will be digested for metals determinations. The analyst selects a specific beaker which he/she always uses to digest the blank because it seems to produce non-detect or very low results . This practice is:

A. Perfectly acceptableB. A deceptive lab practiceC. An improper lab practiceD. None of the aboveE. Both B & C

Case Study 2• An analyst discovers that a water sample has

accidentally been left out of a sample batch which was extracted for pesticides analysis. He/she included the sample in a different extraction batch which was prepared eight days later. He/she records the sample extraction date as the date of the first batch because the sample holding time was exceeded when the second batch was extracted. And besides, the pesticides supervisor does not like to qualify data for missed HTs. This practice is:

A. A good reason for a time off awardB. An improper lab practice C. An example of good sample managementD. Lab fraudE. All of the above

Case Study 3• An analyst who performs total suspended solids

analyses remembers on his/her way home that the balance calibration check for that day’s sample batch was forgotten. He/she performs the balance calibration as soon as he/she arrives at work the next day. The 100g, 1000 mg and 500mg weights are all within acceptable tolerances. The analysts is unsure how to proceed. He/she should:

A. Record the calibration check and not mention it to anyoneB. Repeat the analyses on the entire batchC. Request that the TSS secondary analyst repeat the calibration

check to confirm it was done properly.D. Ask his/her co-worker for advice

Case Study 4

• The TSS analyst asked a co-worker what to do about the balance calibration check. The co-worker says “No harm, no foul” and advises him/her to just record the calibration results and keep quiet. The TSS analyst should:

A. Follow the co-workers advice B. Turn the co-worker in to the EPA IG.C. Tell his/her supervisor that the co-worker is unethical.D. Go to upper management and spill everythingE. Seek counsel from the lab supervisor and QA manager.

Case Study 5• Near the end of the day, a VOA analyst notices all

surrogate standard recoveries are running about 40% low for the day’s batch and are outside control limits. The analyst does not have time to prepare a new surrogate standard solution before the shift ends, but at his/her previous job he/she was taught that it was OK to increase the volume of surrogate standard in such instances. The analyst increases the volume of surrogate injected from the normal 1.0 uL to 1.4 uL and sets up a run to reanalyze the batch during the night. This is:

A. A good way to make sure the QC is within limits. B. An improper lab practice.C. Lab fraud.D. A failure to follow the SOP.E. Difficult to assess without more information.

Case Study 6• During the preparation of soil SV samples, the

method blank was lost due to an instrument malfunction during cleanup. It has been 48 hours since the batch was extracted. The correct thing to do is:

A. Re-extract the entire batch of samples.

B. Prepare and extract another method blank only.

C. Go ahead and analyze the batch w/o a MB and flag the data.

D. Perform maintenance on the clean-up instrument so the next batch will be OK.

E. None of the above

Correcto!!Click here to return

Sorry that is incorrect!

Click here to return