AME Quality Assurance Performance - civilavmed.org · AME Quality Assurance Performance Stephen Véronneau, MD ... Aviation Medical Examiner Program, ... NUMBER OF EXAMS FOR RANGES

Federal AviationAdministrationCAMA Annual Meeting

AMEQuality Assurance

Performance

Stephen Véronneau, MD

Manager, Aerospace Medical Education Division

Regional Flight Surgeon

International, Military, and Federal Region

2Federal AviationAdministration

AME Performance Reporting

• Leah Olson, managing [email protected]

• Frances Parker, secretary, [email protected]





Objectives

• Policies/Orders• Why Monitoring is Needed• Methods of Surveillance• AME Performance Reports• Renewal, Termination, Appeal of Termination





Requirement by FAA Order

• VS 1100.2, Managing AVS DelegationPrograms

• FAA Order 8520.2G, Aviation MedicalExaminer Program, was subsumed into8000.95 CHG 2 in 2015



U.S. DEPARTMENT OF TRANSPORTATIONFEDERAL AVIATION ADMINISTRATION

AVS PolicyORDER

VS 1100.2

Managing Designees

•Selection

•Appointment

•Oversight

•Training

•Renewal

•Termination

•Appeal



Aerospace Medical EducationDivision Manager

AME Responsibilities:• Plan, develop, administer and evaluate

training programs• Monitor AME performance• Provide information to other Regional Flight

Surgeons



Why Performance Monitoring isNeeded



Why Monitoring is NeededThe Issued Exam QA Study

• All 323,304 records issued by AMEs in CY2005 met selection criteria

• A random sample of 2000 MID numberswere selected by stratified random sampling

• 2000 records, each one reviewed by 2independent reviewers

• Average review time 10 to 15 minutes/case• 4 months of review



Did AME issue appropriately?

• Population correlation ifrates are applied to the323,304 airman records sentto file– Issued appropriately 309,402– Issued inappropriately 8083– Incomplete 5819

95.7%

2.5%

1.80%

0.0%

20.0%

40.0%

60.0%

80.0%

100.0%

Total 2000 Exams

YesNoIncomplete

Did AME issued airman medicalcertificate appropriately in CY 2005?

Subtotal 1914 50 36



Aviation Medical Examiner QA Program

• Initiated July 2007, FAA Order VS 1100.2Managing AVS Delegation Programs

• Based upon procedures and softwaredeveloped during a 2006 QA Study

• Additional means of AME performanceoversight (Rv, Hx, C letters, SPA, site visit)



2008 Issued Reject Results

81.8%

18.2%

97.7%

2.3% 0.3%

99.7%

0.0%

20.0%

40.0%

60.0%

80.0%

100.0%

commentappropriately

issueappropriately

possiblefalsification

yes no

3405

1763

44

18911527



AME Comment Appropriate block 60



AME Appropriate Issue



Most common causes for problems with issuedcertificates Jan. to June 2012 compared to prior period

Block Description OccurrencesJuly–Dec 2011

OccurrencesJan–Jun 2012

17 Unacceptable Medication 12 3

58 ECG 9

18h High or low Blood Pressure 9 4

18V DUI

18a Headache 1

18M Mental/Depression

18U Rheumatoid Arthritis 1

18 L Neurological disorder, Parkinsonism 1

18J Kidney stone

18u Admission to Hospital

18x Prostate cancer surgery,parathyroidectomy, cataract surgery

18D Retinopathy /Optic Neuritis 1

18X Sleep Apnea 1

52 Fail color vision 1

•Please note that there can more than one error in a single case



Most common causes for problems with issuedcertificates Jul to Dec 2014 compared to prior period

Block Description OccurrencesJuly-Dec 2014

Past OccurrencesJan-Jun 2014

17 Unacceptable Medication 16 4

58 ECG 0 0

18h High or Low Blood Pressure 40 75

18u Laryngeal cancer 1

18m Mental disorder 1 1

18f Asthma 4 5

18f Sarcoidosis 1

18u, x Rheum. Arthritis, Thyroiditis 1

18k Diabetes Mellitus

18d Vision trouble

19 Sleep apnea 1



Trends 2007 to 2014



AME Error Summary Jul to Dec 2014 compared toprior period Jan to June 2014

• AME error rate of the sample was 6.2% versus 9.1%

• Incomplete evaluation of applicants with HTN (CACI) is

the most common problem

• AMEs issuing certificates with inappropriate medications

• No falsifications 0 of 893; 0 of 898 records examined

• CACI related errors are 90% of total observed





FAA Aerospace MedicalCertification Services

Airman CustomerSatisfaction Survey

2012 and 2014



Survey AdministrationRandom sample of airmen who applied for pilotmedical certificates in 2012 and 2014

2012 2014Sampling frame 10,439 10,299Undeliverable 158 81

10281 10218

Returned and metcriteria for inclusion 4,256 3703

Percent returned andmet criteria for inclusion 41% 36%



Who reviewed your medical history withyou?

17%10%

85.40%

1% 4%5%

80%

18%

0%

25%

50%

75%

100%

AME Non-AMEPhysician

Non-Physician No Review

2012 2014



Who performed your physical exam?94.7 96.4

11.2 8.26.1 4.43.1 0.71.6 1.51 0.50

102030405060708090

100

2012 2014

AME

Nurse

PA

Non-AMEPhysicianOther

no one



Did your AME perform a thoroughexam?

95% 93%

5% 7%

0%10%20%30%40%50%60%70%80%90%

100%

2012 2014

YesNo



Did your AME



FAA Aerospace MedicalCertification Services

AME CustomerSatisfaction Survey

2012 and 2014



Regions Represented

0

5

10

15

20

25

30

AK CE EA GL NE NM SO SW WP IMF

RESPONDENTS (N=1739) ALL AMEs (N=3231)



Demographics• AME respondents: n=1,613• Average Yrs. worked: 19• Senior AMEs: 71%• AME-related practice: 5%• Applicants seen (median): 50



Overall Approval 2014

SERVICES %Deferrals Standards and Guidelines 72Aerospace Medical Certification Division 92Aerospace Medical Education Division 93Aerospace Medical Certification System 95Regional Flight Surgeon Office 92MedXPress 84Office of Aerospace Medicine Website 88Digital ECG System 80



Recommendations for Improvement

•AMEs suggested improvement for FAAservices by ranking their top 3

recommendations from multiple choiceoptions.



How Can FAA Improve Training?

58.0

45.0

22.0

0 10 20 30 40 50 60 70 80

CACI Theme seminar

More Regional Seminars

Live Video seminars at RFSOffice



How can FAA Improve AMCD? 2014

66.0

65.0

61.0

0 10 20 30 40 50 60 70 80

Attach supporting documents inAMCS

Attach supporting documents inMedXPress

Electronically check status ofdeferral



How can FAA Improve RFS? 2014

• Live video seminars at RFS office (21%)



63.0

61.0

58.0

0 10 20 30 40 50 60 70 80

Electronic notification of changesin deferral status

Feedback on inappropriatedeferrals

Online tracking of deferrals

How can FAA Improve Deferral Process? 2014



How can FAA improve OAM Website?2014

58.0

48.0

43.0

0 10 20 30 40 50 60 70 80

Easy access to CACIworksheets

Online spec sheets

Consistency between NavAidsand AME online publications

PDF







VIRTUAL SITE VISITS



SITE VISITS



Judgment Errors



Error LettersDefinitions

• Critical Error (“Rv” Error)– AME issued certificate when should have deferred– AME deferred when should have issued– Letter Sent to AME and RFS

• Less Critical Error (“Hx” Error)– Not enough information to support AME issuing– “Tic Mark”, Not Letter– Part of annual AME performance report



Error LettersDefinitions

• CACI Errors (“C” Error)– AME issues but deviates from the CACI worksheet– AME sent in CACI worksheet and supporting

information (Guide states do not send). Note:please remember that this only applies to the CACIcondition(s) there might be other information thatdoes need to be submitted for AASI

– AME deferred when should have issued usingCACI

– AME does not use the correct verbiage in Block 60





www.faa.gov/GO/AME/



http://www.faa.gov/other_visit/aviation_industry/designees_delegations/designee_types/ame/dms/



https://designee.faa.gov/



Individual AME Performance Summary Report

Report Date: 9/1/2016

THIS IS A REPORT LISTING YOUR PERFORMANCE FOR THE PERIOD 1/1/2015 THROUGH 9/1/2016. IF YOU HAVE QUESTIONSABOUT THIS REPORT, CONTACT YOUR REGIONAL FLIGHT SURGEON. MILITARY, FEDERAL, AND INTERNATIONALEXAMINERS SHOULD CONTACT THE MANAGER, AEROSPACE MEDICAL EDUCATION DIVISION, AAM-400, P.O. BOX 25082,OKLAHOMA CITY, OKLAHOMA, 73125.

Name and Address

REGION: INTERNATIONALREGION

STATE: UNKNOWN COUNTY: UNKNOWN

AME # : 00000xxxx STATUS: ACT SENIORAME:

Y APPOINTMENTDATE:

01/01/1994

1. INFORMATION REGARDING TRAINING AND DELINQUENCY:If "NEXT TRAINING DUE DATE" is the same as "NEXT SEMINAR DUE DATE" then there is no option fordistance learning. If these two dates differ, then there is a choice of MAMERC or SEMINAR training to becompleted prior to "NEXT TRAINING DUE DATE".

Next Training Due Date: 01/31/2019 Next Seminar Due Date: 01/31/2022 Years Delinquent:

Last Class TrainingDate:

01/31/2016 Last Self Training Date: 02/17/2014 (16.94)

2.TRANSMISSION DELAYS: NUMBER OF EXAMS FOR RANGES OF DELAY

Total ExamsTransmitted:

Under 15 Days 15-30 Days 31-60 Days 60+ Days %Over 60 Days

145 140 3 1 1 3.6

NOTE: Delayed exams may have been performed earlier than this report period, so total numbers in para4 may differ

3. EXAMINATION INFORMATION (BASED ON EXAMINATION TYPE REQUESTED BY THE AIRMAN)

1ST 2ND 3RD Total Exams Deferred Denied % Deferred/Denied

55 54 36 145 4 0 18.1

4. INFORMATION REGARDING EXAMS WITH 'Rv' AND 'Hx' ERRORSNOTE: An Rv error is assigned when an AME issued a certificate when it should have been deferred ordenied, requiring reversal by AMCD. An Hx error is assigned when the medical history was judged to beinadequate.

2.TRANSMISSION DELAYS: NUMBER OF EXAMS FOR RANGES OF DELAY

Total Exams Transmitted: Under 15 Days 15-30 Days 31-60 Days 60+ Days %Over 60 Days

145 140 3 1 1 3.6

NOTE: Delayed exams may have been performed earlier than this report period, so total numbers in para 4 maydiffer

3. EXAMINATION INFORMATION (BASED ON EXAMINATION TYPE REQUESTED BY THE AIRMAN)

1ST 2ND 3RD Total Exams Deferred Denied % Deferred/Denied

55 54 36 145 4 0 18.1

4. INFORMATION REGARDING EXAMS WITH 'Rv' AND 'Hx' ERRORSNOTE: An Rv error is assigned when an AME issued a certificate when it should have been deferred ordenied, requiring reversal by AMCD. An Hx error is assigned when the medical history was judged to beinadequate.

MID Applicant Name Exam Date Error Code Code Description

200007267448 name 04/12/2016 Other Other

200007118275 name 12/22/2015 Rv REVERSAL

200006969115 name 07/20/2015 Hx INADEQUATE HISTORY

200006809733 name 03/16/2015 Other Other



AME records must be retained by the Examiner as their file copy for atleast 3 years unless exceeded by state law requirements …. (scannedcopies are equivalent to file copies)

All completed applications and medical examinations, unless directed bythe FAA, must be transmitted electronically via AMCS within 14 days aftercompletion to the AMCD.



Student Pilot

• No longer issued by the AME as of April 1,2016



Student Pilot no longer an AME Activity

EXAM Certificate



Examination Submission DelaysConcerns

• No Record in System to Support IssuedCertificate– Hassle With Ramp Checks– Could Result in Assessment of Flying Without

License If Crash and Burn Accident



Submission Delay > 60 Days

0.2

0.5

0.4

0.2 0.2

0.5

0.1 0.1

0.2

0.10.1

0.1 0.1 0.1

0.1 0.1

0.4

0.10.1 0.1 0.1 0.1

0.1

0.10.1

0.10.10.1

0.0

0.1

0.2

0.3

0.4

0.5

0.63/

31/1

2

6/30

/12

9/30

/12

12/3

1/12

3/31

/13

6/30

/13

9/30

/13

12/3

1/13

3/31

/14

6/30

/14

9/30

/14

12/3

1/14

3/31

/15

6/30

/15

Per

cent

All Domestic Only







Training IS Required Every 3 Years-An AME must attend an AME Seminar, or equivalent training asdetermined by the AMED Manager, every 3 years, as a requirementfor continued designation.

- As an option, an AME may alternate Multimedia Aviation MedicalExaminer Refresher Course (MAMERC) in lieu of attending an FAAseminar, but under no circumstances should more than 6 years (72months) elapse between AME seminar attendance, or more than 3years (36 months) pass between seminar attendance and MAMERCcompletion.

Training Requirements



AME Performance ReportingDeferral Rates

4.7 4.54.84.84.54.44.54.44.44.24.14.14.03.63.63.93.83.63.8

4.85.0

4.7

0

1

2

3

4

5

6

3/31

/201

0

6/30

/201

0

9/30

/201

0

12/3

1/20

10

3/31

/201

1

6/30

/201

1

9/30

/201

1

12/3

1/20

11

3/31

/201

2

6/30

/201

2

9/30

/201

2

12/3

1/20

12

3/31

/201

3

6/30

/201

3

9/30

/201

3

12/3

1/20

13

3/31

/201

4

6/30

/201

4

9/30

/201

4

12/3

1/20

14

3/31

/201

5

6/30

/201

5

Per

cent



Renewal of Designation



Designee Management PolicyOrder 8000.95 CHG 2

Published 6 March 2016– Duration of designation 3 years– 10 exams per year to maintain proficiency– NO exams on immediate family– No appeal for loss of credential, failure to train or

lack of FAA need for your designation– All FAA exams must be conducted in English– AMCS changes

• Timeout• Confirmation of staff q 90 days



Termination of Designation



Overview of Designee Lifecycle• Suspension and Termination

– Input: Trigger for suspension and termination are identified inFAA Order 8000.95

– Suspension for not meeting current requirements• The designee is afforded an opportunity to correct the issue within a

specific time frame• If not corrected, the designee may be terminated

– Termination For Cause:• “Bad” termination• Can be appealed by designee, follows Appeal process• Designee cannot re-apply

– Termination Not For Cause:• “Not bad” termination• Cannot be appealed• Designee CAN re-apply

– Output: Suspended or Terminated designee

61



Reasons for termination• No exams in first 12 months• Less than 10 exams per year• Not following FAA rules• Poor reporting of medical examinations• Failure to train• Practice in unapproved location• Unprofessional office• Unprofessional performance of exams• Failure to mail in exams



Reasons for termination• Loss, restriction, or limitation of a license to

practice medicine• Any action that compromises public trust or

interferes with the AME’s ability to carry outdesignated responsibilities

• Arrest, indictment or conviction• Any other reason the Federal Air Surgeon or

RFS deems appropriate• Exams not conducted in English



Risk

Severity of an outcome x likelihood ofoccurrence





Voluntary Separations2012 2013 2014

Death 9 8 16

Resigned 105 95 86

Retired 38 48 65



Voluntary Separations2013 2014 2015

Death 8 16 1

Resigned 95 86 25

Retired 48 65 12



Involuntary Separations

2012 2013 2014

Failure to Train 75 112 136Low Volume 72 25 32Medical Incapacitation 0 2 6Performance 4 2 6Unable to Locate 9 13 19Unapproved OfficeLocation

4 8 2

Transmission Delays 3 0 0Failed Site Visit 1 1 1



Involuntary Separations

2013 2014 2015

Failure to Train 112 136 57Low Volume 25 32 2Medical Incapacitation 2 6 1Performance 2 6 1Unable to Locate 13 19 1Unapproved OfficeLocation

8 2 1

Transmission Delays 0 0 0Failed Site Visit 1 1 2



Appeal of Termination



U.S. DEPARTMENT OF TRANSPORTATIONFEDERAL AVIATION ADMINISTRATION

AVS PolicyORDER

VS 1100.2

Provides for an appeal panel ofthree physicians with a decisionreturned within 45 days.



Can Not Appeal . . .

• Lack of need• Separation for failure to train



New Designee Management System

• November 2015 AAM went into DMS• Electronic

– Application– Renewal– Record Keeping



New Designee Management System

• Electronic– Application– Renewal– Record Keeping– Interaction with Managing Specialist (Regional staff)– AME performance report



New in DMS for AMEs• AMEs will be able to view their information

on training and performance• AMEs will be able to submit updates to their

information through DMS• AMEs will receive and be able to respond to

corrective actions through DMS whereneeded

• AMEs will request renewal of theirdesignation through DMS



•https://www.faa.gov/tv/?mediaId=1180



Designee Management System

•https://designee.faa.gov



Designee Registration System



New in DMS for AMEs• AMEs will be able to view their information

on training and performance• AMEs will be able to submit updates to their

information through DMS• AMEs will receive and be able to respond to

corrective actions through DMS whereneeded

• AMEs will request renewal of theirdesignation through DMS



Documents

AME Quality Assurance Performance - civilavmed.org · AME Quality Assurance Performance Stephen Véronneau, MD ... Aviation Medical Examiner Program, ... NUMBER OF EXAMS FOR RANGES