Full Terms & Conditions of access and use can be found athttps://www.tandfonline.com/action/journalInformation?journalCode=imte20

Medical Teacher

ISSN: 0142-159X (Print) 1466-187X (Online) Journal homepage: https://www.tandfonline.com/loi/imte20

A general practice workplace-based assessmentinstrument: Content and construct validity

Alison Fielding, Katie Mulquiney, Rosa Canalese, Amanda Tapley, ElizabethHolliday, Jean Ball, Linda Klein & Parker Magin

To cite this article: Alison Fielding, Katie Mulquiney, Rosa Canalese, Amanda Tapley,Elizabeth Holliday, Jean Ball, Linda Klein & Parker Magin (2019): A general practice workplace-based assessment instrument: Content and construct validity, Medical Teacher, DOI:10.1080/0142159X.2019.1670336

To link to this article: https://doi.org/10.1080/0142159X.2019.1670336

View supplementary material

Published online: 09 Oct 2019.

Submit your article to this journal

View related articles

View Crossmark data

A general practice workplace-based assessment instrument: Content andconstruct validity

Alison Fieldinga,b , Katie Mulquineya,b, Rosa Canalesec, Amanda Tapleya,b, Elizabeth Hollidayb, Jean Balld,Linda Kleina,b and Parker Magina,b

aGP Synergy NSW and ACT Research and Evaluation Unit, Mayfield West, Australia; bSchool of Medicine and Public Health, University ofNewcastle, Callaghan, Australia; cAvant Mutual, Sydney, Australia; dClinical Research Design IT and Statistical Support, Hunter MedicalResearch Institute, New Lambton, Australia

ABSTRACTIntroduction: Relatively few general practice (GP) workplace-based assessment instruments have been psychometricallyevaluated. This study aims to establish the content validity and internal consistency of the General Practice RegistrarCompetency Assessment Grid (GPR-CAG).Methods: The GPR-CAG was constructed as a formative assessment instrument for Australian GP registrars (trainees). GPR-CAG items were determined by an iterative literature review, expert opinion and pilot-testing process. Validation data werecollected, between 2014 and 2016, during routine clinical teaching visits within registrars’ first two general practice trainingterms (GPT1 and GPT2) for registrars across New South Wales and the Australian Capital Territory. Factor analysis and expertconsensus were used to refine items and establish GPR-CAG’s internal structure. GPT1 and GPT2 competencies were ana-lysed separately.Results: Data of 555 registrars undertaking GPT1 and 537 registrars undertaking GPT2 were included in analyses. A four-fac-tor, 16-item solution was identified for GPT1 competencies (Cronbach’s alpha range: 0.71–0.83) and a seven-factor 27-itemsolution for GPT2 competencies (Cronbach’s alpha: 0.63–0.84). The emergent factor structures were clinically characterisableand resonant with existing medical education competency frameworks.Discussion: This study establishes initial evidence for the content validity and internal consistency of GPR-CAG. GPR-CAGappears to have utility as a formative GP training WBA instrument.

Introduction

The development of learner competence is a cornerstone ofmedical education, representing a core determinant of fit-ness for practice (Wass et al. 2001). Clinical and professionalcompetence can be characterised as practicing ‘the habitualand judicious use of communication, knowledge, technicalskills, clinical reasoning, emotions, values, and reflection indaily practice for the benefit of the individuals and com-munities being served’ (Epstein and Hundert 2002).

Assessment of competence in medical educationencompasses both summative and formative modalities(Wass et al. 2001). Summative assessment provides assur-ance that a medical practitioner meets requirements forsafe practice (Holmboe et al. 2010). Formative assessmentprovides a basis for developing clinicians’ skills and attrib-utes for competent practice and lifelong learning (Wasset al. 2001; Epstein 2007; Holmboe et al. 2010). Thisincludes identifying areas requiring further development(Holmboe et al. 2010), and facilitating early identification oftrainees not performing to expectations (Holmboe et al.2010). Formative assessment is essential in implementationof competency-based medical education (CBME), anincreasing focus for medical education programs (Holmboeet al. 2010; Iobst et al. 2010).

Formative assessment of clinical competence is complexand layered (Wass et al. 2001; Epstein 2007). Miller’s

pyramid of competence provides a theoretical underpin-ning of competence assessment encompassing knowledge(knows), competence (knows how), performance (showshow), and action (does) (Miller 1990; Wass et al. 2001).Ideally, each level of the pyramid should be assessed usingmultiple modalities to achieve a holistic viewpoint of over-all competence (Wass et al. 2001; Epstein 2007; Holmboeet al. 2010).

The apex of Miller’s pyramid, the ‘does’ level, involvesintegration and application of the foundation levels of thepyramid (Miller 1990; Wass et al. 2001). Its assessment

Practice points� Within medical education, workplace-based

assessment (WBA) tools that are valid, reliable,useful, cost-effective, and feasible are essential forthe provision of effective feedback for traineereflection and learning.

� The General Practice Registrar CompetencyAssessment Grid (GPR-CAG) shows promise as aformative WBA method for general prac-tice trainees.

� Results of this study provide content and con-struct validity evidence for the GPR-CAG.

CONTACT Alison Fielding alison.a.fielding@newcastle.edu.au GP Synergy NSW and ACT Research and Evaluation Unit, 20 McIntosh Drive, MayfieldWest, NSW 2304, Australia

Supplemental data for this article can be accessed here.

� 2019 Informa UK Limited, trading as Taylor & Francis Group

MEDICAL TEACHERhttps://doi.org/10.1080/0142159X.2019.1670336

typically requires direct workplace observation (Wass et al.2001; Kogan and Holmboe 2013), evaluating performancein the context of actual clinical practice (Kogan et al. 2009;Miller and Archer 2010; Crossley and Jolly 2012; Kogan andHolmboe 2013; van der Vleuten and Verhoeven 2013).Workplace-based assessment (WBA) utilising direct observa-tion of performance is central to formative competenceassessment (Epstein 2007; Kogan et al. 2009, 2017).

Valid and reliable WBA instruments are essential intrainee doctors’ learning and skills development (Koganand Holmboe 2013). Formative WBA is particularly import-ant for vocational general practice (GP) training in Australiaand in other nations with practice-based apprenticeship-like models of learning (Dick et al. 2007; Hays and Morgan2011; Thompson et al. 2011). Relatively little research, how-ever, has explored the construct validity of direct observa-tion instruments (Kogan et al. 2009; Pelgrim et al. 2011;Wetzel 2012; Sandilands and Zumbo 2014). Construct valid-ity encompasses evidence for instrument content, internalstructure, response process, relationships with other varia-bles and consequences/outcomes (Wetzel 2012; Cook et al.2014). Establishing content validity and internal structureare important first steps in the psychometric evaluation ofWBA instruments (Wetzel 2012).

Most existing validity evidence for WBA instruments hasfocused on medical students, or internal medicine residentsand/or fellows, with relatively little attention given to thevocational GP (family medicine) training setting (Koganet al. 2009; Pelgrim et al. 2011). Contextual, content, andtrainee development factors are not necessarily generalis-able across different specialties (Epstein 2007; Tri-PartiteAlliance (RACP; RACS; RCPSC) 2014). Kogan et al.’s system-atic review found only 11 instruments that were imple-mented within GP/family medicine specialties (Kogan et al.2009) and, of these, only three had evidence of contentvalidity and only four had evidence of internal structure(Kogan et al. 2009).

An established Australian GP training WBA instrument isHays’ Consultation Assessment Scale (Hays 1990). Whilstthis instrument has established content validity (Hays 1990)and has been evaluated for interobserver reliability (Hayset al. 1990), offering an accepted framework for observingcompetence development of GP registrars (trainees) (Haysand Wellard 1998; Fraser 2007), to our knowledge, it hasnot undergone comprehensive psychometric evaluation.Furthermore, the instrument was developed to assessvideo-taped simulations and does not assess some import-ant features of competence, for example, record-keeping.The lack of interactivity also precludes deeper insight intoclinical knowledge and reasoning (Hays 1990). Introducedin 1990, it also may not reflect subsequent fundamentalchanges in Australian GP education (Trumble 2011; Guptaand Hays 2016).

The Mini-Clinical Evaluation Exercise (mini-CEX) is themost extensively used and studied WBA in medical educa-tion internationally (Pelgrim et al. 2011; Ansari et al. 2013;Sandilands and Zumbo 2014). Whilst the mini-CEX hasbeen evaluated in multiple settings including GP (Sidhuet al. 2009; Eggleton et al. 2016), three systematic reviewshave highlighted inconsistencies in validation findings forthe instrument (Kogan et al. 2009; Hawkins et al. 2010;Pelgrim et al. 2011; Sandilands and Zumbo 2014). A key

limitation of the mini-CEX in the context of formativeassessment, is that its factor structure of one global dimen-sion of clinical competence limits its utility in assessing andproviding feedback on discrete clinical skills (Cook et al.2010). Consequently, a need remains for additional psycho-metrically tested formative WBA instruments (Ansari et al.2013), particularly for use in GP vocational training.

The aims of this study were to establish the content val-idity and internal consistency of a WBA instrument, theGeneral Practice Registrar Competency Assessment Grid(GPR-CAG).

Methods

Study design and setting

The GPR-CAG was designed as a face-to-face observationtool to measure the development of GP clinical and profes-sional competencies in registrars undertaking vocational GPtraining with a large Australian training provider (GPSynergy). The primary purpose of GPR-CAG is to provideregistrars with formative educational feedback. During ourstudy period, GP Synergy expanded from two regions ofNSW (greater Sydney and New England) in 2014 to all ofNSW and the Australian Capital Territory (ACT) in 2016.With an intake in excess of 500 registrars per year, GPSynergy educates approximately one-third of Australian GPregistrars (Australian Government Department ofHealth 2017).

GPR-CAG observations were conducted at routine clin-ical teaching visits (CTVs) during core general practicetraining terms (GPTs). During CTVs, an experienced GPfrom outside the practice observes one clinical session(approximately three hours). CTV visitors are provided withtraining for implementing the GPR-CAG and providingfeedback to registrars. Each registrar receives a minimumof two CTVs in each of their first two six-month GP train-ing terms.

Instrument development

An iterative process based on a review of the literature,expert review and pilot testing was used to develop theGPR-CAG in 2012–2013.

One of the authors (RC), a GP and experienced medicaleducator, identified key competencies for GP based on areview of the international medical education literature andmedical education programs. A team of experienced GPmedical educators provided input on the draft list of com-petencies and grouped items under key headings. Thisgroup modelled the instrument into a ‘Denver-style’(Frankenburg et al. 1971) ‘developmental chart,’ reflectingexpected levels of competence development across train-ing. Workshops with experienced GP supervisors and CTVvisitors were conducted to establish expert consensus onexpected training stage ranges for emergence and devel-opment of competencies. In the final draft of the instru-ment, the items were presented as sequential phases of aGP consultation (i.e. initiation, gathering information, struc-ture, building relationship, examination and diagnosis,management and planning) in line with Hays’ WBA instru-ment (Hays 1990).

2 A. FIELDING ET AL.

Competency items were assessed on a four-point scale,with options being: ‘performing below expected level,’‘working towards the expected level,’ ‘performing atexpected level,’ and ‘performing above expected level.’ Afifth option of ‘not assessed’ was also included.

The instrument was pilot tested with 13 medical educatorCTV visitors for acceptability and usability in a subset of 88CTVs over a nine-month period (Casey et al. 2013). Resultsfrom pilot testing supported acceptability and usability, withno further changes made to the instrument. See supplemen-tary online content for original GPR-CAG items.

Validation data collection

ParticipantsData collection was undertaken between 2014 and 2016during routine CTVs for registrars undertaking their firstand second (of three) training terms (i.e. GPT1 and GPT2).GPT1 and GPT2 GPR-CAG observations were used as theseare the early training terms in which formative feedbackand identification of registrars experiencing difficulties is ofmost utility (during term 3 registrars usually sit their fellow-ship examinations).

GPR-CAG observations were completed by accreditedCTV visitors. CTV visitors are vocationally registered Fellowsof the Royal Australian College of General Practitioners(RACGP) and/or the Australian College of Rural and RemoteMedicine undertake specific CTV training. This includesGPR-CAG rater-training (face-to-face workshop supple-mented with a training manual). New CTV visitors firstthree CTV reports were reviewed for QA purposes.

ProcedureGPR-CAG observations were conducted at approximatelyfour months full-time equivalent (FTE) training time for GPT1registrars and approximately 10 months FTE training timefor GPT2 registrars. This typically represented the second(and final) CTV for the relevant 6-month FTE training term.

Statistical analysis

Descriptive analyses were performed using STATA 13.1(StataCorp, College Station, TX) and factor analyses wereperformed using SAS V9.4 (SAS Inc., Cary, NC). Descriptivedata are presented as frequencies for categorical variablesand mean with standard deviation (SD) for continu-ous variables.

Factor analysisGPT1 and GPT2 competencies were analysed separately.Only competency items expected to be reached at orbefore the time of the CTV assessment for the relevanttraining stage were included. There were 25 items includedfor GPT1 and 57 items for GPT2.

Principal components factor analysis was used, with anoblique rotation. The factors in this analysis were assumedto be correlated, based on subject matter expertise. Theoblique method allows for such correlation.

A number of observations had missing values for one ormore items due to individual items being marked as ‘notassessed.’ Missing values were handled by pairwise deletion,

with the input data set comprising a matrix of pairwise cor-relations between all relevant items; thus each pairwise cor-relation was estimated using all available data for the tworelevant items. Based on the number of observations avail-able per variable, there was an average of 22 subjects peritem for GPT1 and nine subjects per item for GPT2.

The number of factors to retain was determined via ana-lysis of eigenvalues and visual examination of scree plots.An initial minimum eigenvalue criterion of >1 was used toidentify factors to retain (Henson and Roberts 2006). Screeplots were then assessed with the eigenvalue restriction,which was relaxed if warranted by examination of the screeplot based on natural break point of the data for the curveflattening out (Costello and Osborne 2005). Internal consist-ency was assessed using Cronbach’s alpha coefficient(Bland and Altman 1997).

Item reductionItem reduction was achieved using a two-step process.First, factor loadings were evaluated for low values (load-ings of <0.3) and for absence of high unique loading (>0.4on more than one factor) (Costello and Osborne 2005).Second, an expert consensus process was used wherebythree senior medical educator GPs were asked to rate therelative clinical and educational importance of each item“within the GPT1 and GPT2 competency item lists.” Theauthors (PM, AF, AT, JB, and EH) then triangulated statis-tical and consensus process findings to establish itemsfor deletion.

Principal components analyses were conducted on therefined competency lists for GPT1 and GPT2 for evaluationof the internal consistency of the emergent factor struc-tures. Resultant factor solutions were assessed for clinicalcongruence before confirming the optimal number of fac-tors to include in the final solutions.

The final factor solutions were thematically characterisedand named by two GPs with extensive experience in GP andmedical education. Theoretical support for the derived factorswas established by evaluating the factors against existingcompetency frameworks and the medical education literature.

Ethics approval

The study was assessed by the University of Newcastle’sHuman Research Ethics Committee (HREC) as a qualityassurance activity not requiring further HREC review (refer-ence number QA147).

Results

Data of 555 registrars undertaking GPT1 and 537 registrarsundertaking GPT2 were included in analyses. See Table 1for demographic characteristics. See Figure 1 for a flow-chart of the statistical analysis process and findings.

GPT1 competencies

Of the original 25 competency items for GPT1, nine wereidentified for removal based on low factor loadings andrelatively low clinical and/or educational importance. This

MEDICAL TEACHER 3

resulted in a four-factor, 16-item solution (Table 2),accounting for 60% of total variance in the data.

The GPT1 factors were characterised as: (1) Consultationtechniques subserving patient-centeredness ‘Caring’; (2) Skillsin formulating and articulating coherent hypotheses andmanagement plans; (3) Attention to basic-level clinical profes-sional responsibilities; and (4) Proficiency in physical examin-ation skills (Table 3). Cronbach’s alphas for the four factorsranged from 0.71 to 0.83 (see Table 2).

GPT2 competencies

Of the original 57 competency items for GPT2, 28 itemswere identified for removal, resulting in a seven-factor,29-item solution. There were two items that loaded

2. Principal components analysis (PCA):

(25 items)4 factors identified with factor

loadings established

3. Item refinement using consensus process

9 items removed

4. Repeat PCA(16 items)

4-factor, 16-item solution retained statistical

acceptability

5. Expert assessment of refined factors for clinical

congruence4-factor, 16-item solution

deemed clinically congruent

1. Original GPR-CAG items categorised into GPT1 and GPT2 competencies based on stage of training competency development would be expected

3. Item refinement using consensus process

28 items removed

Item

red

ucti

on

4. Repeat PCA(29 items)

2 items removed (pragmatic decision for ease of

interpretability due to negative factor loading)

7-factor, 27-item solution established

5. Expert assessment of refined factors for clinical

congruence7-factor, 27-item solution

deemed clinically congruent

Evalu

ati

on

of

em

erg

en

t fa

cto

r str

uctu

res

6. Characterisation and naming of factors

6. Characterisation and naming of factors

GPT1 competencies

GPT2 competencies

ExplanationA factor analysis that can be used to

collapse a large number of items into fewer interpretable underling factors; provides a

metric for how strongly each item relates to the underlying factor identified (factor

loadings)

ExplanationExperienced GPs and GP MEs rated

relative importance of items within each identified factor. Authors (AF, PM, AT, EH)

and statistician then triangulated PCA findings with experts ratings to identify

items for removal within each factor

ExplanationSecond PCA of items to check that underlying factor structures remain

internally consistent/ statistically acceptable (Cronbach’s alpha) with reduced number of

items for each factor

ExplanationItems for each factor were assessed by

experienced GPs/MEs to check for clinical congruence

ExplanationExperienced GPs/MEs established thematic categorisations of identified factors based

on clinical and educational characteristics of items within each factor

2. Principal components analysis (PCA):

(57 items)7 factors identified with factor

loadings established

Figure 1. Explanatory flowchart of the item refinement process and results.

Table 1. Registrar demographics.

Variable GPT1 GPT2

Total (n) 555 537GenderFemale 359 (65%) 335 (62%)Male 196 (35%) 202 (38%)

Age (years) (mean ± SD) 32.3 (±6.0) 32.7 (±5.9)Country of medical graduationAustralia 467 (84%) 442 (82%)International 88 (16%) 95 (18%)

PathwayGeneral 391 (70%) 378 (70%)Rural 164 (30%) 159 (30%)

Full-time/part-time statusFull time 483 (87%) 453 (84%)Part time 72 (13%) 84 (16%)

4 A. FIELDING ET AL.

negatively on one factor, which were subsequently deletedfor practicality of interpretation. The deletion of theseitems did not substantially alter internal consistency. Theresulting final seven-factor, 27-item solution accounted for62% of total variance in the data (Table 4).

The GPT2 factors were characterised as: (1) Patient-cen-teredness ‘Sharing’; (2) Structural aspects of history-taking; (3)Higher-level ‘Caring’ patient-centeredness; (4) Minimum-required performance in patient-centred ‘Caring’; (5) Holisticpro-active approach to patient presentations; (6) Attention tominimum standards of professional communication; and (7)High level but structured clinical tasks (Table 5). Cronbach’salphas for the seven factors ranged from 0.63 to 0.84 (seeTable 4).

See supplementary online content for theoretical sup-port for the derived factors.

Discussion

Main findings

This study provides evidence for the content validity andinternal consistency of GPR-CAG. This is a valuable firststep for establishing overall validity of a competency

assessment instrument (Wetzel 2012). To our knowledge,this is one of the only such instruments that has beendeveloped for use in GP vocational training – the individualspecialty in which the greatest proportion of medical grad-uates practice (Eurostat – Statistics Explained 2017;Association of American Medical Colleges 2018; MedicalBoard of Australia 2018). A promising internal structure wasidentified for GPT1 and GPT2 factor solutions, withCronbach’s alphas for individual factors above, or close to,conventionally accepted thresholds for fair to good reliabil-ity (Bland and Altman 1997; Tavakol and Dennick 2011).The factor solutions for GPT1 and GPT2 explained approxi-mately 60% of the variance in GPR-CAG data, supportingvalidity of the internal structure of the instrument (Floydand Widaman 1995).

The relative lack of psychometrically evaluated instru-ments for the assessment of competence is recognised as along-standing limitation in medical education (Kogan et al.2009; Wetzel 2012). Recent guidelines for the direct obser-vation of clinical skills emphasise using an assessmentinstrument with existing validity evidence (Kogan et al.2017). The present study provides initial validity evidenceto support use of GPR-CAG as a GP vocational trainingWBA instrument.

Table 2. GPT1 GPR-CAG final factor solution.

Item no. Item Factor loading

Factor 1: ‘Consultation techniques subserving patient-centeredness “Caring”’ (six items; a¼ 0.83)CG10 Listens attentively – utilises appropriate listening skills & silence 0.81CG9 Uses open & closed techniques appropriately 0.77CG5 Listens attentively to patient’s opening, without interrupting 0.74CG23 Demonstrates appropriate non-verbal behaviour – eye contact, posture, position &

movement, vocal cues, rate volume, tone0.71

CG15 Uses concise easily understood questions & comments, avoids jargon 0.57CG6 Confirms list & screens for further problems 0.50

Factor 2: ‘Skills in formulating and articulating coherent hypotheses and management plans’ (four items; a¼ 0.77)CG39 Clearly outlines the plan of management for each defined problem; Explains expected

outcomes & influence on ongoing management of the problem0.79

CG40 Provides clear information on investigations, procedures and explains process for results 0.78CG48 Prescribes medications and treatments as appropriate to diagnostic conclusions; adopts a

quality use of medications framework; prescribes safely0.70

CG36 Appropriate hypotheses are articulated & problems defined 0.62Factor 3: ‘Attention to basic-level clinical professional responsibilities’ (four items; a¼ 0.71)CG2 Introduces self & clarifies role 0.70CG51 Medical records accurate & contemporaneous; all key & relevant info for the consultation

recorded; management plan and follow-up arrangements clearly recorded0.68

CG54 Writes clear referral letters stating reason for referral, expected outcomes & provides allnecessary patient info to facilitate the referral

0.60

CG3 Shows interest & respect, attends to patient’s physical comfort 0.49Factor 4: ‘Proficiency in physical examination skills’ (two items; a¼ 0.75)CG35 Physical examination is accurate & clinical signs correctly elicited 0.82CG34 Performs an appropriate physical examination; is mindful of patient comfort & privacy

throughout the physical examination0.78

Table 3. GPT1 GPR-CAG factor descriptions.

Factora Label Description

3 Attention to basic-level clinical professionalresponsibilities

This reflects minimum expected professional responsibilities, as characterisedby treating patients with respect and communicating adequately withprofessional colleagues.

1 Consultation techniques subserving patient-centeredness ‘Caring’

This encompasses techniques used in the consultation that enable practice ofthe ‘Caring’ elements of patient-centeredness, as measured by the PatientPhysician Orientation Scale (Krupat et al. 2000). ‘Caring’ involves doctorsbeing aware of the meaning of the disease to the individual patient andits impact on the patient’s life (Krupat et al.).

4 Proficiency in physical examination skills This reflects competent performance of appropriate physical examinations,inclusive of accuracy and patient comfort elements.

2 Skills in formulating and articulating coherenthypotheses and management plans

This encompasses the clinical reasoning skills required to synthesiseinformation into a coherent formulation, facilitating definition of problem/s,generation of hypotheses and formulation of management plans.

aFactors have been re-ordered to reflect a typical consultation sequence and/or inherent complexity of the competency trait and are therefore not innumerical order.

MEDICAL TEACHER 5

As an assessment of the ‘does’ level in Miller’s pyramid,GPR-CAG may complement validated WBA instrumentsincluding the mini-CEX (Sidhu et al. 2009; Ansari et al.2013). As GPR-CAG assesses several distinct domains ofcompetence specific to GP, it has the potential to providefiner-grained competency development feedback for GPtrainees, extending upon the global competency measureof the mini-CEX.

Best-practice validity assessment emphasises a need toexplore the theoretical relationship between the skillsassessed and the traits identified from factor analysis(Wetzel 2012; Sandilands and Zumbo 2014). The final factorsolutions were aligned with competency themes within theRACGP’s competency profile of the Australian general prac-titioner at the point of fellowship (Royal Australian Collegeof General Practitioners 2015), and similar alignment canbe identified within other competency frameworks in GP(Patterson et al. 2013) and the broader medical educationsetting, including for example, the General MedicalCouncil’s Good Medical Practice Framework and CanMEDS(General Medical Council 2013; Frank and Sherbino 2015).

The overlap between the factors identified and existingcompetency domains/frameworks was considerable but notexact. This reflects inherent complexity associated withdetermining and defining traits that underpin clinical com-petencies. Furthermore, RACGP’s Competency Profile and

other general-practice specific reference frameworks(Patterson et al. 2013) designate expected competenciesfor established GPs, whereas GPR-CAG assesses compe-tence as a developing construct during training.

There was overlap, though not perfect, of factor load-ings and structures for items common to GPT1 and GPT2competencies. It is likely that sets of traits as measured ear-lier in training may evolve into more nuanced domains asthe registrar gains experience and expertise.

Implications for practice and further research

While we have validated the GPR-CAG in the context ofCTVs within Australian GP training, it will be suitable for tri-alling in other direct observation WBA modalities in otherGP vocational training settings.

Further validation research for GPR-CAG will includeadditional response process and internal structure evalu-ation, and examination of relation with other variables(convergent and predictive validity) and consequences evi-dence (Kogan et al. 2009; Cook et al. 2014).

Response process evidence, reflecting alignmentbetween rater responses and the intended construct (Cooket al. 2014), is underpinned by comprehensive trainingprocesses (Kogan et al. 2009). Whilst rater-training for GPR-CAG was provided to CT visitors, this should be for-mally evaluated.

Table 4. GPT2 GPR-CAG final factor solution.

Item no. Item Factor loading

Factor 1: ‘Patient-centredness; “Sharing”’ (five items; a¼ 0.84)CG38 Appropriately explains causation; seriousness; expected duration, short and long-term consequences 0.77CG37 Clearly explains diagnostic conclusions, justification & check patients understanding 0.75CG43 Discusses possible options for management if appropriate and relevant and elicits

patients’ viewpoint0.69

CG41 Relates investigations and procedures to the management plan, explains value and purpose 0.67CG39 Clearly outlines the plan of management for each defined problem; Explains expected outcomes &

influence on ongoing management of the problem0.63

Factor 2: ‘Structural aspects of history-taking’ (four items; a¼ 0.79)CG14 Periodically summarise to verify own understanding, invites patient to correct interpretation 0.87CG19 Summarises periodically to confirm understanding before moving on 0.85CG21 Structures interview in a logical sequence 0.53CG7 Negotiates agenda taking patient & doctor’s needs into account 0.48

Factor 3: ‘Higher-level “Caring” Patient-centeredness’ (six items; a¼ 0.80)CG23 Demonstrates appropriate non-verbal behaviour – eye contact, posture, position & movement, vocal

cues, rate volume, tone0.73

CG26 Accepts legitimacy of patient’s views & is non-judgemental 0.67CG10 Listens attentively – utilises appropriate listening skills & silence 0.49CG18 Encourages the patient to express their feelings 0.49CG12 Picks-up verbal and non-verbal cues; appropriately addresses & acknowledges 0.44CG15 Uses concise easily understood questions & comments, avoids jargon 0.43

Factor 4: ‘Minimum-required performance in patient-centred “Caring”’ (three items; a¼ 0.71)CG2 Introduces self & clarifies role 0.82CG3 Shows interest & respect, attends to patient’s physical comfort 0.73CG4 Identifies patient’s problems or issues 0.62

Factor 5: ‘Holistic pro-active approach to patient presentations’ (three items; a¼ 0.63)CG47 Appropriately implements health promotion & identifies opportunities of behaviour change

if applicable0.72

CG6 Confirms list & screens for further problems 0.62CG49 Discusses non-medication options as appropriate and relevant to clinical context 0.54

Factor 6: ‘Attention to minimum standards of professional communication’ (three items; a¼ 0.69)CG52 Pt medical summary info & relevant family and preventative health info recorded & regularly

updated as required0.66

CG51 Medical records accurate & contemporaneous; all key & relevant info for the consultation recorded;management plan and follow-up arrangements clearly recorded

0.60

CG54 Writes clear referral letters stating reason for referral, expected outcomes & provides all necessarypatient info to facilitate the referral

0.43

Factor 7: ‘High level but structured clinical tasks’ (three items; a¼ 0.66)CG34 Performs an appropriate physical examination; is mindful of patient comfort & privacy throughout

the physical examination0.58

CG35 Physical examination is accurate & clinical signs correctly elicited 0.40CG48 Prescribes medications and treatments as appropriate to diagnostic conclusions; adopts a quality

use of medications framework; prescribes safely0.35

6 A. FIELDING ET AL.

Further evaluation and refinement of the internal struc-ture of GPR-CAG is warranted. Factors 5, 6, and 7 for GPT2had less than four items, with moderate factor loadingsand alphas. A confirmatory factor analysis in new cohortsof GP registrars is indicated. An examination of inter-rateragreement will provide evidence of reliability of GPR-CAG(Cook et al. 2014).

Examining the validity and utility of the GPR-CAG in pre-dicting fellowship exam and/or other training performanceis a key area for further research. This would provide evi-dence for utility in identifying registrars at risk of poor per-formance, and may enable implementation of early,proactive and tailored support strategies.

Beyond psychometric evaluation, it is also essential thatthe educational usefulness of GPR-CAG is established(whether its intended effect on learning is being realised)and that the feasibility and acceptability of GPR-CAG isassessed (Sandilands and Zumbo 2014).

Given the growing emphasis on programmatic assess-ment within medical education (Kogan and Holmboe2013), pending further validity assessment (as per above)there is also potential for the GPR-CAG to be consideredalongside other validated WBA modalities (such as multi-source feedback) and specific rubrics to assess clinical skillsand attributes to help gauge progression through training(Kogan and Holmboe 2013; Miller-Kuhlmann et al. 2016). Aswith any WBA, the effective implementation of theGPR-CAG (including for summative assessment) will bepredicated on the knowledge and skill of the educationalend-user (e.g. medical educator) in interpreting results(Kogan and Holmboe 2013).

Strengths and limitations

This study provides robust validity evidence when comparedwith most other WBA instruments in vocational GP training(Hays 1990; Nyman and Sheridan 1997; Fraser 2007).

Content validity of the instrument was established using athorough process, including a comprehensive literaturereview, expert opinion-seeking, and pilot testing (Cook et al.2014). Item refinement and internal structure was estab-lished via factor analysis using a large sample of registrardata and conceptual support for the competency traits iden-tified was established (Wetzel 2012; Sandilands andZumbo 2014).

In recognition that the utility of a competence assess-ment instrument is also determined by acceptability andcost (Van Der Vleuten 1996), where the resource demandsof WBAs are a known barrier to their optimal conduct (VanDer Vleuten 1996; Swayamprakasam et al. 2014), GPR-CAGhas been developed with attention to practicality, asreflected by refining the GPT1 assessment to 16 items andGPT2 to 27 items, without compromising internal structure.This may make the GPR-CAG more acceptable and cost-effective for implementation.

Several limitations are noted. While CTV visitors receivedcomprehensive training, their performance (including inter-rater reliability, test–retest reliability) was not evaluated.Also, we did not have patient data (demographics ornature and complexity of presentations) which would assistin considering external validity of the findings.

An inherent limitation of GPR-CAG is that not allincluded competencies will be observable in any onethree-hour CTV. In our analysis, we accounted for this byusing pairwise, rather than listwise deletion, thus maximis-ing use of the observed data.

Conclusions

Our study establishes evidence of content validity andinternal structure for GPR-CAG. Continued research intoother psychometric properties and performance of theinstrument will further enhance its validation profile andutility. GPR-CAG appears to have utility as a formative WBA

Table 5. GPT2 GPR-CAG factor descriptions.

Factora Label Description

6 Attention to minimum standards of professionalcommunication

This reflects clinical notes and referral letters being of an acceptable standard tocommunicate important clinical information to GP and specialist colleagues.

2 Structural aspects of history-taking This relates to specific techniques that allow the registrar to ensure that theyhave systematically elicited the appropriate history and understood thepatient’s history within a patient-centred context.

4 Minimum-required performance in patient-centred ‘Caring’

This reflects the minimum-level of performance in communicating with patientsin a patient-centred manner.

3 Higher-level ‘Caring’ patient-centeredness This encompasses higher levels skills necessary for the ‘Caring’ aspect of patient-centeredness (Krupat et al.) extending beyond the minimum requirements asper factor 3 of GPT1.

1 Patient-centeredness; ‘Sharing’ This represents the ‘Sharing’ aspect of patient-centeredness, involving doctorsand patients sharing power and information in consultations (Krupat et al.).

7 High-level, but structured clinical tasks Conceptually, the items within this factor (including physical examination andprescribing practices) require considerable training and experience but arenevertheless ‘structured.’ Physical examination for each body system is highlystandardised; and once a diagnosis has been made, for most instances,conformity with evidence-based prescribing guidelines and quality use ofmedicines is reasonably standardised. These tasks require high-levelprofessional competence but can be contrasted with the clinical reasoningskills often employed in hypothesis generation, diagnosis and managementdecisions/plans taking into account complexities of the context of thepatient/problem.

5 Holistic pro-active approach to patientpresentations

This reflects taking a pro-active rather than reactive approach to consultationsand adopting a holistic approach. The registrar does more than the minimumrequired to get through the consultation. This includes exploration ofadditional issues (including prevention) and going beyond simple medicationprescription in their management approach.

aFactors have been re-ordered to reflect a typical consultation sequence and/or inherent complexity of the competency trait and are therefore not innumerical order.

MEDICAL TEACHER 7

instrument for competence development within GP voca-tional training, contributing to assessment of the ‘does’level of Miller’s pyramid.

Acknowledgements

The authors wish to acknowledge GP Synergy’s CompetencyAssessment Working Group (Dr Graham Lee, Dr Tony Saltis, Dr JoannaBruce, and Dr Richard Griffiths) for their advisory input on evaluationof the GPR-CAG; Dr Catherine Casey and Dr Eszter Fenessey for theircontribution to initial development of the GPR-CAG; and Dr ChrisStarling for assistance with characterisation of the competencies.

Disclosure statement

The authors report no conflicts of interest. The authors alone areresponsible for the content and writing of the article.

Glossary

Clinical Teaching Visit: A common type of formative work-place-based assessment in Australian vocational general prac-tice training, where a trained, experienced general practitionerobserves the consultations of general practice trainees and pro-vides structured feedback and makes recommendations toimprove performance.

Fraser J. 2007. Registrar clinical teaching visits: evaluation of anassessment tool. Aust Fam Physician. 36(12):1070–1072.

Notes on contributors

Alison Fielding, BND (Hons), PhD, is a Conjoint Lecturer at theUniversity of Newcastle and the Research Project Manager at GPSynergy, New South Wales, Australia

Katie Mulquiney, BND (Hons), is a Senior Research Assistant, GPSynergy, New South Wales, Australia.

Rosa Canalese, MBBS, Dip Paed, FRACGP, MPH, Grad. Cert. Uni Teach,is a Senior Medical Adviser, Avant Mutual Group Limited, Australia andformer Director of Training (2011–2015), GP Synergy, New SouthWales, Australia

Amanda Tapley, M Med Stat, B Biomed Sci, is the Senior ResearchOfficer and Statistician, GP Synergy, New South Wales, Australia

Elizabeth Holliday, PhD, BSc, MSc, is an Associate Professor ofBiostatistics in the School of Medicine and Public Health at theUniversity of Newcastle, and senior statistical affiliate at the HunterMedical Research Institute, New South Wales, Australia.

Jean Ball, B Math, Grad Dip Med Stats, is a statistical programmer withthe Hunter Medical Research Institute, Newcastle, Australia.

Linda Klein, BSc (Hons), MSc, PhD, is a Conjoint Senior Lecturer,University of Newcastle, Honorary Senior Lecturer, University ofSydney, and the Deputy Director, NSW & ACT Research and EvaluationUnit, GP Synergy, New South Wales, Australia.

Parker Magin, MBBS (Hons), PhD, MFM (Clin), MGPP, GDipClinEpi, DPD,DipPaed, FRACGP, is a Conjoint Professor, School of Medicine andPublic Health, University of Newcastle, and the Director, NSW & ACTResearch and Evaluation Unit, GP Synergy, New South Wales, Australia

ORCID

Alison Fielding http://orcid.org/0000-0001-5884-3068

References

Ansari AA, Ali SK, Donnon T. 2013. The construct and criterion validityof the mini-CEX: a meta-analysis of the published research. AcadMed. 88(3):413–420.

Association of American Medical Colleges. 2018. Active physicians inthe largest specialties, 2015; [accessed 2018 Aug 13]. https://www.aamc.org/data/workforce/reports/458480/1-1-chart.html.

Australian Government Department of Health. 2017. General practicestatistics: GP workforce statistics – 2001–02 to 2016–17; [accessed2018 Apr 6]. http://health.gov.au/internet/main/publishing.nsf/Content/General+Practice+Statistics-1.

Bland JM, Altman DG. 1997. Statistics notes: Cronbach’s alpha. BMJ.314(7080):572.

Casey C, Sallos A, Canalese R. 2013. ‘PICC’ing our way to the diamonds– performance indicators in clinical competencies for general prac-tice (conference presentation); [accessed 2018 Sept 10]. http://www.onqconferences.com.au/resources/files/gpet2013/absother/W-CatherineCasey%20Piccing.pdf.

Cook DA, Beckman TJ, Mandrekar JN, Pankratz VS. 2010. Internal struc-ture of mini-CEX scores for internal medicine residents: factor ana-lysis and generalizability. Adv Health Sci Educ Theory Pract. 15(5):633–645.

Cook DA, Zendejas B, Hamstra SJ, Hatala R, Brydges R. 2014. Whatcounts as validity evidence? Examples and prevalence in a system-atic review of simulation-based assessment. Adv Health Sci Educ.19(2):233–250.

Costello A, Osborne J. 2005. Best practices in exploratory factor ana-lysis: four recommendations for getting the most from your ana-lysis. Pract Assessment. 10(7):1–9.

Crossley J, Jolly B. 2012. Making sense of work-based assessment: askthe right questions, in the right way, about the right things, of theright people. Med Educ. 46(1):28–37.

Dick M-L, King DB, Mitchell GK, Kelly GD, Buckley JF, Garside SJ. 2007.Vertical Integration in Teaching And Learning (VITAL): an approachto medical education in general practice. Med J Aust. 187(2):133–135.

Eggleton K, Goodyear-Smith F, Paton L, Falloon K, Wong C, Lack L,Kennelly J, Fishman T, Moyes S. 2016. Reliability of mini-CEX assess-ment of medical students in general practice clinical attachments.Fam Med. 48(8):624–630.

Epstein RM. 2007. Assessment in medical education. N Engl J Med.356(4):387–396.

Epstein RM, Hundert EM. 2002. Defining and assessing professionalcompetence. JAMA. 287(2):226–235.

Eurostat – Statistics Explained. 2017. Healthcare personnel statistics –physicians; [accessed 2018 Aug 13]. http://ec.europa.eu/eurostat/statistics-explained/index.php?title=Healthcare_personnel_statistics_-_physicians.

Floyd FJ, Widaman KF. 1995. Factor analysis in the development andrefinement of clinical assessment instruments. Psychol Assess. 7(3):286–299.

Frank SL, Sherbino J, editors. 2015. CanMEDS 2015 physician compe-tency framework; [accessed 2018 May 8]. http://www.royalcollege.ca/rcsite/documents/canmeds/canmeds-full-framework-e.pdf.

Frankenburg WK, Camp BW, Van Natta PA, Demersseman JA. 1971.Reliability and stability of the Denver Developmental ScreeningTest. Child Dev. 42(5):1315–1325.

Fraser J. 2007. Registrar clinical teaching visits—evaluation of anassessment tool. Aust Fam Physician. 36(12):1070–1072.

General Medical Council. 2013. Good medical practice; [accessed 2018May 8]. https://www.gmc-uk.org/ethical-guidance/ethical-guidance-for-doctors/good-medical-practice.

Gupta T, Hays R. 2016. Training for general practice: how Australia’sprograms compare to other countries. Aust Fam Physician. 45(1):18–21.

Hawkins RE, Margolis MJ, Durning SJ, Norcini JJ. 2010. Constructing avalidity argument for the mini-clinical evaluation exercise: a reviewof the research. Acad Med. 85(9):1453–1461.

Hays R, Wellard R. 1998. In-training assessment in postgraduate train-ing for general practice . Med Educ. 32(5):507–513.

Hays RB. 1990. Assessment of general practice consultations: contentvalidity of a rating scale. Med Educ. 24(2):110–116.

Hays RB, Jones BF, Adkins PB, McKain PJ. 1990. Analysis of videotapedconsultations to certify competence. Med J Aust. 152(11):609–611.

8 A. FIELDING ET AL.

Hays RB, Morgan S. 2011. Australian and overseas models of generalpractice training. Med J Aust. 194(11):S63–S64.

Henson R, Roberts J. 2006. Use of exploratory factor analysis in pub-lished research: common errors and some comment on improvedpractice. Educ Psychol Meas. 66(3):393–416.

Holmboe ES, Sherbino J, Long DM, Swing SR, Frank JR. 2010. The roleof assessment in competency-based medical education. Med Teach.32(8):676–682.

Iobst WF, Sherbino J, Cate OT, Richardson DL, Dath D, Swing SR, HarrisP, Mungroo R, Holmboe ES, Frank JR. 2010. Competency-basedmedical education in postgraduate medical education. Med Teach.32(8):651–656.

Kogan J, Holmboe E. 2013. Realizing the promise and importance ofperformance-based assessment. Teach Learn Med. 25(Suppl. 1):S68–S74.

Kogan JR, Hatala R, Hauer KE, Holmboe E. 2017. Guidelines: the do’s,don’ts and don’t knows of direct observation of clinical skills inmedical education. Perspect Med Educ. 6(5):286–305.

Kogan JR, Holmboe ES, Hauer KE. 2009. Tools for direct observationand assessment of clinical skills of medical trainees: a systematicreview. JAMA. 302(12):1316–1326.

Krupat E, Rosenkranz SL, Yeager CM, Barnard K, Putnam SM, Inui TS.2000. The practice orientations of physicians and patients: theeffect of doctor-patient congruence on satisfaction. Patient EducCouns. 39(1):49–59.

Medical Board of Australia. 2018. Registrant data; [accessed 2018 Jun 26].http://www.medicalboard.gov.au/documents/default.aspx?record=WD18%2f25279&dbid=AP&chksum=6JdBYTqxEIO4aRsnbdrFzQ%3d%3d.

Miller-Kuhlmann R, O’Sullivan PS, Aronson L. 2016. Essential steps indeveloping best practices to assess reflective skill: a comparison oftwo rubrics. Med Teach. 38(1):75–81.

Miller A, Archer J. 2010. Impact of workplace based assessment ondoctors’ education and performance: a systematic review. BMJ.341(1):c5064.

Miller GE. 1990. The assessment of clinical skills/competence/perform-ance. Acad Med. 65(9 Suppl.):S63–S67.

Nyman K, Sheridan B. 1997. Evaluation of consulting skills of traineegeneral practitioners. Aust Fam Physician. 26(Suppl. 1):S28–S35.

Patterson F, Tavabie A, Denney M, Kerrin M, Ashworth V, Koczwara A,MacLeod S. 2013. A new competency model for general practice:implications for selection, training, and careers. Br J Gen Pract.63(610):e331–e338.

Pelgrim EAM, Kramer AWM, Mokkink HGA, van den Elsen L, Grol R,van der Vleuten C. 2011. In-training assessment using direct obser-vation of single-patient encounters: a literature review. Adv HealthSci Educ. 16(1):131–142.

Royal Australian College of General Practitioners. 2015. Competencyprofile of the Australian general practitioner at the point of fellow-ship; [accessed 2017 Nov 4]. https://www.racgp.org.au/download/Documents/VocationalTrain/Competency-Profile.pdf.

Sandilands D, Zumbo BD. 2014. (Mis)Alignment of medical educationvalidation research with contemporary validity theory: the mini-CEXas an example. In: Zumbo BD, Chan EKH, editors. Validity and valid-ation in social, behavioral, and health sciences. Cham: SpringerInternational Publishing; p. 289–310.

Sidhu RS, Hatala R, Barron S, Broudo M, Pachev G, Page G. 2009.Reliability and acceptance of the mini-clinical evaluation exercise asa performance assessment of practicing physicians. Acad Med.84(Suppl.):S113–S115.

Swayamprakasam AP, Segaran A, Allery L. 2014. Work-based assess-ments: making the transition from participation to engagement.JRSM Open. 5(3):2042533313515861.

Tavakol M, Dennick R. 2011. Making sense of Cronbach’s alpha. Int JMed Educ. 2:53–55.

Thompson S, Anderson K, Mara P, Stevenson A. 2011. Supervision —growing and building a sustainable general practice supervisor sys-tem. Med J Aust. 194(11):S101–S104.

Tri-Partite Alliance (RACP; RACS; RCPSC). 2014. Work-based assessment:a practical guide; [accessed 2017 Dec 19]. https://www.racp.edu.au/docs/default-source/default-document-library/worked-bases-assessment-racp.pdf?sfvrsn=2.

Trumble SC. 2011. The evolution of general practice training inAustralia. Med J Aust. 194(11):S59–S62.

Van Der Vleuten C. 1996. The assessment of professional competence:developments, research and practical implications. Adv Health SciEduc. 1(1):41–67.

van der Vleuten C, Verhoeven B. 2013. In-training assessment develop-ments in postgraduate education in Europe. ANZ J Surg. 83(6):454–459.

Wass V, Van der Vleuten C, Shatzer J, Jones R. 2001. Assessment ofclinical competence. Lancet. 357(9260):945–949.

Wetzel AP. 2012. Factor analysis methods and validity evidence: areview of instrument development across the medical educationcontinuum. Acad Med. 87(8):1060–1069.

MEDICAL TEACHER 9