Assessment Development: AnIntroduction to NBCC & CCEAssessment Processes

2019 /2020

C A L L I E M A R I N OI N T E R I M V I C E P R E S I D E N T , C R E D E N T I A L I N GA N D Q U A L I T Y A S S U R A N C E I S B A H A L I F A R Z A ND I R E C T O R , P S Y C H O M E T R I C I A N D A V I D W H I T T I N G H I L LC O N T E N T D E V E L O P M E N T C O O R D I N A T O R

K Y L I E D O T S O N - B L A K EI N T E R I M P R E S I D E N T & C E O

A S S E S S M E N TP R O G R A M L E A D E R S

I N T R O D U C T I O N

The National Board for Certified Counselors maintains credentialing

examinations for the profession of counseling that are utilized for state

licensure and national certification. The infrastructure for developing,

administering and maintaining these exams is extensive and engages

stakeholders and professionals throughout the process. There are many

levels of controls and oversight to ensure the integrity of the process.

While it is critical that NBCC maintains the privacy of proprietary information

to ensure test security, our organization is striving to become more

transparent about our test development process whenever possible to better

inform stakeholders of the processes and intense data-driven work that goes

into maintaining valid and reliable credentials.

This report shares information about the development process of the

following two exams:

National Counselor Examination (NCE)

National Clinical Mental Health Counselor Examination (NCMHCE)

PROFESSIONAL VOICE

Subject Matter Experts (SMEs)Examination CommitteesTechnical Advisory CommitteeProfessional Community Nominees

CONTENT OUTLINEDEVELOPMENT

Job AnalysisContent Outline Development

ITEM AND FORM DEVELOPMENT

Item WritingForm DevelopmentStandard Setting

ITEM REVIEW AND REVISION

Item Review

03

A S S E S S M E N TD E V E L O P M E N TC O M P O N E N T S

06

07

08

ACCOUNTABILITY CONTROLS

Layered Professional Oversight09

PROFESSIONALVOICEA fundamental requirement for a strong national modelis the inclusion of the profession's voice in theprocess of developing and maintaining the model. Theassessments are central to the national model ofcertification and the profession has voice in theassessments at a number of different points, primarilyby serving as Subject Matter Experts (SMEs) and bysharing their experiences through the job analysissurvey responses. Counseling professionals have voice in theassessment development process in the followingways:

Though there are specific requirements for SMEs foreligibility on certain committees, all SMEs have the baselinerequirements for eligibility:

SMEELIGIBILITY

P A G E 3

SMEs serve on the committee to create the JobAnalysisCounselors complete the Job Analysis surveyExam Committee SMEs review the statisticaloutcomes from the job analysis and developthe content outlineSMEs serve as item writers to create contentfor the assessmentsPsychometric Experts serve as leaders on theTechnical Advisory Committee (TAC)SMEs engage in standard setting and formreview

Education: Minimum of a Master’s Degree in theirrespective field. Individuals with a Doctoral Degree mustmake up the majority of committee membership.Experience: Minimum of five years of professionalexperience within the field. Experience must correlate withthe exam of focus.Certification. Be in good standing with their respectivecredentials and licenses.

The Technical Advisory Committee (TAC) provides guidanceon technical assessment matters pertaining to contentoutline and examination validity, reliability, accuracy, andfairness. Members of the TAC are highly regarded nationalexperts who have been widely published in their fields.Areas of expertise include: assessment design; computeradaptive testing (CAT); assessment accommodations; usesof tests; mathematics, and disciplines specific to the arrayof professions served by the assessment programs.The TAC is comprised of external professionals. TheAssessment Department Director and thepsychometrician(s) meet directly with the TAC. The TACapproves the form and the scholarly materials utilized forcontent development.

Oversight & Leadership

TECHNICALADVISORYCOMMITTEE

The state regulatory community holds powerful insights aboutthe professional responsibilities and necessary competenciesfor practicing counselors. The involvement of professionalsfrom the regulatory community in the development of theexams ensures that the protection of the public remains at theheart of all that we do as an organization. The regulatorycommunity is recruited into the examination developmentprocess through AASCB. AASCB will consistently provide upto six nominations for exam committee membership, withmembership seats for each examination selected from AASCBnominees. AASCB nominees must also meet eligibilityrequirements, ensuring that professionals selected for SMEservice have the necessary assessment expertise to contributeeffectively to the work of the committee.

Engaging the Regulatory Community

AASCB NOMINEES

P A G E 4

E X A M D E V E L O P M E N TA M u l t i f a c e t e d P r o c e s s

Exam development for standardized exams is a multifaceted process with extensive

infrastructure requirements and engagement by a wide group of professionals. Most

critically, exam development is an ongoing, cyclical process.

Exam development includes the following components:

Job Analysis & Content Outline Development

Item Writing

Form Creation and Standard Setting

Item Review and Revision

Throughout these steps, Subject Matter Experts provide critical voice and external,

autonomous oversight. The development of the NCE and NCMHCE has always relied on

the engagement of external SMEs and experts in statistical analyses and psychometric

work. These professionals have been critically important to ensuring the strength and

integrity of the assessment programs.

NBCC approached the development of the

clinical certification exam with an intentional

focus on drawing from the common core

knowledge of professional counselors. In

order to develop the exam, NBCC convened

a committee of subject matter experts

(SMEs) to explore exam format and develop

items built on the core knowledge base of

practicing counselors. This committee was

led by psychometric experts with a deep

knowledge of the counseling profession and

the clinical practice of counselors. The dual

expertise in psychometrics and counseling

uniquely positioned the committee to frame

an exam that reflected the central clinical

requirements of counselors through real

world simulated cases.

H I S T O R Y O F N C M H C E : U T I L I Z I N G E X P E R T K N O W L E D G E

The exam was developed with a clear intention

to focus on the clinical application of core

counseling knowledge. This team of SMEs

developed a content outline for the

assessment reflective of core counseling

clinical areas of practice. The format of the

assessment was presented as counseling case

simulations comprising real-life clinical

narratives followed by questions measuring the

skills of the examinees to gather information

and formulate clinical decisions. This case

simulation format has served the needs of the

profession well for many years and

contemporary technology, and deeper

psychometric analysis now offers innovative

opportunities to continue to strengthen the

alignment of the assessment with the clinical

practice of counselors.

P A G E 5

02

J o b a n a lys i s & C O n t e n t o u t l i n e

d e v e l o p m e n tThe most recent job analysis was f inal ized in the Summer of 2019 and a newcontent out l ine has been released. The job analysis process begins wi thsurvey development. In 2016-2017, two commit tees of Subject Matter Expertsworked together to develop the job analysis survey. These commit teemembers part ic ipated in the ident i f icat ion of professional components and job-related tasks. The commit tee also worked to draf t i tems, review the surveyformat and provide overal l feedback and direct ion to the psychometr ic ian andstaf f working on the job analysis. The survey was distr ibuted to more than100,000 physical addresses and more than 130,000 emai l addresses. The pr imary goal of the job analysis survey was to ident i fy the work tasksmost prevalent wi th in the counsel ing profession. After complet ion of thestat ist ical analysis of survey resul ts, the Psychometr ic ian and staf f reviewedthe data wi th the Exam Commit tee and the SMEs engaged in an intenseper iod of developing the content out l ine. Fol lowing the development of thecomprehensive content out l ine, the SMEs reviewed and recategor ized eachi tem in the i tem bank to al ign wi th the new content out l ine.

j o b a n a lys i s : H i s t o r i c a l i n f o r m a t i o n

Prior to the complet ion of the 2019 jobanalysis and content out l ine, the previousjob analysis and content out l ine for the NCEwas completed in 2010. The 2010 NCE jobanalysis was conducted with theengagement of an exper ienced commit tee ofSubject Matter Experts and staf f , who werealso responsible for the development of thecontent out l ine. Al l subsequent NCEexaminat ion forms were developed by ExamCommittees ut i l iz ing the content out l inefrom the 2010 publ ished job analysis. TheNCMHCE forms have al l been developedfrom the content out l ine approved by theExam Commit tee based upon expertknowledge held by the SMEs in 2008.

P A G E 6

07

I t e m wr i t i n g & f o r m c r e a t i o nCONTENT DEVELOPMENT

S M E T E A M : I T E M W R I T E R S &

E X A M C O M M I T T E E S

S T A N D A R DS E T T I N G

C O M M I T T E E

Histor ical ly and for the current exam i tems in use, i tems are developed by SMEsand stored on secure servers and in secure i tem bank databases. Throughextensive engagement by Exam Commit tee SMEs, i tem wri ters and the ContentDevelopment Coordinator, i tems are constructed and research ci tat ions areprovided for each i tem. I tems are developed and reviewed by the SMEs, andconsensus must be reached by the ful l commit tee before an i tem is approved foruse as a f ie ld i tem. I tems are f ie ld- tested and reviewed again by thePsychometr ic ian and Exam Commit tee pr ior to being released as an approved i temfor inclusion in the assessment. The process ensures that each i tem is reviewedby a number of professionals in a control led, secure environment before beingapproved for ut i l izat ion. Each form of the NCE and NCMHCE includes anestabl ished number of i tems being f ie ld- tested. Test forms are assembled by the Psychometr ic ian and reviewed and approved bythe Exam Commit tee pr ior to being inserted into a test ing cycle. Once 300examinees or more have tested on a form, standard sett ing is conducted by thePsychometr ic ian and Commit tee and then the form is f inal ized released for fu l luse.

C O N T E N TD E V E L O P M E N TC O O R D I N A T O R

P A G E 7

09

O n g o i n g q u a l i ty c o n t r o l :

I t e m R e v i ew a n d R e v i s i o n

SUSTAINING

CURRENT

EXAMINATION

PERFORMANCE

EVALUATION

AND

DEEPENING

ITEM LEVEL

REVIEW

Current assessment development best pract icesindicate that assessments should be evaluated annual lyby SMEs across mult ip le dimensions, including: i temdif f icul ty, i tem discr iminat ion, d istractor ef fect iveness,candidate volume, score distr ibut ion, re l iabi l i ty , averagescore, standard deviat ion, standard error of measureand percentage of candidates passing. PreviouslyNBCC Psychometr ic ians and SMEs evaluatedassessments, but did not ut i l ize i tem discr iminat ion inthe evaluat ion. Start ing wi th the 2019-2020 examinat ioncycle, the lead Psychometr ic ian wi l l evaluateassessments by reviewing i tems for al l components,including i tem discr iminat ion, deepening the review ofi tem level data. I tems that are determined stat ist ical lyto less ef fect ively discr iminate wi l l be reviewed furtherby the Content Development Coordinator, who wi l l makea determinat ion to ei ther revise or ret i re the i tem. Byengaging in an ongoing, mult i faceted review of i tems,the assessment team wi l l be able to ensure that theassessments cont inue to remain al igned with the needsof the profession and current best pract ices inassessment.

P A G E 8

05

A S S E S S M E N TD E V E L O P M E N T C O M P O N E N T ESTABLISHED CONTROL

As s e s s m e n t P r o g r a m

Ac c o u n t a b i l i ty C o n t r o l s

Involves survey creation, mail ing &distribution of surveys via email.

J O B A N A L Y S I S S U R V E Y C R E A T I O NJOB ANALYSIS COMMITTEE

OF SUBJECT MATTEREXPERTS

J O B A N A L Y S I S S T A T I S T I C A L A N A L Y S I S

PSYCHOMETRICIANInvolves statistical analysis, report creation& presentation of report to Exam Committee

C O N T E N T O U T L I N E D E V E L O P M E N T PSYCHOMETRICIAN & EXAM COMMITTEE SUBJECT

MATTER EXPERTSInvolves review of the job analysisreport and identif ication of key tasks

I T E M W R I T I N G & F O R M C R E A T I O N CONTENT DEVELOPMENTCOORDINATOR &

EXAM COMMITTEE SMES,ASSESSMENT STAFF

Involves developing items based on thecontent outl ine and compiling form.

S T A N D A R D S E T T I N G PSYCHOMETRICIAN &EXAM COMMITTEE SUBJECT

MATTER EXPERTSAfter the new form has been uti l ized by atleast 300 candidates, the standard settingprocess is completed.

I T E M R E V I E W A N D R E V I S I O N

CONTENT DEVELOPMENTCOORDINATOR & EXAM

COMMITTEE SMES,ASSESSMENT STAFF

Items are monitored after exam cycles anditems that don't meet necessary standards areflagged to be revised or retired.An established number of items are field testedduring each examination cycle to ensure thatthere is a robust pool of approved items.

P A G E 9