Assessment Development: AnIntroduction to NBCC & CCEAssessment Processes
2019 /2020
C A L L I E M A R I N OI N T E R I M V I C E P R E S I D E N T , C R E D E N T I A L I N GA N D Q U A L I T Y A S S U R A N C E I S B A H A L I F A R Z A ND I R E C T O R , P S Y C H O M E T R I C I A N D A V I D W H I T T I N G H I L LC O N T E N T D E V E L O P M E N T C O O R D I N A T O R
K Y L I E D O T S O N - B L A K EI N T E R I M P R E S I D E N T & C E O
A S S E S S M E N TP R O G R A M L E A D E R S
I N T R O D U C T I O N
The National Board for Certified Counselors maintains credentialing
examinations for the profession of counseling that are utilized for state
licensure and national certification. The infrastructure for developing,
administering and maintaining these exams is extensive and engages
stakeholders and professionals throughout the process. There are many
levels of controls and oversight to ensure the integrity of the process.
While it is critical that NBCC maintains the privacy of proprietary information
to ensure test security, our organization is striving to become more
transparent about our test development process whenever possible to better
inform stakeholders of the processes and intense data-driven work that goes
into maintaining valid and reliable credentials.
This report shares information about the development process of the
following two exams:
National Counselor Examination (NCE)
National Clinical Mental Health Counselor Examination (NCMHCE)
PROFESSIONAL VOICE
Subject Matter Experts (SMEs)Examination CommitteesTechnical Advisory CommitteeProfessional Community Nominees
CONTENT OUTLINEDEVELOPMENT
Job AnalysisContent Outline Development
ITEM AND FORM DEVELOPMENT
Item WritingForm DevelopmentStandard Setting
ITEM REVIEW AND REVISION
Item Review
03
A S S E S S M E N TD E V E L O P M E N TC O M P O N E N T S
06
07
08
ACCOUNTABILITY CONTROLS
Layered Professional Oversight09
PROFESSIONALVOICEA fundamental requirement for a strong national modelis the inclusion of the profession's voice in theprocess of developing and maintaining the model. Theassessments are central to the national model ofcertification and the profession has voice in theassessments at a number of different points, primarilyby serving as Subject Matter Experts (SMEs) and bysharing their experiences through the job analysissurvey responses. Counseling professionals have voice in theassessment development process in the followingways:
Though there are specific requirements for SMEs foreligibility on certain committees, all SMEs have the baselinerequirements for eligibility:
SMEELIGIBILITY
P A G E 3
SMEs serve on the committee to create the JobAnalysisCounselors complete the Job Analysis surveyExam Committee SMEs review the statisticaloutcomes from the job analysis and developthe content outlineSMEs serve as item writers to create contentfor the assessmentsPsychometric Experts serve as leaders on theTechnical Advisory Committee (TAC)SMEs engage in standard setting and formreview
Education: Minimum of a Master’s Degree in theirrespective field. Individuals with a Doctoral Degree mustmake up the majority of committee membership.Experience: Minimum of five years of professionalexperience within the field. Experience must correlate withthe exam of focus.Certification. Be in good standing with their respectivecredentials and licenses.
The Technical Advisory Committee (TAC) provides guidanceon technical assessment matters pertaining to contentoutline and examination validity, reliability, accuracy, andfairness. Members of the TAC are highly regarded nationalexperts who have been widely published in their fields.Areas of expertise include: assessment design; computeradaptive testing (CAT); assessment accommodations; usesof tests; mathematics, and disciplines specific to the arrayof professions served by the assessment programs.The TAC is comprised of external professionals. TheAssessment Department Director and thepsychometrician(s) meet directly with the TAC. The TACapproves the form and the scholarly materials utilized forcontent development.
Oversight & Leadership
TECHNICALADVISORYCOMMITTEE
The state regulatory community holds powerful insights aboutthe professional responsibilities and necessary competenciesfor practicing counselors. The involvement of professionalsfrom the regulatory community in the development of theexams ensures that the protection of the public remains at theheart of all that we do as an organization. The regulatorycommunity is recruited into the examination developmentprocess through AASCB. AASCB will consistently provide upto six nominations for exam committee membership, withmembership seats for each examination selected from AASCBnominees. AASCB nominees must also meet eligibilityrequirements, ensuring that professionals selected for SMEservice have the necessary assessment expertise to contributeeffectively to the work of the committee.
Engaging the Regulatory Community
AASCB NOMINEES
P A G E 4
E X A M D E V E L O P M E N TA M u l t i f a c e t e d P r o c e s s
Exam development for standardized exams is a multifaceted process with extensive
infrastructure requirements and engagement by a wide group of professionals. Most
critically, exam development is an ongoing, cyclical process.
Exam development includes the following components:
Job Analysis & Content Outline Development
Item Writing
Form Creation and Standard Setting
Item Review and Revision
Throughout these steps, Subject Matter Experts provide critical voice and external,
autonomous oversight. The development of the NCE and NCMHCE has always relied on
the engagement of external SMEs and experts in statistical analyses and psychometric
work. These professionals have been critically important to ensuring the strength and
integrity of the assessment programs.
NBCC approached the development of the
clinical certification exam with an intentional
focus on drawing from the common core
knowledge of professional counselors. In
order to develop the exam, NBCC convened
a committee of subject matter experts
(SMEs) to explore exam format and develop
items built on the core knowledge base of
practicing counselors. This committee was
led by psychometric experts with a deep
knowledge of the counseling profession and
the clinical practice of counselors. The dual
expertise in psychometrics and counseling
uniquely positioned the committee to frame
an exam that reflected the central clinical
requirements of counselors through real
world simulated cases.
H I S T O R Y O F N C M H C E : U T I L I Z I N G E X P E R T K N O W L E D G E
The exam was developed with a clear intention
to focus on the clinical application of core
counseling knowledge. This team of SMEs
developed a content outline for the
assessment reflective of core counseling
clinical areas of practice. The format of the
assessment was presented as counseling case
simulations comprising real-life clinical
narratives followed by questions measuring the
skills of the examinees to gather information
and formulate clinical decisions. This case
simulation format has served the needs of the
profession well for many years and
contemporary technology, and deeper
psychometric analysis now offers innovative
opportunities to continue to strengthen the
alignment of the assessment with the clinical
practice of counselors.
P A G E 5
02
J o b a n a lys i s & C O n t e n t o u t l i n e
d e v e l o p m e n tThe most recent job analysis was f inal ized in the Summer of 2019 and a newcontent out l ine has been released. The job analysis process begins wi thsurvey development. In 2016-2017, two commit tees of Subject Matter Expertsworked together to develop the job analysis survey. These commit teemembers part ic ipated in the ident i f icat ion of professional components and job-related tasks. The commit tee also worked to draf t i tems, review the surveyformat and provide overal l feedback and direct ion to the psychometr ic ian andstaf f working on the job analysis. The survey was distr ibuted to more than100,000 physical addresses and more than 130,000 emai l addresses. The pr imary goal of the job analysis survey was to ident i fy the work tasksmost prevalent wi th in the counsel ing profession. After complet ion of thestat ist ical analysis of survey resul ts, the Psychometr ic ian and staf f reviewedthe data wi th the Exam Commit tee and the SMEs engaged in an intenseper iod of developing the content out l ine. Fol lowing the development of thecomprehensive content out l ine, the SMEs reviewed and recategor ized eachi tem in the i tem bank to al ign wi th the new content out l ine.
j o b a n a lys i s : H i s t o r i c a l i n f o r m a t i o n
Prior to the complet ion of the 2019 jobanalysis and content out l ine, the previousjob analysis and content out l ine for the NCEwas completed in 2010. The 2010 NCE jobanalysis was conducted with theengagement of an exper ienced commit tee ofSubject Matter Experts and staf f , who werealso responsible for the development of thecontent out l ine. Al l subsequent NCEexaminat ion forms were developed by ExamCommittees ut i l iz ing the content out l inefrom the 2010 publ ished job analysis. TheNCMHCE forms have al l been developedfrom the content out l ine approved by theExam Commit tee based upon expertknowledge held by the SMEs in 2008.
P A G E 6
07
I t e m wr i t i n g & f o r m c r e a t i o nCONTENT DEVELOPMENT
S M E T E A M : I T E M W R I T E R S &
E X A M C O M M I T T E E S
S T A N D A R DS E T T I N G
C O M M I T T E E
Histor ical ly and for the current exam i tems in use, i tems are developed by SMEsand stored on secure servers and in secure i tem bank databases. Throughextensive engagement by Exam Commit tee SMEs, i tem wri ters and the ContentDevelopment Coordinator, i tems are constructed and research ci tat ions areprovided for each i tem. I tems are developed and reviewed by the SMEs, andconsensus must be reached by the ful l commit tee before an i tem is approved foruse as a f ie ld i tem. I tems are f ie ld- tested and reviewed again by thePsychometr ic ian and Exam Commit tee pr ior to being released as an approved i temfor inclusion in the assessment. The process ensures that each i tem is reviewedby a number of professionals in a control led, secure environment before beingapproved for ut i l izat ion. Each form of the NCE and NCMHCE includes anestabl ished number of i tems being f ie ld- tested. Test forms are assembled by the Psychometr ic ian and reviewed and approved bythe Exam Commit tee pr ior to being inserted into a test ing cycle. Once 300examinees or more have tested on a form, standard sett ing is conducted by thePsychometr ic ian and Commit tee and then the form is f inal ized released for fu l luse.
C O N T E N TD E V E L O P M E N TC O O R D I N A T O R
P A G E 7
09
O n g o i n g q u a l i ty c o n t r o l :
I t e m R e v i ew a n d R e v i s i o n
SUSTAINING
CURRENT
EXAMINATION
PERFORMANCE
EVALUATION
AND
DEEPENING
ITEM LEVEL
REVIEW
Current assessment development best pract icesindicate that assessments should be evaluated annual lyby SMEs across mult ip le dimensions, including: i temdif f icul ty, i tem discr iminat ion, d istractor ef fect iveness,candidate volume, score distr ibut ion, re l iabi l i ty , averagescore, standard deviat ion, standard error of measureand percentage of candidates passing. PreviouslyNBCC Psychometr ic ians and SMEs evaluatedassessments, but did not ut i l ize i tem discr iminat ion inthe evaluat ion. Start ing wi th the 2019-2020 examinat ioncycle, the lead Psychometr ic ian wi l l evaluateassessments by reviewing i tems for al l components,including i tem discr iminat ion, deepening the review ofi tem level data. I tems that are determined stat ist ical lyto less ef fect ively discr iminate wi l l be reviewed furtherby the Content Development Coordinator, who wi l l makea determinat ion to ei ther revise or ret i re the i tem. Byengaging in an ongoing, mult i faceted review of i tems,the assessment team wi l l be able to ensure that theassessments cont inue to remain al igned with the needsof the profession and current best pract ices inassessment.
P A G E 8
05
A S S E S S M E N TD E V E L O P M E N T C O M P O N E N T ESTABLISHED CONTROL
As s e s s m e n t P r o g r a m
Ac c o u n t a b i l i ty C o n t r o l s
Involves survey creation, mail ing &distribution of surveys via email.
J O B A N A L Y S I S S U R V E Y C R E A T I O NJOB ANALYSIS COMMITTEE
OF SUBJECT MATTEREXPERTS
J O B A N A L Y S I S S T A T I S T I C A L A N A L Y S I S
PSYCHOMETRICIANInvolves statistical analysis, report creation& presentation of report to Exam Committee
C O N T E N T O U T L I N E D E V E L O P M E N T PSYCHOMETRICIAN & EXAM COMMITTEE SUBJECT
MATTER EXPERTSInvolves review of the job analysisreport and identif ication of key tasks
I T E M W R I T I N G & F O R M C R E A T I O N CONTENT DEVELOPMENTCOORDINATOR &
EXAM COMMITTEE SMES,ASSESSMENT STAFF
Involves developing items based on thecontent outl ine and compiling form.
S T A N D A R D S E T T I N G PSYCHOMETRICIAN &EXAM COMMITTEE SUBJECT
MATTER EXPERTSAfter the new form has been uti l ized by atleast 300 candidates, the standard settingprocess is completed.
I T E M R E V I E W A N D R E V I S I O N
CONTENT DEVELOPMENTCOORDINATOR & EXAM
COMMITTEE SMES,ASSESSMENT STAFF
Items are monitored after exam cycles anditems that don't meet necessary standards areflagged to be revised or retired.An established number of items are field testedduring each examination cycle to ensure thatthere is a robust pool of approved items.
P A G E 9