TABLE OF CONTENTS - Monterey County, CA · sample - oral examination questions ... suggestions for training interview panelists ... sample - road maintenance helper instruction for

3/6/00

TABLE OF CONTENTS

SECTION 6. TESTING GUIDELINES......................................................................................6-1 6.1 EXAMINATION GUIDELINES .......................................................................................................... 6-1

6.2 TESTING ADMINISTRATION ........................................................................................................... 6-1

6.3 QUALIFICATIONS APPRAISAL (QA)............................................................................................... 6-2

6.4 ORAL EXAMINATIONS ................................................................................................................... 6-4

6.5 WRITTEN EXAMINATIONS ............................................................................................................. 6-8

6.6 PERFORMANCE EXAMINATIONS.................................................................................................... 6-9

6.7 TESTING ACCOMMODATION.......................................................................................................... 6-9

6.8 ELIGIBLE LISTS............................................................................................................................ 6-11

6.9 EXAM APPEALS ........................................................................................................................... 6-14

GUIDELINES CONFIRMATION LETTER TO ORAL EXAM PANEL MEMBERS ..........................................................G 6.4.1

PREPARATION OF EXAM MATERIALS ............................................................................................G 6.4.2

ORAL EXAM APPEAL PROCEDURE.................................................................................................G 6.4.3

WRITTEN EXAMS - SETTING PASS-POINTS.....................................................................................G 6.5.1

SUBDIVISION OF ELIGIBLE LISTS...................................................................................................G 6.8.1

ALTERNATE ELIGIBLE LISTS .........................................................................................................G 6.8.2

SIDE-BY-SIDE ELIGIBLE LIST.........................................................................................................G 6.8.3

MERGING OF CANDIDATES TO A NEW EXAM PROCESS..................................................................G 6.8.4

REMOVAL FROM ELIGIBLE LIST ....................................................................................................G 6.8.5

TIE SCORES - ESTABLISHING RANK...............................................................................................G 6.8.6

EXAM APPEALS .............................................................................................................................G 6.9.1

CONTINUANCE OF EXAMINATION PROCESS ..................................................................................G 6.9.2

COUNTY OF MONTEREY EXAMINATION APPEAL FORM.................................................................G 6.9.3

DISCRIMINATION COMPLAINTS.....................................................................................................G 6.9.4

EXHIBITS PANEL MEMBER SIGN-IN SHEET.................................................................................................... E 6.1.1

TEST MATERIAL REQUEST FORM................................................................................................... E 6.1.2

PROCEDURES FOR QUALIFICATIONS APPRAISAL........................................................................... E 6.3.1

BENCHMARK DEFINITIONS - QUALIFICATIONS APPRAISAL........................................................... E 6.3.2

MONTEREY COUNTY POSITION TITLE SCREEN GRID ..................................................................... E 6.3.3

QA PANEL BRIEFING...................................................................................................................... E 6.3.4

MONTEREY COUNTY PERSONNEL GUIDELINE














EXAM ACTION SHEET - QA ............................................................................................................ E 6.3.5

CHECKLIST FOR QUALIFICATIONS APPRAISAL.............................................................................. E 6.3.6

COUNTY OF MONTEREY KNOWLEDGE LEVEL DEFINITIONS .......................................................... E 6.3.7

THANK YOU LETTER QUALIFICATIONS APPRAISAL PANEL MEMBER ............................................ E 6.3.8

ORAL DIMENSION LIST.................................................................................................................. E 6.4.1

A GUIDE FOR ORAL BOARD MEMBERS .......................................................................................... E 6.4.2

MAP OF SALINAS SHOWING PRINCIPAL MONTEREY COUNTY BUILDINGS ..................................... E 6.4.3

MOTELS IN THE SALINAS AND SURROUNDING AREAS................................................................... E 6.4.4

CHECKLIST FOR ORAL EXAMINATION........................................................................................... E 6.4.5

SAMPLE - PROCTOR SCRIPT ADMINISTRATIVE SERVICES DIRECTOR PRE-EXAM EXERCISE.......... E 6.4.6

COUNTY OF MONTEREY SAMPLE - SCRIPT PRE-EXAM WRITING EXERCISE ................................... E 6.4.7

SAMPLE - ORAL EXAM PANEL RATING FORM................................................................................ E 6.4.8

SAMPLE - ORAL EXAMINATION QUESTIONS.................................................................................. E 6.4.9

ORAL EXAMINATION PANEL BRIEFING ....................................................................................... E 6.4.10

ORAL EXAMINATION CANDIDATE INFORMATION SHEET ............................................................ E 6.4.11

COUNTY OF MONTEREY HUMAN RESOURCES DIVISION EXAMINATION SCHEDULE .................... E 6.4.12

COUNTY OF MONTEREY PERSONNEL DIVISION EXAMINATION SCHEDULE - 2 PHASE................. E 6.4.13

BENCHMARK DEFINITIONS ORAL RATING................................................................................... E 6.4.14

EXAMPLE OF GOOD VS. POOR BEHAVIORAL STATEMENTS ......................................................... E 6.4.15

SUGGESTIONS FOR TRAINING INTERVIEW PANELISTS................................................................. E 6.4.16

COUNTY OF MONTEREY ORAL EXAM QUESTION APPEAL FORM.................................................. E 6.4.17

EXAM ACTION SHEET - ORAL...................................................................................................... E 6.4.18

ORAL BOARD CONFIRMATION LETTER........................................................................................ E 6.4.19

THANK YOU ORAL PANEL MEMBERS .......................................................................................... E 6.4.20

COUNTY OF MONTEREY ORAL EXAMINATION PROCESS APPEAL FORM ...................................... E 6.4.21

EXAM ACTION SHEET - WRITTEN .................................................................................................. E 6.5.1

TEST PROCTORING STANDARDS.................................................................................................... E 6.5.2

CHECKLIST FOR WRITTEN EXAMS................................................................................................. E 6.5.3

COUNTY OF MONTEREY WRITTEN EXAM QUESTIONS APPEAL FORM ............................................ E 6.5.4

COUNTY OF MONTEREY WRITTEN EXAMINATION PROCESS APPEAL FORM .................................. E 6.5.5

SAMPLE - ROAD MAINTENANCE HELPER INSTRUCTION FOR MAIN RECEPTION AREA................... E 6.6.1

SAMPLE - PERFORMANCE EXAM WAIVER OF RESPONSIBILITY ..................................................... E 6.6.2

COUNTY OF MONTEREY PERFORMANCE EXAM QUESTIONS APPEAL FORM................................... E 6.6.3

COUNTY OF MONTEREY AMERICANS WITH DISABILITY ACT (ADA) TESTING ACCOMMODATION .......... REQUEST FORM .............................................................................................................. E 6.7.1

SCREENED OUT LETTER ................................................................................................................ E 6.8.1

PASSED PAPER - ON LIST............................................................................................................... E 6.8.2

INVITED TO ORAL EXAM ............................................................................................................... E 6.8.3

INVITED TO WRITTEN EXAM ......................................................................................................... E 6.8.4

PASSED EXAM - ON LIST ............................................................................................................... E 6.8.5

FAILED EXAM................................................................................................................................ E 6.8.6

LIST ABOLISHED ........................................................................................................................... E 6.8.7

NOT SELECTED - RETURN TO LIST................................................................................................. E 6.8.8

NOT SELECTED - REMOVE FROM LIST ........................................................................................... E 6.8.9

CONTINUOUS RECRUITMENT - ON LIST....................................................................................... E 6.8.10

EXAM ACTION SHEET - ELIGIBLE LIST ........................................................................................ E 6.8.11

COUNTY OF MONTEREY EXAMINATION APPEAL FORM................................................................. E 6.9.1

3/6/00

SECTION 6. TESTING GUIDELINES

6.1 EXAMINATION GUIDELINES

Examination development is critical to the quality of the recruitment and selection process and truly sets the standard for all candidates. This stage of recruitment and selection is hinged on the knowledge, skill and abilities identified in the job analysis. Examination factors are developed around the critical work behaviors linked to the job specifications.

While the Analyst and the Hiring Manager play important roles in the examination process, it is the subject matter experts (SME’s) who determine qualification based on information on the application materials and SQ responses.

The success of a recruitment hinges on competent SME’s who can assist in insuring that the examination process is structured to predict successful work performance. Exam development involves establishment of examination procedures and types of tests that will best measure candidate qualifications and differentiate between candidates. Examinations should meet all of the legal considerations outlined in Section 2.

Examinations shall be competitive and designed to qualify and/or rank candidates based on the knowledge, skills and abilities required of the position. Applicants shall be notified of the time, place and types of examination, and will be notified of their status after each examination process is complete. Subject Matter Experts (SME’s) shall be used in the preparation of examinations and may be used in the rating of applicants.

In situations involving disabled applicants, the examination process shall not be a barrier to the employment of qualified individuals. The applicant must make requests for accommodation and special examination arrangements to the Analyst handling the recruitment. Special arrangements can then be made to accommodate the applicant.

6.2 TESTING ADMINISTRATION

Examination administration relates to the coordination and management of all elements of the examination process. Analysts are expected to maintain control of the examination process through proper and adequate administration. This control includes a very structured procedure to ensure that each recruitment is equitable and consistent. Testing administration includes:

• Date(s)/Timeframe of Exam(s); • Time and Location of Exam(s); • Qualification Rater Selection; • Candidate Notification; • Briefing and Debriefing; • Documentation.

6-1

In terms of managing the testing itself, the Analyst will oversee the process, including logistical arrangements and appointed proxies, and will provide general support to the SME’s and Hiring Manager. This support may mean intervening, if necessary or appropriate, to preserve the integrity of the examination proceedings.

REVIEW OF EXAM MATERIALS To ensure that future use of exams will not be invalidated, contents of written, oral or performance exams are confidential and may not be reviewed by candidates. Analyst must exercise good professional judgement in the sharing of exam materials with other Personnel professionals. When sharing exam materials, the Analyst should obtain a statement from the recipient that the test materials will be kept secure. Analysts are expected to share any materials requested by WRIPAC members on the Test Material Request Form (E 6.1.2).

SCORING Scoring will be based on the responses or performance demonstrated during the examination. A candidate may not alter examination responses after the conclusion of the examination.

PASS-POINTS: Pass-points shall be reasonable and consistent with normal expectations of acceptable proficiency within the workforce, and generally accepted professional standards. Candidates failing any phase may be disqualified from the examination process. CONDUCT DURING EXAMINATIONS A candidate's examination may be disqualified for:

• Improperly communicating with any other candidate or giving or receiving any unauthorized assistance;

• Copying or collusive act(s); • Directly or indirectly obtaining any confidential examination information; • Any action which threatens the security or violates the prescribed examination instructions; • Falsifying identity; • Reporting late; • Disruptive behavior.

6.3 QUALIFICATIONS APPRAISAL (QA)

Applicants meeting the basic application requirements are advanced to the QA phase of the examination process. In the QA phase the Personnel Analyst and/or Subject Matter Experts (SME’s) review the applications, resumes and supplemental question responses (if applicable) of applicants, and assign a numerical rating to each of the minimum qualifications which have been pre-determined to be assessed at this phase. A numerical rating is necessary to distinguish which candidates will be advanced further in the process and to establish rank if no further exam process is conducted.

6-2

Exceptions to the QA process are acceptable when pre-determined through the R&E Plan; in order to:

• Advance all applicants directly to the written or performance examination phase without a qualifications appraisal or;

• Screen, rate and/or rank applicants by submission of answers on a Scantron Form.

To properly evaluate the applicants, the Analyst develops a screening grid (E 6.3.3), which reflects the specific criteria required in the SQ’s, to identify those applicants determined to be best qualified. Screening teams of SME’s representing one or more departments participate in the paper screening process, with support from the Analyst. Raters shall be subject matter knowledgeable and familiar with the job classification being tested. Good faith efforts shall be made to provide raters with ethnic and gender diversity that is reflective of the applicant pool. The following guidelines should be helpful in this process:

1. Follow the rating structure established based on the job analysis or other relevant information;

2. Respect the confidentiality of each applicant; 3. Avoid conflicts of interest in rating an application—ratings assigned should be based

exclusively on the data provided in the application materials; 4. If rating for specific work experience, distinguish direct and actual work experience from

theory statement; 5. Avoid “cloning”, and do not allow prejudices about alumni schools, businesses other

agencies to skew the evaluation; 6. Allow sufficient time for effectively rating each application.

SELECTING SUBJECT MATTER EXPERTS AS QA RATERS QA panel rater(s) should be Subject Matter Experts who are familiar with the requirements of the job or character of the work in the classification being examined. If more than one rater is used, a good faith effort should be made to have raters who are from different race/ethnic groups and genders. In cases where the examination is offered as (promotional only), or where a large number of current County employees are applicants, it is recommended that all, or a majority of raters be from an organization other than Monterey County. The same rater should not participate in adjacent steps in the examination and selection process.

The identity of raters is considered confidential and should only be shared with department representative involved in the recruitment and exam process. The identity of raters may not be shared with the applicant.

The preferred situation is to have all raters screen applications materials at the same location and time to allow the sharing of rater perspectives and to assure that all raters receive and understand the same instructions. In some cases, it may be necessary to have raters screen independently of each other, with little or no discussion or interaction on the final results. In such cases, an increased burden is on the Analyst to assure the consistency of direction and information given to the raters and to determine the appropriate disposition of an application over which the raters may disagree.

6-3

BRIEFING AND DEBRIEFING QA PANELS The Analyst must assure that all raters are appropriately and consistently instructed on the Qualifications Appraisal process. The Analyst must present a (briefing) to all raters before the rating process begins to familiarize them with the position, the requirements, the rating criteria and their responsibilities to rate objectively and maintain confidentiality. At the conclusion of the rating process, the Analyst must present a (debriefing) to all raters to review ratings, assure that appropriate comments support all disqualified applicants, and remind raters of their responsibility to maintain confidentiality.

Briefing and debriefing must be performed in accordance with the Qualifications Appraisal Rater Briefing Outline. (E 6.3.4)

A rater may disqualify him/herself from rating a particular applicant when the rater feels that s/he cannot rate the applicant without bias. If all raters disqualify themselves from rating the same applicant, a single alternate rater may be used, however that rater must rate all applicants.

FEEDBACK TO APPLICANTS Applicants may request feedback regarding their Qualifications Appraisal rating. The Analyst should exercise professional judgment in the release of information to the applicant within the following criteria:

1. The identity of the raters is confidential

2. The average rating in each minimum qualification area may be released (not each raters individual rating)

3. Any comments must be attributed to the (Panel) and not to any one rater

Information should be given in a way as to provide constructive feedback to the applicants which can assist them understanding how they didn’t meet the minimum qualifications or how they could better prepare themselves to compete in the future.

6.4 ORAL EXAMINATIONS

ORAL EXAMINATION PROCESS The oral examination is typically the final phase of the examination process and is used in determining the final rank and score of each applicant, thus establishing the Eligible List. The decision on whether or not to administer an oral examination should be made during the recruitment and examination planning process by the Analyst and the Hiring Manager. Considerations to be made in determining the appropriateness of an oral examination include:

• Level of position (oral exams are highly recommended for positions which involve supervisory or administrative duties);

• Number of applicants; • Number of internal candidates; • Departmental timeline for filling the position; • Scope of minimum qualifications assessed during the paper screening process.

6-4

DEVELOPING ORAL EXAMINATION QUESTIONS Analysts will work closely with the departmental subject matter expert in the development of appropriate and defensible questions that will measure the critical minimum KSA’s as identified in the job analysis for the recruitment. The best sources for question gathering/development are the Analyst, a departmental SME, and/or previously used questions found in the history file for this classification.

In testing, the Analyst should choose the KSA’s that will distinguish the superior worker from the barely acceptable; identify the global KSA’s; and eliminate the KSA’s that cannot be tested in an oral exam setting.

Once the appropriate KSA’s have been identified, the dimensions to be rated must be selected. The identified KSA’s should be grouped into related clusters under the umbrella of a global dimension (e.g., “ability to promote cooperation” and “ability to mediate conflicts” could be grouped under the dimension of “Interpersonal Skills”).

Types of questions to be considered are:

• Knowledge-Based – provides information for assessing knowledge’s possessed. • Training and Experience – assesses a candidate’s achievements gained through previous

training and experience. • Situational – asks the candidate to describe how they would handle or respond to a job-

related problem or predicament; panel looks at the behaviors the candidate describes. The following customer service question must be included in every oral panel:

“Providing excellent customer service is a critical component of each job in the County of Monterey. Tell us about your specific customer service experience and skills relevant to this position.”

Phrase questions as simply and directly as possible; avoid excessive verbiage that does not provide information necessary for the candidate to respond.

The use of follow-up questions by the panel should be limited to clarification of the candidate’s response or requesting a more in-depth response; any modification to the original questions must be avoided in order to maintain consistency for each candidate.

E 6.4.9 and E 6.4.8 show the formatting for oral questions and rating forms.

CHOOSING DIMENSIONS (See “Oral Dimension List” E 6.4.1).

SELECTING ORAL PANEL MEMBERS Oral Examination panels should consist of three or more persons, the majority of whom are deemed to be Subject Matter Experts who are familiar with the requirements of the job or character of the work in the classification being examined.

In determining Oral Examination panel members, the following criteria should be used to establish a panel in which the individuals:

• Reflect the ethnicity/race and gender of the applicants participating in the exam; • Are job experts in the same or higher classifications;

6-5

• Are bias-free regarding the candidates; • Possess good judgment and skill in observing and rating behavior.

In order to enhance protection from allegations of bias, when County employees are used as panel members, there should be no more than one panelist from the same department or major division within a department.

The same rater should not participate in adjacent steps in the examination and selection process.

In the event that a previously scheduled panel member is unable to participate and there is not adequate time to secure an alternate rater, two members may make a final rating. A rater may disqualify him/herself from rating a particular applicant when the rater feels that s/he cannot rate the applicant without bias.

PREPARATION OF EXAM MATERIALS The following are included for your reference:

• Exam Action Sheet—Oral (E 6.4.18) • Confirmation Letter to Oral Exam Panel Members (E 6.4.19) • Accommodations List (E 6.4.4) • Map of County Facilities (E 6.4.3) • A Guide to Oral Board Members (E 6.4.2) • Oral Exam Candidate Information Sheet (E 6.4.11) • Oral Exam Schedule (w/pre-exam exercise)— (E 6.4.13) • Oral Exam Schedule (w/o pre-exam exercise)— (E 6.4.12) • Checklist for Oral Examination (E 6.4.5) • Proctor Script—Pre Oral Exam Exercise (E 6.4.6) • Pre-Exam Study Period Proctor Instructions • Briefing Outline—Oral Exam (E 6.4.10) • Oral Exam Benchmark Definitions (E 6.4.14) • Examples of Good vs. Poor Behavioral Statements (E 6.4.15) • Oral Exam Questions (E 6.4.9) • Rating Form—Oral Exam (E 6.4.8) • Oral Exam Question Appeal Form (E 6.4.17) • Oral Exam Process Appeal Form (E 6.4.21)

ORAL PANEL—USE OF APPLICATIONS All application materials, including resumes, shall be provided to the oral exam panel for their review. If time permits, copies of the application materials will be mailed to the oral exam panel members prior to the day of the exam. If time does not allow for this, copies will be provided in the exam panel packets and sufficient time allowed between candidates to review the materials.

The application materials are confidential and panel members must be informed that they are not to share them with anyone outside of the Human Resources Division. They must bring these documents with them to the exam or shred them if they are unable to attend the exam.

6-6

If, because of unusual circumstances, the panel will not review the application materials, the Analyst is expected to:

• Inform candidates via the invitation to interview letter and the candidate information sheet, and; (E 6.4.11)

• Inform the panel members in the confirmation letter. (E 6.4.19)

ORAL PANEL—HIRING DEPARTMENT ROLE Human Resources is responsible for protecting the County and the hiring function from even the appearance or suggestion of bias or impropriety in violation of Federal and State law, County ordinance, and/or existing decrees and agreements. Therefore, the assigned Analyst must carefully plan for appropriate and prudent use of hiring department representatives in the qualifications appraisal and exam processes.

During the oral exam process, a hiring department representative may:

• Serve as a SME during the exam development process, and/or suggest people to participate in the process;

• Sit on the panel, provided s/he is not participating in another adjacent stage of screening, testing or selection;

• Be used in orientation if the Analyst carefully defines his/her purpose and monitors the orientation;

• Take part in the debriefing only if serving as a panel member;

ORAL PANEL—PROCTORING DURING LUNCH Oral panel members often come from outside our local area and are willing to assist us in finding the best candidates for our positions with no remuneration for their services. As a panel is typically utilized for a minimum of one day, the hiring department provides their lunch, either at a local restaurant or by having food brought to the site of the exam.

If the oral panel is taken to a restaurant for lunch, the Analyst or Personnel Technician must accompany the panel in order to monitor the conversation. Confidentiality could easily and unintentionally be breached through a public conversation regarding the candidates who have already been through the process. It is the responsibility of the Analyst or Personnel Technician to assure that the panel understands the need for confidentiality and that the lunch conversation does not include any discussion of the candidates or exam process.

ORAL EXAM APPEALS The Human Resources Division has established procedures for addressing concerns regarding any aspect of the examination process. Concerns regarding examinations must be filed in a timely manner and presented to the Analyst prior to a score being assigned. The Analyst is expected to actively assist in the resolution of all appeals at the lowest possible level. Individuals filing appeals must be notified in writing of the determination of the appeal and any further appeal actions they may exercise.

• Oral Exam Questions Appeal Form (E 6.4.17)

6-7

6.5 WRITTEN EXAMINATIONS

EXAM DEVELOPMENT During the recruitment and exam planning, the Analyst will assist the department in deciding if a written exam will be an effective method of evaluating those applicants who are the most qualified to perform the duties of the job. The Analyst will utilize professional personnel standards and guidelines in the development of the written exam.

The Analyst should work with the Senior Analyst responsible for Recruitment and Exam and may access all resources available in the Human Resources Division regarding the development of written exams.

PURCHASING EXAMS Cooperative Personnel Services (CPS) has a variety of exams available for rent including P.O.S.T. and Board of Corrections (B.O.C.) exams. A purchase order and security agreement is established with CPS. When an exam is ordered from CPS, they will ship the requested number of exam booklets directly to the Analyst responsible for the exam. The department is charged a base fee and an additional charge per booklet ordered. Booklets come pre-packaged in batches along with proctor instructions, Scrantron Forms and scratch paper. All booklets must be returned to CPS. CPS will score the exams and send the item analysis and scoring information back to the department analyst for review. Any adverse impact should be reported to CPS for correction. CPS provides specific instructions on the return shipment of test booklets.

For more information on the services provided by CPS and a listing of the exams offered for rent, call (916) 263-3600.

SCORING EXAMS Written exams developed by Monterey County professional Human Resources staff for specific classifications is scored electronically through the SIGMA AMS.

Written exams that are purchased or rented by Monterey County from an outside agency may be scored either, by that agency or scored electronically through the SIGMA AMS.

If any appealed questions have been deleted, they are not to be counted in the total number of questions.

(See (E 6.5.1) “Exam Action Sheet—Written Form", (E 6.5.3) “Checklist for Written Examination”, and (E 6.5.4) “Written Exam Question Appeal Form” and (E 6.5.5) "Written Exam Process Appeal Form".

ROOM ARRANGEMENT Test sites should be selected that provide the optimal testing situation. When selecting a site, consider the following:

• Number of applicants to be tested; • Length of the test; • Number of test administrators; • Accessibility to public transportation;

• Environment; • Temperature; • Lighting; • Noise level.

6-8

Test rooms should be arranged in a manner that provides adequate space between applicants. In a classroom with desks, applicants should be seated at least one seat apart. If there are tables and chairs, there should be at least one chair between applicants and chairs should be staggered instead of placed directly across from one another.

A table with two chairs should be provided for the proctors to conduct the check-in procedure.

CHECK-IN PROCESS FOR WRITTEN EXAMS Written exams always required the applicant to provide photo identification before being allowed to participate in the written exam process. Applicants are notified in their invite letter that this is a requirement for admittance to the exam.

Proctors should have a separate table at the entrance to the exam room. At the time that was noted on the invite letter, the proctors open the doors to the exam room and ask that everyone please have their photo ID ready to present. One proctor checks the photo ID of the applicant, saying the name aloud. The other proctor checks off the name on the alpha roster. Prior to beginning the exam, the number of seated applicants is compared to the number of names checked off the alpha roster to be sure that no applicant was missed in the check-in process.

6.6 PERFORMANCE EXAMINATIONS

Performance exams, or work samples, are useful for measuring job skills or the application of knowledge(s). It is difficult and costly to develop reliable performance tests, and they are expensive to administer. The more complex the skill, the more difficult test development becomes. Therefore, as a general rule, performance tests should be used only in those situations where other testing methods are not desirable; for example, where qualified applicants have low reading ability.

The decision to use a performance exam is made mutually between the Analyst and the department. Using the Job Analysis for Recruitment, the Analyst determines whether a performance exam would distinguish between the superior and acceptable candidates.

These exams may require special locations, equipment, scheduling candidates for multiple tests, and separate rater panels.

6.7 TESTING ACCOMMODATION

The County recognizes the value persons with disabilities can contribute to the services and products provided to the public. Disabled persons are encouraged to pursue employment with Monterey County. For this reason, the issue of “reasonable accommodation” may arise in the testing and examination process. This means that no candidate is denied the opportunity to apply for a position because of barriers that may preclude him/her from completing the application process or participating in the examination.

DETERMINING ELIGIBILITY REQUIREMENTS Persons with disabilities must be able to:

• Complete and submit the application and any supplemental materials;

6-9

• Have access to facilities where the examination processes are conducted and to the workplace where the position exists;

• Participate in the examination process.

Disabled applicants should be evaluated solely on ability to perform the job. Once the disabled applicant successfully completes the examination process and is determined to be qualified, he/she is placed on the eligible list. At the point an offer of employment is anticipated, a preliminary evaluation should be developed in coordination with the Analyst and the Hiring Manager regarding the department’s ability to “reasonably” accommodate the specific disability. Decisions to accommodate must be reviewed by HRD prior to offering employment.

ACCOMMODATION OPTIONS Monterey County’s reasonable accommodations in testing is outlined below:

I. DISABILITY/HANDICAPPED DEFINED; Various government agencies and organizations have developed different definitions of disability or handicapped. The most recent and most comprehensive law is the Americans with Disabilities Act (ADA) of 1990. The Personnel Division will use ADA’s definitions and past legal determinations for granting preference points and for evaluating requests for reasonable accommodations.

A. An individual with a “disability” has (1) a physical or mental impairment that substantially limits one or more major life activities; OR (2) a record of such impairment: OR (3) is regarded as having such an impairment.

B. “Major life activities” generally include: caring for oneself, walking, seeing, hearing, speaking, and working.

C. Compliance agencies have included physiological disorders, cosmetic disfigurements, mental illness, mental retardation, and specific learning disabilities. Recovered users of illegal drugs and former alcoholics are generally considered to have a disability.

D. EXCLUDED conditions include homosexuals, bisexuals, transvestites, transsexuals, pedophiles, compulsive gamblers, kleptomaniacs, pyromaniacs, and current users of illegal drugs. An alcoholic with unsatisfactory job performance is not disabled, even if the unsatisfactory performance is related to the alcoholism.

II. A request for reasonable accommodation for pre-employment testing is not automatic. Case by case decisions are made based on the nature of the request, nature of the disability, and the nature of the job. All requests for reasonable accommodations must be discussed with the Senior Analyst.

A. The applicant completes and returns the Reasonable Accommodation Request Form (RARF) (E 6.7.1) to the Analyst.

B. Within one working day the Analyst will review the RARF and contact the applicant to discuss any possible accommodations that may be required. The “verification contact” should be contacted if more information is needed.

6-10

C. The Analyst will make a recommendation to the Senior Analyst concerning possible reasonable accommodations in testing. In recommending this, the Analyst should consider the following:

1. The Monterey County HRD is committed to being an equal opportunity employer as a matter of practice. The testing medium should not be a barrier to the employment of qualified individuals. We will make a reasonable accommodation in testing whenever such accommodation is not needed on the job or could be made on the job.

2. We will not make an accommodation for a disability that would obviously prevent an applicant from performing the essential functions of the job. For example, we will not provide the reader for a totally blind applicant for Animal Control Officer or Produce Inspector.

3. The need and the provision of reasonable accommodation must be decided case by case. Generalizations about a specific disability and accommodation needed must be avoided. A decision to grant or deny an accommodation should only be made after obtaining all necessary information. The applicant should be contacted before a decision is made. What might have been successful for one individual, may not be appropriate for another. An applicant may have adjusted to his/her disability so that an accommodation is not necessary or wanted.

4. Many accommodations will require testing the applicant at a different time and/or location from the general session. This should be the practice when time limits are extended or when the test medium is modified. The convenience and comfort of the applicant should be considered. Also, strive to reduce the possible disruption at the general test session.

5. Common modifications to the test medium include having the test read to the applicant, enlarged print, recording the test on cassette, having the test translated into Braille or sign language. Consider the preferences of the applicant, possible job accommodations, and the length and complexity of the test material.

6. Time limits are frequently increased when an applicant has neurological, muscular, skeletal or similar disabilities. There is no established standard for determining the time to allow. Time and one-half or double time are the most common.

7. Other accommodations may include providing staff to turn the pages of the test booklet and marking the answer sheet. On occasion, a part of the general test has been eliminated. If a reader or other personal assistant is needed, it is our obligation to provide whatever is needed.

8. Individuals with temporary disabilities (i.e., cast on hand) may be accommodated when it is practical. Allowing an applicant to mark his/her response in the test booklet rather than on an answer sheet would be practical. Extensive accommodations are not normally provided for individuals with temporary disabilities.

D. The Analyst will complete the RARF indicating the accommodations made or the reasons for denial. The Analyst will notify the applicant of the decision made. The completed RARF should be filed in the exam file.

6.8 ELIGIBLE LISTS

In accordance with established procedures and practices, no individual can be hired to a “permanent position” with the County without being placed first on a referral list. The first step in creating a referral list is to establish an Eligible List of potential candidates. Analysts are responsible for developing an Eligible List at the completion of every recruitment. The Eligible List is composed of candidates who meet the minimum requirements of the position and have passed each phase of the examination process.

6-11

ESTABLISHING AN ELIGIBLE LIST Eligible Lists become effective on the date the Analyst completes the analysis of examination scores and assigns final scores and ranks to candidates. All candidates shall be listed on the Eligible List in rank order of scores, including applicable veteran’s preference points.

All candidates beinfg placed on an Eligible List shall be so notified through the receipt of Sigma Notice E – Passed Exam on List or similar letter.

Changes in rank, addition and deletion of names because of errors or re-rankings shall not change the effective date of the Eligible List.

VETERAN’S PREFERENCE POINTS An applicant who has received a passing grade in an open entry level examination for Monterey County employment and who is a veteran shall receive credit for an additional five (5) grade points to be added to the final examination grade; except that a veteran’s widow or disabled veteran shall be entitled to ten (10) grade points to be added to the final examination grade. The widow of a veteran, who is likewise a veteran, shall not be entitled to more than ten (10) grade points to be added to the final examination grade.

INTEREST SURVEYS During the duration of an Eligible List, the Analyst may request that all remaining candidates specify their interest in remaining on the Eligible List. Interest surveys must be conducted in writing, to the last known address of the candidate, and should clearly identify the name of the Eligible List and the deadline date for submission of written response to the interest survey. Candidates should be allowed a response time of at least seven (7) working days past the date the interest survey reasonably would have been received. A candidate can be removed from the Eligible List if he/she makes the request in the survey response, or fails to respond by the established deadline. Interest surveys may also be used to establish a candidate’s interest in particular job openings for purposes of referring candidates based on Special Certification for Candidate Qualifications.

ELIGIBLE LIST MAINTENANCE CORRECTIONS TO ELIGIBLE LISTS When it becomes evident that an error occurred in the preparation or management of an Eligible List, the Analyst must prepare a corrected list. The names of individuals who do not qualify shall be removed from the List, however, any prior appointments shall remain valid.

REINSTATEMENT TO ELIGIBLE LIST Upon receipt of acceptable written request from the candidate, the Analyst may reinstate a candidate to an active Eligible List upon which he/she formerly existed, for the following reasons:

• Candidate was appointed to a permanent position from which he/she subsequently resigned. • Candidate was removed from an Eligible List based on a (promotional only) exam process

due to resignation from the County and has returned as a regular County employee. • Candidate presents good reason for failing to respond to an interest survey, invitation to

interview, or scheduled interview.

6-12

PREFERRED ELIGIBLE LISTS TRANSFER, VOLUNTARY DEMOTION AND REINSTATEMENT As an alternative to appointment from an Eligible List, a position may be filled by transfer, voluntary demotion, or reinstatement, as defined in the Personnel Policies and Practices Resolution.

CODING RETURNED REFERRALS LIFE OF THE ELIGIBLE LIST ESTABLISHING THE LIFE OF AN ELIGIBLE LIST Eligible Lists established solely on a Qualifications Appraisal (paper-screening) expire six (6) months from the date of establishment. If a continuous or open application filing exam process was used, the candidate may remain active on the Eligible List for no more than six (6) months from the date he/she was originally placed on the Eligible List.

Eligible Lists established from a competitive examination process other than solely a Qualifications appraisal expire one (1) year from the date established. If a continuous or open application filing exam processes was used, the candidate may remain active on the eligible list for no more than one (1) year from the date he/she was originally placed on the eligible list.

EXTENSION OF ELIGIBLE LIST An Eligible List may be extended for six (6) months past the expiration date upon the request of the hiring authority when:

• There are qualified, available candidates on the existing List; • There would be no adverse impact on protected group members; • The Eligible List has not been officially abolished.

The Analyst is responsible to consider all the factors surrounding the determination and exercise professional judgment in recommending courses of action to the hiring authority.

ABOLISHING ELIGIBLE LISTS Eligible Lists may be abolished prior to their expiration date for the following reasons:

• Selective certification is required and the Eligible List contains no minority candidates; • Fewer than five candidates remain on the Eligible List and the hiring authority wishes to

conduct a new recruitment and exam process; • The Eligible List no longer meets the needs of the County.

When an Eligible List is abolished prior to the expiration date, all candidates remaining in active status on the List shall be notified of such through the receipt of SIGMA Notice G – List Abolished Applicant Notifications.

(For details, see the following exhibits at the end of this section):

• SIGMA NOTICE A – Screened out later • SIGMA NOTICE B – Passed Paper – On List • SIGMA NOTICE C – Invited to Oral Exam • SIGMA NOTICE D – Invited to Written Exam

6-13

• SIGMA NOTICE E – Passed Exam • SIGMA NOTICE F – Failed Exam • SIGMA NOTICE G – List Abolished • SIGMA NOTICE H – Not Selected – Return to List • SIGMA NOTICE I – Not Selected – Remove from List • SIGMA NOTICE J – Continuous Recruitment – On List

6.9 EXAM APPEALS

6-14

3/6/00

G 6.4.1 CONFIRMATION LETTER TO ORAL EXAM PANEL MEMBERS

Upon a person’s agreement to serve as an Oral Exam Panel member, a confirmation letter should be sent an adequate amount of time before the oral exam.

The content of the letter should include:

• Dates, starting and ending times of oral exam. • Location of oral exam. • Parking instructions. • Notice that lunch will be provided. • Notice that mileage will be reimbursed at the standard County rate. • Notice if over-night accommodations will be reimbursed. • Analyst name and phone number.

Enclosures to the letter should include:

• "A Guide to Oral Board Members" Pamphlet. (E 6.4.2) • List of Local Accommodations. (E 6.4.4) • Temporary Parking Permit (if applicable). • Map to Examination Site. (E 6.4.3) • Job Announcement and Supplemental Questions. • Class Specification.


Reference: Approved: 5/21/99

3/6/00

G 6.4.2 PREPARATION OF EXAM MATERIALS

The consistent presentation of exam materials assists both the raters and the Analyst in quickly identifying documents.

Two pocket presentation folders are used to hold all of a rater's exam materials. To enable any Personnel staff member to carry out their assigned responsibility in the Oral Exam process, the following standardized forms are utilized:

• Oral Exam Rating Form (blue paper); Exhibit 6.4.8 • Oral Exam Questions Format (yellow paper); Exhibit 6.4.9 • Oral Exam Panel Briefing Outline; Exhibit 6.4.10 • Candidate Information Sheet Format; Exhibit 6.4.11 • Oral Exam Schedule (green paper); Exhibit 6.4.12 & E 6.4.13 • Oral Exam Benchmarks (salmon paper). Exhibit 6.4.14

To ensure consistency in the presentation of our oral exam packets for panel members, the following setup is used:

Left side of folder contains

• Oral Exam Questions; • Candidate Information Sheet; • Oral Exam Benchmarks; • Job Announcement and Supplemental Questions; • Class Specification (if needed); • Examples of Good and Bad Behavioral Statements. (E 6.4.15)

Right side of folder contains:

• Rating Forms; • Oral Exam Schedule.

There is a space provided in the folder for the Analyst to place a business card for the panel members to keep.



3/6/00

G 6.4.3 ORAL EXAM APPEAL PROCEDURE

APPEAL OF EXAM QUESTIONS Appeals of individual exam questions must be filed with the Personnel Analyst immediately following the oral exam and before leaving the premises. Appeals of exam questions can not be accepted once a score has been assigned.

RESOLUTION OF APPEALED EXAM QUESTIONS The Personnel Analyst must review the appealed question with the subject matter experts and testing specialists and make a final determination on its retention or deletion. The appellant must receive written notification of the determination as soon as possible. There is no further appeal of this determination.

APPEAL OF EXAM PRACTICES Any appeal of an examination practice must be filed in writing with the Analyst immediately following the administration of the exam and before leaving the premises. Appeals of an examination practice will not be accepted once a score has been assigned.

RESOLUTION OF APPEAL OF EXAM PRACTICES The Analyst must provide the appellant with a written response, within seven (7) working days of receipt of the appeal. Appeals of the Analyst’s decision must be presented in writing to the Principal Personnel Analyst within five working days of receipt the Analyst’s response. The Principal Personnel Analyst will respond in writing, within seven (7) working days of receipt of the appeal. This decision may be appealed in writing to the Assistant CAO – Human Resources within seven days of receipt of the notice. The decision of the Assistant CAO – Human Resources shall be final.

NOTIFICATION OF CANDIDATES Appeal procedures must be mailed to the candidates before the oral exam is administered, along with the notice of the test date and location. Candidates must give their objections to test questions in writing on the Examination Appeal Form (E 6.4.17) provided by Human Resources, by the end of the day on which the test was administered.

CONTINUANCE OF EXAMINATION PROCESS The examination process will continue unless the applicant appeals the decision of the Personnel Analyst under the “Appeals of Exam Practices”. At that time, the process will be suspended until the appeal process is exhausted.



3/6/00

G 6.5.1 WRITTEN EXAMS - SETTING PASS-POINTS

In order to maximize the validity of cut-off scores for written multiple-choice/true-false examinations, and establish a consistent agency approach, an Analyst/Specialist should consider and document the applicable steps used in determining pass-point.

1. Chance Level: The minimum score that a candidate could achieve on the exam if they were to blindly guess at each item. This is calculated by multiplying the number of test items times a fraction representing the number of choices for each test item.

• Example: A multiple-choice exam with 90 total items; 57 items have four answer choices, 21 items have five answer choices, and 12 items have three answer choices. 57 ( ) + 21 ( ) + 12 ( ) = CHANCE LEVEL

2. Minimally Acceptable Candidate (MAC) Level: The minimum score that a minimally acceptable candidate could be expected to achieve on the exam. The minimally acceptable candidate is one who possesses the minimum acceptable level of competence to fulfill the job duties effectively. This score can be established using either the Angoff, Ebel, or Nedelsky methods, or using them in combination. Generally, the Angoff method is considered the more valid approach. All methods require the use of SME's.

• The pass-point should always be set at or above the MAC level. 1. Certified Group: The average score achieved by testing a representative mix of incumbents currently in the job class. This not only provides good information for pass-point setting but, provides an opportunity to "field test" the exam. The control group can provide excellent feedback regarding the test instructions, the test items, and the time allotment.

• Generally, the certified group will achieve higher exam scores than the MAC level because they have been doing the job.

2. Adverse Impact Analysis: This is a statistical analysis which measures the impact of potential pass-points on protected members of the applicant pool.

• You should always strive to eliminate or minimize adverse impact in setting your pass-points.

3. Item Analysis: This is a statistical analysis which identifies test items warranting additional evaluation.

• You should re-evaluate these items with your SME's to make sure they key is correct and that the item is not too ambiguous, or that there aren't more than one correct answer, or a partially correct answer, etc. This re-evaluation should be documented.

4. Statistical Cut-Off Analysis: This is an analysis of the basic test statistics. Each of the following statistics should be evaluated, although the degree to which each impacts your cut-off decision will vary by test.

• Mean • Reliability • Mode • Average Difficulty • Standard Deviation • Skewness • Difficulty • Kurtosis • Key-Total Correlation • Standard Error of Measure • Percent Correct • Item Difficulty


continued on following page

5. Speeded or Power Test: This is an evaluation of whether your test was designed so that not all applicants would finish (speeded test), or whether it was designed to test for job knowledge (power test). A speeded test can greatly skew your test statistics since not all applicants will finish.

6. Practical Considerations: These include the number of vacancies to be filled, difficulty of recruitment, anticipated life of the list, whether additional tests will be administered, whether you're using it as a pass/fail or for ranking, whether the list is generic or specialized, etc.

7. Prior Exams: This includes a review of any prior administration of the SAME exam. Things to evaluate include the prior pass-point, whether or not there was adverse impact, how long did the list last, a comparison of the test statistics, etc. SUMMARY There is no "right" way to set a pass-point. Pass-point setting is an art - not a science.



3/6/00

G 6.8.1 SUBDIVISION OF ELIGIBLE LISTS

Where the duties of a position or the efficiency of recruitment's require, Eligible Lists may be subdivided, or separate Eligible Lists established for various departments, divisions of department or sections of the County. Subdivision of an Eligible List is appropriate when the distinction between duties of different positions within the same classification and the requisition of different minimum qualifications are clearly noted on the job announcement. Throughout the exam process and upon establishment of the Eligible Lists, each applicant should receive notices that clearly explain their status with respect to each sub-list for which interest was expressed.



3/6/00

G 6.8.2 SIDE-BY-SIDE ELIGIBLE LIST

Where the duties of the position or compelling management needs require, Eligible Lists may be subdivided, or separate Eligible Lists established for the various departments, division or department or sections of the County. Establishment of a separate and apart Eligible List (side-by-side list), where there is an existing valid Eligible List, may occur when the duties of the position or compelling management needs require it. The Analyst must exercise professional judgment in determining that the creation of another Eligible List for the same classification is based on bona fide job-related qualifications, and that it is legally defensible, appropriate, and does not adversely affect any applicants on the original Eligible List.



3/6/00

G 6.8.3 MERGING OF CANDIDATES TO A NEW EXAM PROCESS

Candidates on an existing Eligible List that will be abolished prior to its expiration date may be merged with applicants in a new examination process up through the point in the process where the exam processes are identical. Depending on the examination process, it may be appropriate to merge current candidates with new applicants after the paper-screening stage, or after the oral, written or performance exam stages.

Candidates on existing Lists should be notified of their option to be considered for inclusion in the new examination process without the need to submit new application materials. Candidates must be informed of their option to re-test, and that the most recent test score will be used.

When merging candidates to a new process, applicants will be considered for advancement through the processes based on exam score. Merging of a candidate to a new exam process does not necessarily advance him/her to the next exam process or on to the Eligible List.



3/6/00

G 6.8.4 REMOVAL FROM ELIGIBLE LIST

Candidates may be removed from an Eligible List for the following:

• Fraudulent conduct or false statements by self, or others with his/her concurrence, on any application, examination or selection procedure.

• Failure to sign an “Authorization to Inquire, Verify and/or Release Information”. • Failure to meet any of the requirements or qualifications as published in the job

announcement. • Inability to be located by the U.S. Postal Service. • Receipt of a statement declining an interview and stating no further interest in employment

with Monterey County. • Failure to respond to an interest survey or a request to schedule a selection interview. • Appointment to a permanent position in the classification for which the Eligible List was

established. • Resignation from Monterey County employment of a candidate on a "promotional-only"

Eligible List. • Failure to participate or inability to qualify in a post-job offer medical examination,

evaluation or background investigation. • Conviction (including pleas of nolo contendre) of a felony or misdemeanor, if the Assistant

CAO – Human Resources determines a rational relationship to the position. In the cases of applicants or candidates for peace officer positions (as defined in Penal Code §830, et seq.), a felony conviction shall automatically disqualify the individual.

• Written request by candidate. • Declination of three offers of employment to a permanent position in the classification for

which the Eligible List was established. • Participation in three separate final selection processes for the same appointing authority

without being selected. (In this case, the candidate can be omitted from subsequent Referrals to the same appointing authority from the same Eligible List.)



3/6/00

G 6.8.5 TIE SCORES - ESTABLISHING RANK

Consistent reporting practices must be applied when the exam process results in tied scores and, therefore, tied ranks. All candidates with the same final score shall have the same rank. The rank on the original Eligible List is based on the number of individual candidates with higher scores, as in the following example:

FINAL SCORE

RANK

99

1

95

2

95

2

92

4

90

5

88

6

88

6

85

8

83

9

83

9

82

11



3/6/00

G 6.9.1 EXAM APPEALS

The Human Resources Division has established procedures for addressing concerns regarding any aspect of the examination process. Concerns regarding examinations must be presented to the Analyst prior to a score being assigned. The Analyst is expected to actively assist in the resolution of all appeals at the lowest possible level. Individuals filing appeals must be notified in writing of the determination of the appeal and any further appeal actions they may exercise.

APPEAL OF EXAM QUESTIONS Appeals of individual exam questions must be filed with the Analyst immediately following the administration of the examination. Appeals of exam questions cannot be accepted once a score has been assigned.

RESOLUTION OF APPEALED EXAM QUESTIONS The Analyst must review the appealed questions with the SME's and testing specialists and make a final determination on its retention or deletion. The appellant must receive written notification of the determination as soon as possible. There is no further appeal of this determination.

APPEAL OF EXAM PRACTICES Any appeal of an examination practice must be filed in writing with the Analyst immediately following the administration of the exam. Appeals of an examination practice will not be accepted once a score has been assigned.

RESOLUTION OF APPEAL OF EXAM PRACTICES The Analyst must provide the appellant with a written response within seven (7) working days of receipt of the appeal. Appeals of the Analyst's decision must be presented in writing to the Principal Personnel Analyst within five (5) working days of receipt of the Analyst's response. The Principal Personnel Analyst will respond in writing within seven (7) working days of the appeal. This decision may be appealed in writing to the Assistant CAO- Human Resources within seven (7) working days of receipt of the notice. The decision of the Assistant CAO - Human Resources shall be final.

NOTIFICATION OF CANDIDATES Appeal procedures will be mailed to the candidates before an exam is administered, along with the notice of the test date and location. Candidates must give their objections to test questions in writing on the Examination Appeal Form (E 6.9.1) provided by Human Resources by the end of the day on which the test was administered.



3/6/00

G 6.9.2 CONTINUANCE OF EXAMINATION PROCESS ON EXAM APPEALS

The examination process will continue unless the appellant appeals the decision of the Analyst under the "Appeal of Exam Practices" (G 6.9.1). At that time, the process will be suspended until the appeal process is exhausted.



3/6/00

G 6.9.3 COUNTY OF MONTEREY

EXAMINATION APPEAL FORM

Name:

Address:

Phone:

Exam Title:

Specific action being appealed:

Corrective action desired:

You will receive a written response from the Analyst with seven (7) working days, Thank You



3/6/00

G 6.9.4 DISCRIMINATION COMPLAINTS

In the course of doing business, Analysts may receive a written or verbal allegation or inference of discrimination or harassment from a county employee, member of the public, or union or legal representative.

In compliance with County Ordinance, as a representative of the Assistant CAO - Human Resources, an Employment Services Analyst hearing or receiving such an allegation is required to inform the party making the allegation of his/her rights. If the party feels that he/she is a victim of discrimination or harassment, he/she is informed of his/her right to:

Contact and/or file a complaint with;

• County Equal Opportunity Office (Provide the EO Officer's name and give, or offer to give, his/her address and phone number: P.O. Box 1877, Salinas CA 93902. Phone # (831) 755-5117

• Federal Equal Employment Opportunity Commission (EEOC) 96 N. 3rd Street, Suite 200 San Jose, CA 95112 Phone #: (408) 291-7352

• California Department of Fair Employment and Housing (DFEH) 888 N. 1st Street, #316 San Jose, CA 95112 Phone #: (408) 277-1264

In the event the Analyst receives notification of a complaint/request for information filed directly from the DFEH or EEOC:

1. Immediately inform the Monterey County Equal Opportunity Officer (EOO).

2. Keep photocopy of the correspondence.

3. Immediately send original correspondence to the EOO.

4. Begin gathering the documentation requested.

5. The EEO will inform you of any requirement to submit documentation/narrative and of the date information is due to the EOO.

6. The HRD will inform you of any items to which it will respond.



3/6/00

E 6.1.1 PANEL MEMBER SIGN-IN SHEET

Exam Number: Class Title: Exam Component: QA Oral Written Performance Panel Member Information: Name: Title: Agency: Address: Phone Number: Race/Ethnicity:

Asian/Pacific Islander Black Hispanic Filipino Native American White

Gender: Female Male Professional Background/Qualifications Relevant to this process:

I understand that all information relating to the applicants participating in this process, the particulars of this exam process and the proceedings today are confidential and agree to refrain from sharing any such information with any party other than the Analyst assigned to this process.

Signature: Date:

3/6/00

E 6.1.2 TEST MATERIAL REQUEST FORM

3/6/00

E 6.3.1 PROCEDURES FOR QUALIFICATIONS APPRAISAL

1. Pre-screen application.

2. Determine SME's as raters.

3. Brief raters.

4. Present paperwork as follows: • Panel Member Sign-In Sheet (E 6.1.1) • Screening Grid; (E 6.3.3) (names can be downloaded from SIGMA) • Benchmark Definitions - Qualifications Appraisal; (E 6.3.2) • Knowledge Level Definitions; (E 6.3.7) • Job Announcement; • Supplemental Questions; • Classification Specification; • Applications Materials as follows:

• Completed Application; • Resume; • Supplemental Questions Response;

5. Raters conduct Qualifications Appraisal.

6. Debrief raters.

7. Enter scores into SIGMA: average scores among raters, round score.

8. Determine applicants to advance in process.

9. Send notices to applicants.

10. Send thank you note to raters.

3/6/00

E 6.3.2 BENCHMARK DEFINITIONS - QUALIFICATIONS APPRAISAL

SUPERIOR (8-9)

WELL QUALIFIED (6-7)

ACCEPTABLE (4-5)

BELOW STANDARD (1-3)

Candidate possesses extensive, directly related experience. Candidate demonstrated an outstanding amount of the knowledges, skills or abilities required by the position. Candidate answers the questions clearly and concisely. Candidate could immediately perform the duties of this position.

Candidate demonstrated many of the knowledges, skills or abilities required in the position. This rating should typify an above average skill level. Answers to exam questions were accurate and covered key points. Candidate could perform the duties of the position with a minimal orientation period.

Candidate demonstrated enough of the knowledges, skills and abilities to acceptably perform the job. Candidate has had some experience applying the knowledges and performing the skills which demonstrate the ability to perform the job. Although responses lack depth, answers to exam questions were substantially correct and presented in a fairly logical and satisfactory manner. Candidate could perform the duties of the job within a normal orientation period.

Candidate possesses few or none of the knowledges and skills needed to perform the duties required and are clearly below what is minimally acceptable for this position. Candidate has had limited experience applying the knowledges and performing the skills which demonstrate the ability to perform the job. Candidate does not demonstrate the ability to perform the job. Candidate does not demonstrate a minimum acceptable level of competency and would require extensive orientation and/or training in the required knowledges and skills before being fully functional at this classification level. Candidate had difficulty understanding and responding to questions; answers were incomplete and difficult to follow.

3/6/00

E 6.3.3 MONTEREY COUNTY

POSITION TITLE SCREEN GRID

3/6/00

E 6.3.4 QA PANEL BRIEFING

GENERAL PROCESS • Introductions. • Thank the rater(s). • Confidentiality - no discussions of applicants outside of this room; only with HR Representative. • Debriefing. • Explain the role of the position/organization. • Those applicants who are basically qualified to perform the duties of the position will pass the

screening. This doesn't mean that they are the most qualified or that you would personally hire them for the position. Those who are not basically qualified, based upon a review of all application materials, should receive below standard scores.

RATING GRID • Read job announcement to determine knowledges, skills and abilities, identify levels of knowledges,

skills and/or abilities. • Read definitions of scores and knowledge levels. • Document to support rating/score; DESIRED for each question; REQUIRED for "Below Standards". • Documentation should describe specific reason for score given.

SCALE • Review rating scale. • Use entire scale. • Passing score: 4 - basically qualified to perform the job. • Failing score: anything below a 4 on any individual question will screen the applicant out of the

process.

RATING PITFALLS • Rate each question separately. • Comparing applicants to each other, instead of to the questions. • Keep applications in alpha order if possible. • Identify applicants known to raters; OBJECTIVITY IS ESSENTIAL. • Equal Opportunity Employer; rate without regard to ethnicity, race, age or gender. • All information reviewed and conversations regarding applicants is confidential and may not be

shared with anyone, including applicants or other raters after the completion of this screening process. • Rate independently; OK to compare scores after initial rating; we'll discuss those who receive widely

different scores. • Comments must be made to support all scores below 4. • All applicants who meet the minimum qualifications may not be invited to participate in the

examination process; this is a competitive screening process.

3/6/00

E 6.3.5 EXAM ACTION SHEET - QA

3/6/00

E 6.3.6 CHECKLIST FOR QUALIFICATIONS APPRAISAL

The following items are used in the Qualifications Appraisal:

• Briefing Outline; • Applicants' Application Materials (copies needed if screeners are not in same location); • Screening Grid; • Benchmarks; • Knowledge Levels Definitions; • Job Announcement; • Supplemental Questions; • Pencils; • Yellow Pads; • Panel Info Sheets.

3/6/00

E 6.3.7 COUNTY OF MONTEREY

KNOWLEDGE LEVEL DEFINITIONS

THOROUGH KNOWLEDGE

WORKING KNOWLEDGE

SOME KNOWLEDGE

Implies an advanced degree of knowledge and complete understanding of the subject. The work calls for sufficient comprehension of the subject to solve unusual and routine work problems, to advise on technical and policy questions and to plan work programs.

Implies sufficient knowledge of the most significant aspects of the subject to enable the employee to work effectively with little direct supervision in standard work situations that include deviations or irregularities.

Implies sufficient familiarity with the subject to know the elementary principles and terminology and to understand and solve simple work problems.

3/6/00

E 6.3.8 THANK YOU LETTER

QUALIFICATIONS APPRAISAL PANEL MEMBER

March 6, 2000

Santa Cruz Emergency Operations Center 495 Upper Park Rd. Santa Cruz, CA 95060 Dear Mike: Thank you very much for serving on the Qualifications Appraisal Panel for the position of Emergency Communications Manager. I appreciate the expertise and professional approach you brought to this process. The panel had a difficult and critical task in determining the candidates who meet the minimum qualifications for this position. The panel worked well together which made my job all that much easier and enjoyable. It was a pleasure working with you. I look forward to the possibility of having you on another of our panels in the future. Thank you again for taking the time and effort to assist us in this process. Sincerely, Name Personnel Analyst

3/6/00

E 6.4.1 ORAL DIMENSION LIST

3/6/00

E 6.4.2 A GUIDE FOR ORAL BOARD MEMBERS

3/6/00

E 6.4.3 MAP OF SALINAS SHOWING

PRINCIPAL MONTEREY COUNTY BUILDINGS

3/6/00

E 6.4.4 MOTELS IN THE SALINAS AND SURROUNDING AREAS

3/6/00

E 6.4.5 CHECKLIST FOR ORAL EXAMINATION

3/6/00

E 6.4.6 SAMPLE - PROCTOR SCRIPT

ADMINISTRATIVE SERVICES DIRECTOR PRE-EXAM EXERCISE

3/6/00

E 6.4.7 COUNTY OF MONTEREY SAMPLE - SCRIPT

PRE-EXAM WRITING EXERCISE

3/6/00

E 6.4.8 SAMPLE - ORAL EXAM PANEL RATING FORM

3/6/00

E 6.4.9 SAMPLE - ORAL EXAMINATION QUESTIONS

3/6/00

E 6.4.10 ORAL EXAMINATION PANEL BRIEFING

3/6/00

E 6.4.11 ORAL EXAMINATION CANDIDATE INFORMATION SHEET

3/6/00

E 6.4.12 COUNTY OF MONTEREY HUMAN RESOURCES DIVISION

EXAMINATION SCHEDULE

3/6/00

E 6.4.13 COUNTY OF MONTEREY PERSONNEL DIVISION

EXAMINATION SCHEDULE - 2 PHASE

3/6/00

E 6.4.14 BENCHMARK DEFINITIONS ORAL RATING

3/6/00

E 6.4.15 EXAMPLE OF GOOD VS. POOR BEHAVIORAL STATEMENTS

Behavioral statements are used to describe candidate responses or behaviors which demonstrate their possession of knowledges, abilities, and other minimum qualifications. Behavioral statements should be specific descriptions of the behavior demonstrated by an individual (what they said or what they did). They should NOT be a summation or conclusion as to why the candidate responded in a particular manner.

The examples below illustrate the difference between good behavioral statements which describe what was actually said or done, and vague or general summations or conclusions which are inconclusive.

Behavioral statements should be used on your rating sheets and can be used as justification for coding a candidate as unqualified on the Department Selection Report.

POOR EXAMPLE 1. Candidate identified several of the

issues in the problem but failed to identify some of the important ones.

2. Candidate demonstrated outstanding

communication skills.

3. Analysis lacking! Missed the point of the whole experience.

4. Candidate demonstrated very low skill in recognizing quality of performance and providing direction.

5. Candidate does not possess the minimum qualifications for this position.

6. Candidate lacks experience; (or) candidate lacks breadth and depth of experience.

7. Candidate did not answer the questions (or) candidate was unresponsive.

8. Candidate did not understand the

questions (or) candidate was slow in answering the questions.

9. Candidate is unqualified for this job.

10. Candidate would be overwhelmed in this job (or) candidate could not handle the pace of this position.

BETTER EXAMPLE 1. Candidate identified the issues of prioritizing backlog cases, accepting handwritten

work, notifying clients of backlog; but failed to mention the more critical factors such as reassigning cases, transferring employees from other units, approving overtime, and expediting an exam process to fill current vacancies.

2. Candidate maintained constant eye contact, pronounced words clearly, paused before responding, explained acronyms used and used hand motions to emphasize points.

3. Candidate discussed the problem briefly and only referred to one of the plans under consideration. Did not provide a recommendation as instructed in the exercise.

4. After asking supervisor (role player) to reinvestigate complaint, candidate did not set due date and did not give any specific instructions regarding what he wanted the supervisor to do.

5. Candidate did not describe any experience with inclement weather road closures and was not able to tell the panel the appropriate procedure for removing a tree which has fallen across a road. These are essential responsibilities for this position and cannot be learned in a short orientation period on the job.

6. Candidate described experience in the inspection of restaurants and hotels. Candidate did not describe experience in inspecting water wells or water systems. Although some of the candidate's restaurant inspection experience is applicable, these are specialized types of inspections performed by this position which require the application of additional complex regulations and procedures which cannot be learned in a short orientation period on the job.

7. Candidate did not give complete, detailed answers to the questions and did not elaborate upon follow-up. This would be problematic in that as a part of this job, the employee must explain, fully and completely, complicated procedures to the public at the counter.

8. We had to ask the questions several times before candidate would answer completely. This is problematic in that the employee in this job is required to respond quickly to instructions from doctors and nurses in a busy clinic.

9. Candidate did not describe any job related experience or training that demonstrated a knowledge of medical terminology. This job requires a working knowledge of medical terminology and some familiarity with common medical procedures used in a medical clinic. These knowledges cannot be learned in a short orientation period on the job.

10. Candidate did not describe any experience working on a job where she had to do more than one thing at a time, had difficulty responding to compound questions, and hesitated and said "I think so" when asked if she could work in a large, busy office taking numerous phone calls, waiting patients at the counter, pulling charts, and completing paperwork. This is a very busy clinic office and these employees are typically doing more than one thing at a time, e.g., pulling charts, answering phones, and waiting phones, and waiting on patients at the counter.

3/6/00

E 6.4.16 SUGGESTIONS FOR TRAINING INTERVIEW PANELISTS

3/6/00


ORAL EXAMINATION QUESTION APPEAL FORM

Name:

Address:

Phone:

Exam Title:



You will receive a written response from the Personnel Analyst within seven (7) days. Thank you.

3/6/00

E 6.4.18 EXAM ACTION SHEET - ORAL

3/6/00

E 6.4.19 ORAL BOARD CONFIRMATION LETTER

3/6/00

E 6.4.20 THANK YOU ORAL PANEL MEMBERS

3/6/00


ORAL EXAMINATION PROCESS APPEAL FORM

Name:

Address:

Phone:

Exam Title:



You will receive a written response from the Personnel Analyst with seven (7) working days, Thank You.

3/6/00

E 6.5.1 EXAM ACTION SHEET - WRITTEN

3/6/00

E 6.5.2 TEST PROCTORING STANDARDS

3/6/00

E 6.5.3 CHECKLIST FOR WRITTEN EXAMS

3/6/00


WRITTEN EXAM QUESTIONS APPEAL FORM

Name:

Address:

Phone:

Exam Title:

Question # Reason for appeal:





You will be notified in writing of the decision made on each of the questions being appealed.

3/6/00


WRITTEN EXAMINATION PROCESS APPEAL FORM

Name:

Address:

Phone:

Exam Title:




3/6/00

E 6.6.1 SAMPLE - ROAD MAINTENANCE HELPER

INSTRUCTION FOR MAIN RECEPTION AREA

3/6/00

E 6.6.2 SAMPLE - PERFORMANCE EXAM

WAIVER OF RESPONSIBILITY

3/6/00


PERFORMANCE EXAM APPEAL FORM

Name:

Address:

Phone:

Exam Title:






You will be notified in writing of the decision made on each of the questions being appealed.

3/6/00

FOR HUMAN RESOURCE USE ONLY

APPROVED DENIED

Comments: By: Date:


AMERICANS WITH DISABILITY ACT (ADA) TESTING ACCOMMODATION REQUEST FORM

Completion of this form is needed to consider your request for testing accommodation. This form will be filed separately from your application. You may use this form to request any special testing arrangements or accommodations which may be necessary as a result of your disability. If you have any questions regarding this form, please call (831) 755-5116 and ask to speak to this Personnel Analyst assigned to the examination.

NAME (PRINT):

EXAM TITLE & NUMBER:

DAY TIME PHONE NUMBER:

Please describe the accommodation you are requesting in the testing process and the reason for your request:

If applicable, who may we contact in providing special services?

Name & Title:

Agency: Phone Number:

Please provide the name of a doctor, agency official or other individual who may be contacted for verification of disability.

Name & Title:

Agency: Phone Number:

I am an applicant for the position named above and may require "reasonable accommodation" in the testing process. I hereby request that the Personnel Division contact me regarding reasonable accommodation in the testing process and authorize the Personnel Division to verify the information on this form.

(Applicant Signature) (Date)

3/6/00

E 6.8.1 SCREENED OUT LETTER

3/6/00

E 6.8.2 PASSED PAPER - ON LIST

3/6/00

E 6.8.3 INVITED TO ORAL EXAM

3/6/00

E 6.8.4 INVITED TO WRITTEN EXAM

3/6/00

E 6.8.5 PASSED EXAM - ON LIST

3/6/00

E 6.8.6 FAILED EXAM

3/6/00

E 6.8.7 LIST ABOLISHED

3/6/00

E 6.8.8 NOT SELECTED - RETURN TO LIST

3/6/00

E 6.8.9 NOT SELECTED - REMOVE FROM LIST

3/6/00

E 6.8.10 CONTINUOUS RECRUITMENT - ON LIST

3/6/00

E 6.8.11 EXAM ACTION SHEET - ELIGIBLE LIST

3/6/00


EXAMINATION APPEAL FORM Name:

Address:

Phone:

Exam Title:




Documents

TABLE OF CONTENTS - Monterey County, CA · sample - oral examination questions ... suggestions for training interview panelists ... sample - road maintenance helper instruction for