JUDGE TRAINING The Judge’s Job is to: Choose the Best Encourage the Rest

JUDGE TRAININGJUDGE TRAINING

The Judge’s Job is to:The Judge’s Job is to:

Choose the BestChoose the Best

Encourage the RestEncourage the Rest

Why Do A Science Project?Why Do A Science Project?

►Valuable real life lesson:Valuable real life lesson: Make a plan & budget time Make a plan & budget time (learn project skills)(learn project skills)

Background ResearchBackground Research Gather MaterialsGather Materials Execute PlanExecute Plan Evaluate ResultsEvaluate Results Present the ResultsPresent the Results Academic “cross training” using multiple Academic “cross training” using multiple

disciplinesdisciplines

What Does A Science Display What Does A Science Display Look Like?Look Like?

►Display BoardDisplay Board►Data Logbook/JournalData Logbook/Journal►Abstract, Problem/Hypothesis, Abstract, Problem/Hypothesis,

Procedure, Results, ConclusionProcedure, Results, Conclusion►SRC Forms: form 1C & form 7 SRC Forms: form 1C & form 7 if needed if needed

must be vertically displayed at exhibit; the rest of the must be vertically displayed at exhibit; the rest of the forms must be available at the exhibit for inspection. forms must be available at the exhibit for inspection. All projects have already been checked for compliance All projects have already been checked for compliance with international rules. with international rules.

Sample BoardSample Board









How to StartHow to Start

► Pull out your score sheet for the project. Pull out your score sheet for the project. It should It should be already labeled with your judge number & the project number, be already labeled with your judge number & the project number, title & student(s) name. You also have a comment sheet checklist. title & student(s) name. You also have a comment sheet checklist. DO NOT tell students their scores nor give them score sheets. DO DO NOT tell students their scores nor give them score sheets. DO NOT give students their comment sheets! Turn both comment & NOT give students their comment sheets! Turn both comment & score sheets in to the Judging Area for all projects.score sheets in to the Judging Area for all projects.

► Ask the student: “Tell me about your project.” Ask the student: “Tell me about your project.” or “What did you do?” LISTEN!or “What did you do?” LISTEN!

► Let the student tell their story and ask Let the student tell their story and ask questions.questions.

Two Types of JudgesTwo Types of Judges

► We have two types of judges on the show floor We have two types of judges on the show floor during judging:during judging: Ribbon judges: Ribbon judges: You are the judges that score the projects. You are You are the judges that score the projects. You are

the heart & soul of this competition! Students will be awarded a green, red, the heart & soul of this competition! Students will be awarded a green, red, or blue ribbon based on their scores. The projects selected to compete at or blue ribbon based on their scores. The projects selected to compete at state will receive a purple ribbon in addition to their blue ribbon. Some state will receive a purple ribbon in addition to their blue ribbon. Some other awards are also based on score, such as top team project in the other awards are also based on score, such as top team project in the grade.grade.

Special Awards judges:Special Awards judges: You are the judges representing You are the judges representing organizations who have provided awards with special criteria. You choose organizations who have provided awards with special criteria. You choose those receiving the “goodies” such as cash, trophies, medals, certificates, those receiving the “goodies” such as cash, trophies, medals, certificates, shirts, further competitions, scholarships, or other opportunities. Be sure to shirts, further competitions, scholarships, or other opportunities. Be sure to select projects that match your criteria when awarding this kind of award.select projects that match your criteria when awarding this kind of award.

Judging the Board / ProjectJudging the Board / Projectfor those scoring projectsfor those scoring projects

►8-12 minutes 8-12 minutes ►Display quality should not impact score Display quality should not impact score

for science knowledge and experimental for science knowledge and experimental designdesign

►Beware of the slick salespersonBeware of the slick salesperson►Focus on the scienceFocus on the science►The presenter can do amazing things – The presenter can do amazing things –

trust what they say to you,trust what they say to you, but ask but ask questions!questions!

Judging Score sheetJudging Score sheet► Optional:Optional: make notes if needed on the score sheet (things done make notes if needed on the score sheet (things done

well, errors in the science or analysis, missing pieces) – students well, errors in the science or analysis, missing pieces) – students do do

notnot get this sheet! Notes are for your benefit only. get this sheet! Notes are for your benefit only. Your job is to Your job is to choose the best in ranking order by the rubric’s choose the best in ranking order by the rubric’s standards!standards!

► Data entry people WILL NOT add or re-add your Data entry people WILL NOT add or re-add your scores.scores. If your scores do not add correctly, the student If your scores do not add correctly, the student will receive the will receive the bottom line scorebottom line score you wrote, NOT the sum you wrote, NOT the sum of the parts. If you changed your bottom line total without of the parts. If you changed your bottom line total without changing the parts, draw a line through the parts and circle changing the parts, draw a line through the parts and circle the correct total so there is no confusion which is correct.the correct total so there is no confusion which is correct.

► Use the grading rubric to value each part correctly. Use the grading rubric to value each part correctly.

The Roles Of All JudgesThe Roles Of All JudgesYou are a…You are a…

►Role ModelRole Model►EvaluatorEvaluator►FacilitatorFacilitator►MotivatorMotivator►CounselorCounselor►Fellow Scientist, interested in the Fellow Scientist, interested in the

student’s workstudent’s work

Judge ExperienceJudge Experience

► Be genuineBe genuine► Let the student show their stuffLet the student show their stuff► Encourage conversation – Encourage conversation – Ask questionsAsk questions► Avoid value judgmentsAvoid value judgments► Complement the student on things done well!Complement the student on things done well!► Give one opportunity for improvementGive one opportunity for improvement► End meeting on a positive noteEnd meeting on a positive note► SmileSmile

Judge BehaviorJudge Behavior

► Put students at easePut students at ease► Show you are interestedShow you are interested► Listen activelyListen actively► Ask students enough questions to satisfy Ask students enough questions to satisfy

yourself that they understood & did the project yourself that they understood & did the project (team projects must show that all members (team projects must show that all members

contributed expertise to the project!)contributed expertise to the project!)► When you have reached the student’s When you have reached the student’s

knowledge limit, STOP asking questionsknowledge limit, STOP asking questions► Remember when you were 12-16 years old?Remember when you were 12-16 years old?

Sample QuestionsSample Questions

►Where did you get the idea for your Where did you get the idea for your project?project?

►What is the purpose of your study?What is the purpose of your study?►What background research did you do? What background research did you do?

What have others done before?What have others done before?►Why is this project important to you? Why is this project important to you?

(OR Why are you interested in this?) …(OR Why are you interested in this?) …

More Sample QuestionsMore Sample Questions

►Explain your procedure. (What did you Explain your procedure. (What did you do? Why did you do that? How does that do? Why did you do that? How does that work?)work?)

►What is your control? Variables?What is your control? Variables?►What instruments did you use for What instruments did you use for

measurements? …measurements? …


►On what did you base your conclusions? On what did you base your conclusions? ►Did you repeat your tests? How many? Did you repeat your tests? How many?

Any statistics? Why did you choose that Any statistics? Why did you choose that method of analysis or statistic?method of analysis or statistic?

►What are your sources of error?What are your sources of error?►How did you calculate (that) or how did How did you calculate (that) or how did

you make that graph? …you make that graph? …


►Do your results indicate further study is Do your results indicate further study is needed? needed?

►Are there other approaches you might Are there other approaches you might have taken to your research? have taken to your research?

►Of what value is your project to society? Of what value is your project to society? (Practical Application) …(Practical Application) …


►What problems arose during your What problems arose during your investigation?investigation?

►How did you overcome problems?How did you overcome problems?►What did you come up with that was What did you come up with that was

new, unique, or creative in this project? new, unique, or creative in this project? ……


►How much time did your study take?How much time did your study take?►What would you do differently?What would you do differently?►What would you do next (Future Work)?What would you do next (Future Work)?►What did you enjoy most about your What did you enjoy most about your

project? project?

Also Consider…Also Consider…

►Knowledge base developedKnowledge base developed►Experimental or Engineering Design – Experimental or Engineering Design –

(follow Scientific Method OR show new (follow Scientific Method OR show new theory or design)theory or design)

►Uniqueness/CreativityUniqueness/Creativity►Hard Work Hard Work ►EnthusiasmEnthusiasm►NOTE: It is NOTE: It is completely acceptablecompletely acceptable to to

DISPROVEDISPROVE your hypothesis! your hypothesis!

Judge TipsJudge Tips► Student understanding & scientific thought Student understanding & scientific thought

may be more important than actual results.may be more important than actual results.► Don’t tally scores in front of students.Don’t tally scores in front of students.► Don’t verbally criticize or degrade projects. (A Don’t verbally criticize or degrade projects. (A

LOT OF EARS ARE LISTENING!)LOT OF EARS ARE LISTENING!)► How much of the project is the student’s How much of the project is the student’s

versus the mentor or parent? (Believe the versus the mentor or parent? (Believe the student. Consider asking them to explain how student. Consider asking them to explain how they came up with the idea(s) if you think they they came up with the idea(s) if you think they were “lab rats” with too much outside help.)were “lab rats” with too much outside help.)

More Judging TipsMore Judging Tips► If stuck on project, ask for helpIf stuck on project, ask for help► Don’t judge a project if you have a conflict of Don’t judge a project if you have a conflict of

interest (related or friends)interest (related or friends)► Note that Teams should all contribute – and show Note that Teams should all contribute – and show

what each team member did for the projectwhat each team member did for the project► Please take just 2 minutes to give the student some Please take just 2 minutes to give the student some

feedback on the feedback on the Improvements & Comments check Improvements & Comments check sheetsheet. This is the ONLY paper the student will . This is the ONLY paper the student will receive back. Be constructive and encouraging. receive back. Be constructive and encouraging. Turn in comments & score sheets to Judging Area. Turn in comments & score sheets to Judging Area. DO NOT GIVE ANY papers or verbal scores to the DO NOT GIVE ANY papers or verbal scores to the students!students!

► Help yourself to dinnerHelp yourself to dinner

► Look over all the projects in your group to Look over all the projects in your group to

get a feel for what to expect.get a feel for what to expect.► Keep on schedule with your judging. Keep on schedule with your judging.

Budget your time Budget your time (10 minutes for many projects, (10 minutes for many projects, and 15 min max judging for top high school projects & and 15 min max judging for top high school projects & 5 min to write scores, write comments & find next 5 min to write scores, write comments & find next project).project). Please stay on schedule. Get your numeric Please stay on schedule. Get your numeric scores in as soon as possible for data entry!scores in as soon as possible for data entry!

► Make sure the best projects receive the best Make sure the best projects receive the best score (according to the criteria given).score (according to the criteria given).

► Most importantly: Most importantly: Have FUN!Have FUN!

How is the project ranking How is the project ranking determined using the scores?determined using the scores?► Each judge needs to score at least 6 projects (8 Each judge needs to score at least 6 projects (8

projects preferred) within a division: high school projects preferred) within a division: high school (grade 9-12) or middle school (grade 6-8). We (grade 9-12) or middle school (grade 6-8). We calculate average score for the project & rank by calculate average score for the project & rank by average score.average score.

► We calculate average Z-score for the project (Z-We calculate average Z-score for the project (Z-score compares how you ranked the project in the score compares how you ranked the project in the group of projects you judged – see next slide for group of projects you judged – see next slide for more.)more.)

► We calculate a normalized blend of average and We calculate a normalized blend of average and Z-score. WHY??? … Z-score. WHY??? …

Scoring UsedScoring Used► Average alone does not tell the entire story because some Average alone does not tell the entire story because some

judges score higher & some lower than others.judges score higher & some lower than others.► Z-score alone at regional does not tell the entire story either Z-score alone at regional does not tell the entire story either

because the method ASSUMES that you are judging a because the method ASSUMES that you are judging a normal distribution of projects. If you happen to get all top of normal distribution of projects. If you happen to get all top of the line projects… these score poorly (near your average) the line projects… these score poorly (near your average) with Z-score. That’s not fair, either!with Z-score. That’s not fair, either!

► So for TCRSF, we rank by average, rank by Z-score, then So for TCRSF, we rank by average, rank by Z-score, then create a blend score which is part average score & part create a blend score which is part average score & part normalized Z-score. Then we analyze for outliers and look at normalized Z-score. Then we analyze for outliers and look at all 3 rankings to be sure that we have indeed chosen the best all 3 rankings to be sure that we have indeed chosen the best based on your scores. If results are not clear, we look at how based on your scores. If results are not clear, we look at how a specific project ranked within each judge’s group they a specific project ranked within each judge’s group they judges and make a determination.judges and make a determination.

Z-Score CalculationZ-Score Calculation

► m = m = x / n x / n calculate mean (m) of judge’s set of scores: mean (m) calculate mean (m) of judge’s set of scores: mean (m) = sum of scores (= sum of scores (x)/number of projects scored (n)x)/number of projects scored (n)

► NOTE: We separate middle school & high school score calculations, so NOTE: We separate middle school & high school score calculations, so you must score at least 6 (8 is the preferred minimum # of scores) in each you must score at least 6 (8 is the preferred minimum # of scores) in each division if you judge in both divisions!division if you judge in both divisions!

► S = √ (n(S = √ (n(xx22) – () – (x)x)22) / (n(n-1)) ) / (n(n-1)) calculate the standard calculate the standard deviation (s) of judge’s set of scores (deviation (s) of judge’s set of scores (x = sum of scores; n = number of x = sum of scores; n = number of projects scored) projects scored)

► Z = (x – Z = (x – ) / s ) / s calculate the Z-score (Z) for each project scored by calculate the Z-score (Z) for each project scored by the judge (x = project score; m = mean of judge’s set of scores; s = the judge (x = project score; m = mean of judge’s set of scores; s = standard deviation of judge’s set of scores) standard deviation of judge’s set of scores)

All “Z-scores” from judges for each project will be averaged; most will rank between –1.5 and +1.5, with 0 representing the average/mean score.

SRC PaperworkSRC Paperwork► All SRC paperwork has already been checked by the TCRSF All SRC paperwork has already been checked by the TCRSF

Scientific Review Committee. You do not need to check the Scientific Review Committee. You do not need to check the forms. The students do not have to display all their forms, but forms. The students do not have to display all their forms, but must have them available for you to see them if you desire. must have them available for you to see them if you desire. Form 1C & 7 must be displayed vertically if required for the Form 1C & 7 must be displayed vertically if required for the project.project.

► All projects must have a Form 1, Form 1A, Complete Research All projects must have a Form 1, Form 1A, Complete Research Plan (Problem/Question, Hypothesis, Detailed Procedure, & Plan (Problem/Question, Hypothesis, Detailed Procedure, & Bibliography of a minimum of 5 references), Form 1B, and a Bibliography of a minimum of 5 references), Form 1B, and a printed abstract. The abstract does printed abstract. The abstract does notnot have to be on the have to be on the official ISEF form. Additional forms are required for projects official ISEF form. Additional forms are required for projects including humans, vertebrates, bacteria, tissues, body fluids, including humans, vertebrates, bacteria, tissues, body fluids, hazardous devices or materials, projects done in an outside lab, hazardous devices or materials, projects done in an outside lab, or continuation projects. Judges do not need to check these, but or continuation projects. Judges do not need to check these, but may look at them if the judge desires. may look at them if the judge desires.

Remember…Remember…

Your job is to:Your job is to:

Choose the BestChoose the Best

Encourage the RestEncourage the Rest

Thank you for encouraging these Thank you for encouraging these students in science/engineering!students in science/engineering!

Documents

JUDGE TRAINING The Judge’s Job is to: Choose the Best Encourage the Rest