SECTION 38: NIH TOOLBOX (ABBREVIATED USER’S MANUAL) · 38.4.1 Create a new participant or select an existing participant Step 1: Once the app is launched, click on the blue “Participants”

CKiD Manual of Procedures (11/01/18)

Page 38-1 of 35

CHRONIC KIDNEY DISEASE IN CHILDREN COHORT STUDY

SECTION 38: NIH TOOLBOX

(ABBREVIATED USER’S MANUAL)

38.1 OVERVIEW

This section describes the procedures to collect NIH Toolbox data. The NIH Toolbox provides a standard

set of comprehensive assessment tools used to assess cognitive and emotional function. The NIH Toolbox

data will be collected starting at V1b and every other year (i.e., V3, V5, V7, V9…). Data will also be

collected at post renal replacement therapy (post RRT) visits (see section 40 for details).

The NIH Toolbox was designed to be user-friendly and easy to administer with practice. It can be

administered by a study coordinator (after training is completed) or by other staff trained to administer

standardized cognitive tests. Training to administer the NIH Toolbox involves 4 basic steps:

1) Completion of an e-learning module:

http://www.healthmeasures.net/NIH_Toolbox_iPad_e-learning/story_html5.html.

This is an online training curriculum (including didactics, video demonstrations, and quizzes)

that can be utilized at the trainee’s convenience. The NIH Toolbox section as well as the

Cognitive and Emotion domain sections should be reviewed.

2) Listen to the 60 minute NIH Toolbox Webinar located on the CKiD website.

3) Practice iPad set up and the administration of assessments.

4) Once IRB is completed, participate in follow-up monthly phone calls until after completion of

the first few administrations to CKiD participants, or until comfortable with the administration

procedures.

38. 2 ESTIMATED TIME REQUIRED

The amount of direct time to complete testing will vary depending on the age, behavioral status, and health

of the participant. On average, the amount of time for test administration will range between approximately

20 to 40 minutes. In addition, it is estimated that approximately 30 minutes will be required to set-up the

iPad prior to test administration and thereafter, to include data scoring, exporting, recording, and

summarization of findings for the medical chart and parent feedback report.

NIH TOOLBOX STRUCTURE

To better understand the NIH Toolbox structure, there are certain terms the user needs to understand. The

terms listed below are used throughout this section.

Domain – An area of interest. The NIH Toolbox spans four domains or broad areas of health and

function. For the CKiD study, we will use two of the domains that include Cognition and Emotion.

Battery – A set or series of tests grouped together.

Instrument – A single test or measure that evaluates a particular construct. There are several

instruments included in a battery. The NIH Toolbox uses the terms “instrument,” “measures,” and

“tests” interchangeably.

Item – A question or task within an instrument or test. An instrument is comprised of items.

Assessment – A group of instruments given to a participant.

http://www.healthmeasures.net/NIH_Toolbox_iPad_e-learning/story_html5.html


Page 38-2 of 35

38.3 DESCRIPTION OF NIH TOOLBOX COGNITION AND EMOTION BATTERIES

The NIH Toolbox Cognition Battery is a multidimensional set of brief measures assessing cognitive

function for participants age 3 and older. The cognition components of the NIH Toolbox will measure:

Executive Function

Attention

Episodic Memory

Language

Processing Speed

Working Memory

The following NIH Toolbox Cognition Battery tests will be conducted for participants, age 3 – 6 years:

Picture Vocabulary Test

Flanker Inhibitory Control and Attention Test

Dimensional Change Card Sort Test (DCCS)

Picture Sequence Memory Test

The following NIH Toolbox Cognition Battery Tests will be conducted for participants who are 7 years and

older:

Picture Vocabulary Test

Flanker Inhibitory Control and Attention Test

Working Memory Test

Dimensional Change Card Sort Test (DCCS)

Pattern Comparison Processing Speed Test

Picture Sequence Memory Test

Oral Reading Recognition Test

The diagram below depicts the age ranges to which each of the tests is applicable.


Page 38-3 of 35

The NIH Toolbox Emotion Battery is an assessment of emotional functioning in children age 3 years and

older. The Emotion Battery measures four subdomains of functioning:

Psychological Well Being

Social Relationships

Stress and Self Efficacy

Negative Affect

The Psychological Well Being subdomain is comprised of assessments of Positive Affect, General Life

Satisfaction, and Meaning and Purpose.

The Social Relationships subdomain is comprised of assessments of Social Support, Companionship, Social

Distress, and Positive Social Development.

The Stress and Self-Efficacy subdomain is comprised of assessments of Perceived Stress and Self-Efficacy.

The Negative Affect subdomain is comprised of assessments of Anger, Fear, and Sadness

The NIH Toolbox Emotion Battery has the following formats:

Parent Proxy Emotion Battery – parents of 3 to 7 years old

Parent Proxy Emotion Battery- parents of 8 to 12 year olds

Self-report Emotion Battery – 8 to 12 years

Self-report Emotion Battery- 13 to 17 year olds

Self-report Emotion Battery-18 year olds and older

The diagram below depicts the age ranges to which each of the tests is applicable.


Page 38-4 of 35

Table 38.3a NIH Toolbox Cognition and Emotion Tests for CKiD by Participant Age

PARTICIPANT AGE TESTS Estimated Assessment Time

for Child or Young Adult

Estimated Assessment

Time for Parent

> 3 years and < 7 years Picture Vocabulary

Flanker Inhibitor Control and Attention

DCCS

Picture Sequence Memory

Parent Proxy Emotion

4 min

3 min

4 min

7 min

10 min



Listing Sorting Working Memory

DCCS

Pattern Comparison Processing Speed


Oral Reading Recognition


4 min

3 min

7 min

4 min

3 min

7 min

3 min

10 min




DCCS





Self-report Emotion

4 min

3 min

7 min

4 min

3 min

7 min

3 min

12 min

10 min

> 12 years and older Picture Vocabulary



DCCS




Self-report Emotion

4 min

3 min

7 min

4 min

3 min

7 min

3 min

12 mins

EQUIPMENT AND MATERIALS NEEDED FOR THE NIH TOOLBOX

Sites should ensure they have the necessary equipment prior to training or testing. The CCC will provide

the following equipment and materials.

iPad with case

Bluetooth wireless keyboard

Home Base – this is used as a place for each participant to place his or her dominant-hand index

finger during tests that measure the participant’s reaction time. The Home Base is used only for

specific tests and should be removed when not designated for other tests.

NIH Toolbox List Sorting Working Memory Test Examiner Answer Sheet

Pronunciation Guide for NIH Toolbox Oral Reading Recognition Test


Page 38-5 of 35

38.4 STEPS TO SET-UP THE iPAD (Steps must be performed prior to administering Toolbox)

38.4.1 Create a new participant or select an existing participant

Step 1: Once the app is launched, click on the blue “Participants” button.

Step 2: If the participant is completing the NIH Toolbox for the first time, then tap “Add New

Participant” in the upper right corner of the Participant screen and go to Step 3.

If participant completed the NIH toolbox at a previous visit, then select the appropriate KID ID from the

list of participants and skip Step 3.


Page 38-6 of 35

Step 3: Complete the new participant’s information: MUST be entered and updated PRIOR to each test

administration.

38.4.2 Steps to assign test batteries

Step 1: Tap on “Add New Assessment”

Identifier – Enter participant’s KID ID Number with no dashes

Example: 777077)

Name – Enter your CKiD Study site number

Example: Enter “77” for site 77

Gender

Birthdate – do not enter the date of birth, leave blank

Race

Ethnicity

Age – enter age of the participant at the time of testing.

Handedness - If unknown the screen will prompt you to answer the following questions. Handedness will be determined by the side indicated for 2 out of the 3 questions below:

Which foot do you use to kick a ball?

Which hand do you use to pick up and throw a ball?

Which hand do you use to write or draw?

Highest Education Level Completed

Mother’s Education – If unknown, the screen will prompt you to record the Father’s education. If that is unknown it will prompt you to record the Legal Guardian’s education.


Page 38-7 of 35

Step 2: Edit the name of the assessment.

The assessment should be renamed based on the visit number (V1b, V3, V5…) and type of assessment

(Cognitive or Emotion). Example of assessments for V1b: V1b Cognitive Assessment, and V1b Emotion

Assessment

To edit the name of the assessment, tap “Assessment” and edit the name using the iPad pop up keyboard or wireless

keyboard.

Once the Assessments are renamed, tap on the “Add Instruments” button to load the age appropriate test battery.


Page 38-8 of 35

Step 3: Select the specific age appropriate test battery.

Under “Available Batteries”, choose a battery by tapping “Select” next to the battery’s name. A battery will

include all the individual test instruments which should be administered for the specified age category. Select one

Cognitive Battery which corresponds to the age of the participant. Based on the participant’s age, the age specific

Emotion Battery may include questions for the participant and/or questions for the parent (Parent Report Battery) to

complete (see table 38.4a). Depending on the age of the participant you will have either one or two Emotion

Batteries as a part of your assessment.

You may filter by different categories to easily find age specific Cognitive or Emotion Battery or Instrument. For example, under Available Batteries tap ALL Ages and select the age range of your participant. This will narrow the choices of batteries that will match the participant’s age.

Select age appropriate Battery

(see Table 38.4a for complete

list of Cognitive and Emotion

Batteries by age listed).


Page 38-9 of 35

Table 38.4a Age of Participant by Test (The table below is a complete list of tests by age)

AGE of Participant

Cognitive Battery Emotional Test Battery Parent

Emotional Test Battery Participant

AGE 3 - 4 Early Childhood Cognition Battery (Ages 3 – 4)

Emotion Parent Report Battery (Ages 3 – 7)

AGE 5 - 6 Early Childhood Cognition Battery (Ages 5 – 6)

Emotion Parent Report Battery (Ages 3 – 7)

AGE 7 Cognition Battery (Age 7) Emotion Parent Report Battery (Ages 3 – 7)

AGE 8 - 11 Cognition Battery (Age 8 - 11) Emotion Parent Report Battery (Ages 8 – 12)

Emotion Battery (Ages 8 – 12)

AGE 12 Cognition Battery (Ages 12 +) Emotion Parent Report Battery (Ages 8 – 12)

Emotion Battery (Ages 8 – 12)

AGE 13 - 17 Cognition Battery (Ages 12 +) Emotion Battery (Ages 13 – 17)

AGE 18 + Cognition Battery (Ages 12 +) Emotion Battery (Ages 18 +)

Once the battery is selected, it will be listed under the “Selected Instruments” at the upper part of the

screen. If an incorrect battery is selected, you can drag and drop the battery under the “available batteries

and profiles” by using the “▬” symbol to the right of the instrument name.


Page 38-10 of 35

Step 4: When finished assigning the battery, tap “Done” in the upper right corner of the screen.

If you are not finished, tap on the “< “arrow to save your selections and to return to the previous screen.

Depending on the age of the child, you may have 2 Emotion Batteries listed under the Emotion Assessment (one for

the parent and one for the child to complete). The order which you administer the batteries can be changed by

tapping on “Edit Instrument” located under the “start” button.

IMPORTANT:

Please note that once you begin an assessment, the order in which the batteries are administered cannot be

changed. We recommend that you set up separate assessments for the Cognitive Battery and Emotion

Battery so that you have the flexibility to pause one assessment and begin the next if needed.

999099

999099 Site 99


Page 38-11 of 35

38.5 STARTING AN ASSESSMENT

Upon completing instrument selection, tap “Start” to launch the assessment.

Next, the assessment launch screen

will appear. When the participant is

ready to start the assessment, tap on

“Start”. During the cognitive testing

you will be seated next to the

participant at a table and you will be

administering the tests. As the

emotion assessment consists of

answering a series of multiple choice

questions, you may give the iPad to

the participant and/or guardian to

complete at their own pace.

Please inform the participant and/or guardian that if they would like to pause, skip or stop the assessment, they

should return the iPad to the test administrator or coordinator (see Section 38.6 for instructions on pausing, skipping

or stopping an assessment).

38.5.1 Error and Warning Messages

Inform the participant that while completing the assessment, they may see “Error” and “Warning” messages.

Inform the participant that if an “Error” or “Warning” message appears, they should return the device to the test

administrator or coordinator.

Error messages inform users about problems that prevent them from launching the assessment. If the

message appears, go back to the New Assessment screen (via Participants) and fix the problem(s). All

problems must be fixed before the assessment can be started.

Warning messages inform users about missing information or characteristics of the assessment that users

may want to revise before starting. It is possible either to go back to the New Assessment screen and

change them or to launch the assessment without making changes.

999099

999099 Site 99


Page 38-12 of 35

38.6 PAUSE, SKIP OR STOP AN ASSESSMENT

38.6.1 Steps to Pause and Resume Assessment

The coordinator can pause an assessment at any time.

Step 1: To PAUSE assessment, put two fingers on the screen and rotate one of them clockwise as if making a

half-circle on the screen. This places the program in “Pause”.

Step 2: To resume the assessment, enter the administrator’s passcode. The passcode is the 4 digit code provided

by your CCC. After entering the passcode, three (3) options will be presented. Tap on “Resume Instrument” to

resume the assessment.


Page 38-13 of 35

38.6.2 Skip or Stop Assessment

In addition to resuming an assessment at any time, the administrator can also skip an instrument or stop the

assessment. If an instrument is paused, skipped, or stopped, this information is recorded in the data export along

with the reason indicated.

After pausing a test and entering the passcode, three (3) options will be presented on the screen:

Resume Instrument

Skip Instrument – Select if you wish to skip to next assessment (you will not be able to resume i.e. go back to

the test)

Stop Assessment – Select if you are stopping but will resume later.

IMPORTANT: Please note that the “Pattern Comparison Processing Speed” assessment cannot be

resumed once you stop it.

If “SKIP Instrument” is selected, you will not be able to resume the test. User will be asked the reason for

skipping. Select one of the options below:

Safety Exclusion

Participant/Respondent Refused

No Time

Physical Limitations

Communication Problems

Language Barrier

Equipment Failure

Ineligible

Participant Ill/Emergency

Target Respondent Unavailable

Other

Once the reason that best describes why the instrument was skipped is selected, user will be taken to the beginning

of the next instrument, or the end of the assessment, if this was the last instrument in their assessment. If “Other”

is selected, users will be taken to a text screen where they will be required to enter another reason and then allowed

to skip the instrument.

If “STOP Assessment” is selected, users will be asked the reason for stopping. Select one of options below:

Safety Exclusion

Participant/Respondent Refused

No Time

Physical Limitations

Communication Problems

Language Barrier

Equipment Failure

Ineligible

Participant Ill/Emergency

Target Respondent Unavailable

Other

Select the reason that best describes why the assessment is being stopped and proceed to the main screen of the app.

If “Other” is selected, users will be taken to a text screen where they will be required to enter another reason and

then allowed to stop the assessment.

IMPORTANT TO NOTE: Various instruments may function differently if “Resume Instrument” or “Stop

Assessment” is selected. In most cases, users will be taken back to where they were before pausing the assessment.

In some cases (e.g., timed tests), users will be taken back to the beginning of the instrument. In select cases, an

instrument may not be restarted once paused.


Page 38-14 of 35

38.7 COGNITION ASSESSMENTS

38.7.1 Picture Vocabulary

Purpose: The NIH Toolbox Picture Vocabulary Test is a measure of receptive vocabulary administered in a

computer-adaptive test (CAT) format (How a participant responds to a question determines the difficulty of the

next item).

Equipment and materials needed:

iPad

Overview: The participant is presented with 4 pictures on the iPad screen and an audio recording saying a word.

The participant is instructed to touch the picture that most closely shows the meaning of the word. After the

participant makes a choice, another set of pictures automatically appears with a new audio recorded word.

The number of items presented depends on age and performance. For most it will contain 25 items

Age 3 – 7 years: The words used are based on their age up through words for 17 years old

Age 8 years and older: The words used are based on their education level

Notes for Administration:

Check volume level of iPad to ensue participant can hear

Administrator reads initial instructions on screen & instructions introducing test items

Participant can go back one item to change previous answer (Tap the Go Back icon at bottom right margin

of screen)

Can replay the word several times (Tap the Ear icon button on the lower right margin of screen)

If the participant says he/she does not understand the word after several repetitions, the examiner, may say

the word one more time.

If a participant has difficulty touching the screen, the participant may point and the examiner can touch the

screen. In this case, the examiner should say something like the following to the participant by saying:

“You can point to your choice and then I will touch the screen for you.”

If you Pause or Stop instrument- You can resume the test


Page 38-15 of 35

38.7.2 Flanker

Purpose: The NIH Toolbox Flanker Inhibitory Control and Attention Test is a measure of inhibitory control and

attention. The Flanker requires the participant to focus on a particular stimulus while inhibiting attention to the

stimuli flanking it.


iPad

Home base - Use for participants who are ≥ 8 years, only used for younger participants if they do well with the

“fish” items and testing progress to the “arrow” trials

Overview: All participants are instructed to choose one of two buttons on the screen that corresponds to the

direction in which the MIDDLE fish or arrow is pointing.

Age 3 – 7 years: Use Fish Stimuli and an Audio reminder to focus on MIDDLE, if >90% correct it progresses to

arrows.

Age 8 years and older: Use Arrow Stimuli and visual reminder to focus on the MIDDLE

All participants are first presented with four practice trials. If the participant responds incorrectly, an audio

recording provides feedback and highlights the correct choice. Similarly, a separate audio file plays each time the

participant gets a practice item correct. Participants must get at least three out of four practice trials correct to

advance to the test items. If these participants get fewer than three out of four practice trials correct, they can

complete up to two more sets of four practice trials, with the same cutoff to advance to the test trials.

Training for home base precedes the arrow trials for both the 3-7 and 8+ year olds.

Timed test – asked to answer as fast as you can. Must use home base for finger position between arrow test items

Notes for Administration:

Requires handedness to be entered for administration

Home Base finger placement between test items for arrow trials

Examiner demonstrates initial practice items

All instructions are on the screen. Examiner reads to or with participant while pointing out the relevant

aspects of the stimuli on the screen.

Automated feedback during practice

Cannot go back to previous question. Instructed if you make a mistake keep going

If the participant does not respond after five seconds to a practice trial, prompt him/her to do so by saying:

Choose one of the buttons.

If you Pause or Stop instrument- You can resume the test

The home base sheet is used so that there is a standard place for all participants to place their dominant-hand index

finger during the tests that measure the participants’ reaction time (NIH toolbox Flanker Inhibitory Control and

Attention Test and NIH toolbox Dimensional Change Cord Sort Test). For both tests, home base is used for all

participants ≥ 8 years. For younger participants it is only used if they progress to more difficult trials.

If a laminated home base sheet is not provided, then the document, located on the next page, can be printed out in

color and laminated. The blue dot will serve as home base. If using the recommended Otterbox iPad case, please

the legs on the case on the greyed out area.


Page 38-16 of 35

Home base Sheet


Page 38-17 of 35

38.7.3 List Sorting Working Memory

Purpose: This task assesses working memory and requires the participant to recall and sequence different visually

and orally presented stimuli.


iPad


NIH Toolbox List Sorting Working Memory Examiner Answer Sheet

Before beginning the test, the wireless keyboard should be paired by Bluetooth with the iPad. While administering

this measure, the examiner needs to be able to view the iPad and have the keyboard and Examiner Answer Sheet

easily accessible for scoring the participant’s responses and advancing the screens.

Overview: Pictures of different foods and animals are displayed with both an accompanying audio recording and

written text that name the item. A blank screen appears and the participant is asked to say the items back to the

examiner in size order from smallest to largest.

The opening /introductory instructions and the instructions to the test items are printed on the iPad screen and

should be read to the participant.

For the examiner, there is a code at the bottom right side of the iPad screen to indicate which test item is being

administered. The examiner should use this code to look up the correct response on the NIH Toolbox List Sorting

Working Memory Examiner Answer Sheet.

Because the participant answers orally, an examiner must record on a wireless keyboard whether each response is

correct; after the participant answers (gives the list of objects), the examiner types:

1, and hits the spacebar if the response (list) is correct

0, and hits the spacebar if the response (list) is incorrect.

During the practice phase the iPad will provide audio feedback based on whether the examiner records the answers

as correct or incorrect. For the practice items, after choosing 1 or 0, the question: “Are you ready?” appears on the

screen. Once confirming that the participant is ready, the examiner presses the spacebar to move on to the next

item (list of objects).

There are two different conditions: 1-List and 2-List.

In the 1-List condition, participants are required to order a series of objects (either food or animals) in size order

from smallest to largest. The test begins with a two object item. The participant must succeed at one of the first two

object items to continue.

In the 2-List condition, participants are presented both food and animals and are first asked to say the food objects

in size order, and then the animal objects in size order (smallest to largest). In each condition, participants ages 7-

85 have two practice items.


Page 38-18 of 35

Key Notes for Administration:

Wireless keyboard should be paired by Bluetooth with the iPad, before testing

Examiner needs to be able to view the iPad and have the keyboard and Examiner Answer Sheet easily

accessible for scoring child responses.

A code at the bottom right side of the iPad screen indicates which item is being administered (for quick

reference on the Answer Sheet)

Administrator marks response as correct (1) or incorrect (0) using keyboard and answer key.

Keyboard spacebar advances to next item

Participant can go back one item to change previous answer. Left arrow key goes back.

If a participant does not complete a practice item correctly, the test itself is not administered

Ctrl + Shift = Stop gesture (to stop the test)

“Correct” means that all stimuli named in correct order; no intrusions; synonyms are ok

Everyone gets 1 list and 2 list conditions

Ok to give rule reminders by saying:

“food first, then animals” during 2-list

“It is important to pay attention to the size of the objects on the screen” (not what they may

have experienced) “when putting things in size order from smallest to biggest”.

If necessary, examiners can provide short breaks between items.

If you Pause or Stop instrument- you can resume the test


Page 38-19 of 35

Laminated List Sorting Examiners Worksheets

If the Laminated List Sorting Examiners Worksheets are not provided, use the answer sheets below.


Page 38-20 of 35

38.7.4 Dimensional Change Card Sort

Purpose: The NIH Toolbox Dimensional Change Card Sort Test is a measure of cognitive flexibility and attention.


iPad

Home base - Use for participants who are ≥ 8 years, only used for younger participants if they do well on

the shape and color test trials and progress to the “mixed block” of shape/color.

Overview: Participants are asked to match pictures based on color or based on the shape.

A cue word (e.g., “shape” or “color”) appears on the screen for all participants. For young children ages 3-11, the

cue word is also spoken by a prerecorded audio file.

All instructions are on the iPad screen. The examiner reads them to and/or with the participant and points out the

relevant aspects of the stimuli on the screen. The next screen appears when the participant makes a choice.

Participants of all ages are given four practice trials with each dimension – color and shape. If the participant

responds incorrectly, an audio recording prompts him/her to choose the correct image. Similarly, a separate audio

file plays each time the participant gets a practice item correct.

Participants must get at least three out of four practice trials correct to advance to the practice trials for the

next dimension and then to the test trials.

If a participant of any age gets fewer than three out of four practice trials correct, he/she will complete up

to two more sets of four practice trials, with the same cutoff to advance to the test trials.

If a participant of any age does not meet the cutoff, the task will automatically discontinue.

Training for Home Base precedes testing of mixed (color/shape) trials

The test trials for participants ≥ 8 years of age consist of a mixed block of 30 shape/color items.

If participants ages 3-7 answer 4 of the 5 test trials for both the color and shape tests correctly, they will progress

onto a mixed block test, consisting of 30 shape/color trials.

Because the mixed trials are scored for reaction time, these young participants need to be trained in the use

of Home Base, where the participant places his/her index finger between trials. Training for Home Base

precedes the introduction of the mixed trials for 3-7-year-olds

Timed test – scored for reaction time. Participants ≥ 8 years old, or younger participants who progress to “mixed

block” of tests must use home base for finger position



Home base finger placement between test items for mixed trials only

Examiner demonstrates initial items

During the practice trials, examiner may prompt participant if they do not respond after 5 seconds by

saying “choose one of the pictures”.

Do not prompt during test trials; the program automatically advances after 10 seconds

May not go back to change answer. Instructed to keep going even if you made a mistake



Page 38-21 of 35

38.7.5 Pattern Comparison Processing Speed

Purpose: The NIH Toolbox Pattern Comparison Processing Speed Test is designed to measure processing speed.


iPad

Small ball (optional)

DO NOT USE HOME BASE. The participant may keep finger close to screen to answer as quickly as possible.

Overview:

The test itself takes less than 90 seconds and requires participants to discern whether two side-by-side pictures are

the same or not. Children and adults (Ages 7+) make this decision with “yes” or “no” buttons on the screen

For both test and practice items, if there is no response in 10 seconds, the pre-recorded audio asks: Are these the

same?

During practice items only, an audio recording will follow each item presentation:

If the item is correct, an audio recording will say: That’s right.

If the item is incorrect, an audio recording will explain why the item is incorrect. The item will appear a

second time and the participant will be asked to respond again.

If a participant (child or adult) continues to make an error after being corrected two times on two different practice

items, the test will discontinue.

Timed test – this test is scored for reaction time. For consistency do not use HOME BASE

Some participants need reminders to stay on task. The examiner may give prompts, although fewer are better as this

is a timed task. Each prompt should be succinct.

IMPORTANT REMINDER: Unlike some of the other tasks, if participants have difficulty touching the screen,

the examiner cannot answer for them, as this task measures processing speed and having the examiner answer

would compromise the accuracy of this measurement.



Do not use Home Base finger position

Examiner demonstrates initial item

Can prompt to stay on task during test trials by saying: “look at the screen”; “are these the same, yes or

no”; “remember to choose a button”; “remember to choose as quickly as you can”.

System automatically asks “Are these the same?” after 10 seconds

Test ends after 130 items or 85 seconds

Participants may be given a small stress ball to keep in their non-dominant hand.

May not go back to change answers. Instructed to keep going even if you made a mistake

Pause or Stop instrument allowed during instructions and practice items. If paused or stopped during the

test items you cannot resume the test.


Page 38-22 of 35

38.7.6 Picture Sequence Memory

Purpose: In this measure of episodic memory, sequences of pictured objects and activities are presented in a

particular order. The participants are asked to reproduce the sequence of pictures that is shown on the screen.


iPad

Overview: The participants are asked to reproduce the sequences of pictured objects and activities presented on the

iPad in a particular order.

All instructions are on the screen. Examiner reads to or with participant while pointing out the relevant aspects of

the stimuli on the screen.

For all participants, a demonstration sequence shows participants how to move the pictures on the screen. After

this, there are practice sequences. Practice and test sequences differ in length for participants of differing ages.

Participants respond by dragging pictures from the yellow box on the screen into the gray boxes on the screen.

They may verbalize the picture labels as they move the pictures. If, however, verbalizing the picture labels seems to

interfere with the participant’s performance, the examiner should say: “You don’t need to say the words/labels.”

If participants indicate that saying the labels helps them, the examiner may reassure them that they may say the

labels if they wish.

If participants look away from the screen while the pictures are being presented, the examiner should remind them

to: Watch the screen. Additionally, examiners should not allow participants to touch the screen while the pictures

are being presented. If participants try to touch the screen while pictures are being presented or while the examiner

is giving instructions, say: “Do not touch the screen until l tell you that it is your turn.”

If necessary, remind participants to: Wait until I tell you it is your turn (to touch the screen).

If the participant has difficulty dragging the pictures across the screen he/she may point and the examiner should

move the pictures in the way the participant indicates. The examiner should ensure the participant completely

understands how the process will work and should say something like the following to the participant: You point to

the picture you want to move, then point to the grey box where you want to move it. I’ll move it there for

you.

3-4 year old test – 6 step sequence

5-6 year old test– 7 step sequence

7 year old test – 15 and 18 step sequence

Age 8 years – Adult: 1 practice sequence, test is a 15 and 18 step sequence


Demonstration sequence must be completed, in order, before practice given

May redirect to pay attention if not watching the screen by saying “Watch the screen.”

Participant should not touch screen while pictures are being presented. If needed, remind participant by

saying “Do not touch the screen until l tell you that it is your turn.” or “Wait until I tell you it is your

turn”

If difficult for participant to move pictures on the screen, administrator may drag pictures as directed by

participant. “You point to the picture you want to move, then point to the grey box where you want to

move it. I’ll move it there for you.”



Page 38-23 of 35

38.7.7 Oral Reading Recognition Test

Purpose: The NIH Toolbox Oral Reading Recognition Test is a measure of reading decoding skills. These skills

are part of what are considered crystalized abilities, abilities that are dependent on past learning and exposure to

other cultural experiences.


iPad


Pronunciation guide

Before beginning the test, the wireless keyboard should be paired by Bluetooth with the iPad.

While administering this measure, the administrator needs to be able to view the iPad and have the keyboard and

word list and pronunciation guide easily accessible for scoring participant responses.

Overview: Each participant will see a series of words presented one at a time on the computer screen. Items are

presented in order of difficulty – the computer adjusts the difficulty level of items depending on the participant’s

performance. The number of items presented will depend on age and performance; for most participants, the

measure will last approximately 3 minutes and will contain about 25 items. The computer will administer each item

one by one, in an untimed fashion, until the test is completed.

After the participant pronounces a word, the examiner checks a pronunciation key as needed and types 1 if the

pronunciation is correct or 0 if the response is incorrect. After choosing 1 or 0, the examiner presses the spacebar to

move on to the next word or letter. The examiner can go back and change one prior answer by touching the left

arrow on the keyboard and choosing a new answer and then touching the spacebar to activate the next item.

Testing is automatically discontinued if a participant makes seven consecutive errors.

Note: Written and audio pronunciation guides are available for the examiner to use. The audio pronunciation guide

can be accessed at https://drive.google.com/drive/folders/0B13QPeR9DsC0WVpPX282aGMyZ3c?usp=sharing

It is essential that the examiner thoroughly review and master the pronunciation guide before administering this

test. The examiner should not rely on his/her understanding of word pronunciations based on previous experience.

Instead, the examiner must adhere to this guide for all item scoring. At the same time, while scoring is strict, it

should take into consideration difficulties in pronunciation that can be attributed to dialect or articulation; to the

extent that these difficulties are consistent throughout the testing, they should be accepted as correct pronunciations

by the examiner.


Age 7 years – Adult: Word based on age and education

Administrator marks response as correct (1) or incorrect (0) using keyboard and pronunciation key

Spacebar to advance to next item

Ctrl + Shift =Stop

Examiner can go back & change 1 prior answer (by using left arrow key on the keyboard)

Consistent articulation/pronunciation difficulties may be considered correct

Requires examiner practice

Can ask participant to repeat word if unsure


Pronunciation guide

https://drive.google.com/drive/folders/0B13QPeR9DsC0WVpPX282aGMyZ3c?usp=sharing


Page 38-24 of 35


Page 38-25 of 35


Page 38-26 of 35


Page 38-27 of 35


Page 38-28 of 35


Page 38-29 of 35


Page 38-30 of 35

38.8 EMOTION ASSESSMENTS

The emotional assessment consists of a series of questions related to strong feelings, such as of joy, sorrow, or fear.

NIH Toolbox Emotion consists of four subdomains: Psychological Well Being, Social Relationships, Stress & Self

Efficacy, and Negative Affect.

The participant answers emotion measure tests at his/her own speed. An examiner need not be involved. After a

question is answered, the screen advances automatically to the next question. All tests for the youngest age group

(3-7) are filled out by the parent. These are often referred to as parent proxy reports. On these tests, the proxy

(parent or guardian) responds to questions about the child according to how he/she thinks the child is doing.

For 8-12 year old children, tests come in both self- and parent-proxy report versions. For ages 13+, only self-report

tests are available.

38.8.1 Parent Proxy Emotion Battery

The Instructions for the Parent/Guardian Report will appear on the screen once the battery is started:

“On the next screens, you will see questions about your child’s feelings and behaviors. Read each question

carefully and answer as best you can. Please consider each question by itself and choose an answer that you believe

honestly shows how your child feels and acts.

Select an answer by touching the answer choice on the screen. After you make your choice, the computer will

automatically go on to the next question. If you want to change your answer, touch the GO BACK button to return

to the question, then touch a different answer choice.”

A button at the bottom of the screen instructs the parent/guardian to: “Touch and Hold Here to Continue.”

38.8.2 Emotion Battery

The Instructions for the Self Report Emotion Battery will appear on the screen once the battery is started:

“On the next screens, we will ask you questions about your thoughts, feelings and behaviors. Read each question

carefully and answer as well as you can. There is no right or wrong answer, only what is true for you and describes

your thoughts, feelings, and behaviors.

“Select an answer by touching your answer choice on the screen. After you make your choice, the computer will

automatically go on to the next question. If you want to change your answer, touch the GO BACK button to return

to the question, then touch a different answer choice.”

A button at the bottom of the screen instructs the participant to: “Touch and Hold Here to Continue.”


Page 38-31 of 35

38.9 EXPORT DATA

The Export Data button allows the person administering the Batteries to export data collected on a given iPad. At

any point when export is selected, it will contain all the data cumulatively collected on that iPad.

There are four options to transfer data files from the iPad to another device (i.e., computer connected to a backup

server):

e-mail (For the CKiD Study this OPTION is preferred)

iCloud drive

configurable web address

file sharing via iTunes (hardwired)

For the CKiD study, data files will be downloaded and sent via email to yourself. Then the files should be renamed

and sent via email to your CCC.

When emailing the data files, users should exercise caution and only email the data to secure email accounts

(e.g., the CKiD coordinator’s institutional/work email).

Prior to exporting files via email, users need to:

Set up the Mail app on the iPad. Mail set-up instructions can be accessed here: https://support.apple.com/en-

us/HT201320

Please use your secured work email account/address.

Activate the ability to email files. On Home page of app, select settings button. Next, select Data Security.

Then slide button to on position to allow Data Export To Use Email. Activate the ability to email files. Refer to

the NIH Toolbox App Settings section for detailed instructions.

38.9.1 Six (6) Steps to Export Files

Once both of the above conditions are met, there are six steps to exporting data. Please note that data files should be

sent as soon as possible after testing is completed in order to reduce the risk of losing data due to technical

malfunctions.

Step 1: Select Green Export Data button home page. Select the files to export by using the filtering options.

The default selection will be data that has not been exported yet. This will include test data you have not exported

yet. If you need to export files previously exported, touch on “Never Exported” (noted in the red circle below) for a

drop down screen that allows you to select “All Assessments”, or “Previously Exported” assessments.

https://support.apple.com/en-us/HT201320

https://support.apple.com/en-us/HT201320


Page 38-32 of 35

Step 2: To choose a file for export tap “select” to the right of the assessment that is ready for export. All

assessments can be selected by touching the “Select All” option. Assessments that have been selected for export

will show a checkmark instead of the word “select”.

Step 3: When ready to export the select files touch the Select Export Method button in the top right corner of the

screen. This will create/update data files for export.

There are three (3) types of data files generated here:

Registration data (contains participant demographic information entered on this iPad. File will include the age

of the participant but will not include date of birth).

Assessment data (contains data for every item administered)

Assessment scores (contains scores for every instrument administered)

999099

999099


Page 38-33 of 35

Step 4: Rename the files

The default file name is always: Date of assessment-Timestamp-File Type. You can rename the file by touching the

file of the file name and altering the name with the keyboard (iPad screen keyboard or wireless keyboard). The file

should be renamed as:

KiD ID_Visit#_Date of Study (year-month-day) _File Type

Example: 9-99-099_V5_2017-11-05_Registration Data.csv

9-99-099_V5_2017-11-05_Assessment Scores.csv

9-99-099_V5_2017-11-05_Assessment Data.csv

Step 5: In order to send the files, tap on the envelope symbol and enter your secure institutional email address,

and the email address of your clinical coordinating center (CCC) contact.

East Coast CCC: Jackie Ndirangu ([email protected])

Midwest CCC: Chris Smith ([email protected])

This message will arrive from the email address that was used when Mail was set up on this iPad. If desired, you

may send the report to your institutional email address and forward the renamed reports to the CCC.

Step 6: After re-naming the files and emailing them to your respective CCC, save a copy of the files in a secure

location or drive at your site.

CONFIDENTIALITY

To maintain confidentiality, no personal patient identifiers are data entered onto the NIH Toolbox app. In place of

a name, test administrators are instructed to enter the participant’s study identification number. Test administrators

are also instructed to enter the participant’s age and not their birthdate. See section 38.4.1 “Create a new participant

or select an existing participant”. This process ensures that no distinct personal identifiers are stored on the iPad.

mailto:[email protected]

mailto:[email protected]


Page 38-34 of 35

38.10 ASSESSING VALIDITY OF NIH TOOLBOX TESTS

In conjunction with completion of the age appropriate NIH Toolbox Cognitive Battery, the test administrator will

complete a Reliability Coding Sheet. This coding mechanism will provide the examiner perception of the

reliability of the test data collected and will assist in determining why a particular test was not administered or why

the data should be marked in the data set because of its low reliability. This rating will be conducted for each test

that is administered (e.g., subtests of larger batteries) and should be completed immediately after the test is

completed. It involves a 2-point code for each test (e.g., 1.0 is the reliability code for a typical reliable

administration).

38.10.1 Reliability Coding for NIH Toolbox Tests

Below is the two-point system to permit a rating for determining the reliability of each test. The reliability code

consists of 2 digits. The second digit will capture the reason why the reliability was compromised. The second

score would be recorded after the decimal point (e.g., 1.0).

If an ENTIRE Battery was not administered, please use the 2 point reliability code to reflect the reason.

Below is the two-point system to permit a rating that specifies the reason why the Battery or the test was not

administered. The second score is recorded after the decimal point (e.g., 5.1 indicates that the participant attempted

the Battery or test but was too impaired to complete, primarily related to physical reasons/limitations).

Primary Codes:

1 Standard procedure, Reliable Results

2 Irregular Procedure, Reliability Impact Minor

(e.g., child tired, but able to complete)

3 Irregular Procedure, Unreliable (e.g., child too active, too ill; examiner errors)

4 Participant Attempted – Participant too Impaired to Complete

5 Participant Attempted – Examiner Discontinued

6 Participant Attempted – Refused to Finish

7 Participant Refused to Begin

8 Not attempted - Reason Unrelated to Participant

(e.g., examiner forgot, task or measure not available)

Secondary Codes:

.0 Use this secondary reliability code when standard procedure with adequate reliability or irregular

procedure with adequate reliability occurred.

OR

.0 Testing problems not directly related to the patient (e.g., trained examiner not available, improper

test administration, procedure not available).

.1 Testing problems primarily related to physical limitations: injuries or disabilities

(not cognitive in nature).

.2 Testing problems primarily related to cognitive deficit: impairments in cognition, and/or behaviors

of the participant.

.3 Testing problems primarily related to English as a second language.

.4 Testing problems primarily related technological problems.


Page 38-35 of 35

38.10.2 Reliability Recording Worksheet

Documents

SECTION 38: NIH TOOLBOX (ABBREVIATED USER’S MANUAL) · 38.4.1 Create a new participant or select an existing participant Step 1: Once the app is launched, click on the blue “Participants”