2013

http://informahealthcare.com/bijISSN: 0269-9052 (print), 1362-301X (electronic)

Brain Inj, Early Online: 1–10! 2013 Informa UK Ltd. DOI: 10.3109/02699052.2013.775484

Let the games begin: A preliminary study using Attention ProcessTraining-3 and Lumosity� brain games to remediate attention deficitsfollowing traumatic brain injury

Samantha Zickefoose1, Karen Hux1, Jessica Brown1, & Katrina Wulf2

1Barkley Memorial Center, University of Nebraska–Lincoln, Lincoln, NE, USA, and 2Quality Living, Omaha, NE, USA

Abstract

Primary objective: Computer-based treatments for attention problems have become increas-ingly popular and available. The researchers sought to determine whether improvedperformance by survivors of severe traumatic brain injury (TBI) on two computer-basedtreatments generalized to improvements on comparable, untrained tasks and ecologically-plausible attention tasks comprising a standardized assessment.Research design: The researchers used an -A-B-A-C-A treatment design repeated across fouradult survivors of severe TBI.Methods and procedures: Participants engaged in 8 weeks of intervention using both AttentionProcess Training-3 (APT-3) and Lumosity� (2010) Brain Games. Two participants received APT-3treatment first, while the other two received Lumosity� treatment first. All participants receivedboth treatments throughout the course of two, 1-month intervention phases.Main outcomes and results: Individual growth curve analyses showed participants madesignificant improvements in progressing through both interventions. However, limitedgeneralization occurred: one participant demonstrated significantly improved performanceon one of five probe measures and one other participant showed improved performance onsome sub-tests of the Test of Everyday Attention; no other significant generalization resultsemerged. These findings call into question the assumption that intervention using either APT-3or Lumosity� will prompt generalization beyond the actual tasks performed during treatment.

Keywords

Acquired brain injury remediation, APT,APT-3, attention deficits, attention deficitsfollowing traumatic brain injury, AttentionProcess Training, attention treatment,brain games, computer-based brain injurytreatment, Lumosity

History

Received 22 March 2012Revised 28 November 2012Accepted 6 February 2013Published online 5 April 2013

Introduction

Computer-based interventions provide an alternative to trad-

itional therapy approaches for remediating some of the

persistent cognitive challenges experienced by people with

traumatic brain injuries (TBIs) [1]. Such interventions have

several potential advantages such as increasing service

availability to survivors in remote areas, supplementing

traditional therapy with access to unlimited independent

practice and reducing costs by decreasing the amount of time

professionals must devote to service provision [2]. However,

despite these potential advantages—as well as the popularity

and increasing availability of such programmes—their effi-

cacy remains largely unexplored [3], especially with regard to

interventions that are hierarchical in nature and present

increasingly complex challenges as a person demonstrates

mastery of more basic skills [4].

One hierarchically-based programme targeting various

aspects of attention and designed specifically for survivors

of brain injury is Attention Process Training-3 (APT-3) [5].

The APT-3 is a computer-based version of the earlier APT-1

[6] and APT-2 [7] programmes developed by the same

authors. It incorporates specific principles assumed to be

important for attention remediation including: (a) hierarchical

organization to stimulate basic attention skills while increas-

ing the complexity of attention targets, (b) intensive repetition

of exercises and targets to facilitate the establishment of

neural connections and (c) graphic representation of progress

[8]. Because several research groups (e.g. [9–11]) have

documented through pre-/post-intervention comparisons of

participant scores on standardized attention measures that the

APT-1 is effective in improving the attention skills of

survivors of TBI, a logical assumption is that the APT-3

will be comparable in its effectiveness. Questions persist,

however, regarding the functionality of these improvements

given that positive changes following APT-1 intervention

appear primarily at the impairment level of the World Health

Organization’s Model of Disability [12] rather than the

activity and participation levels—that is, the levels most

reflective of skill generalization [13].

Computer-based alternatives to APT-3 are activities avail-

able via the Internet designed to help people maintain or

improve a variety of cognitive functions. Some of these

offerings claim to slow cognitive decline associated with

Correspondence: Karen Hux, PhD, 351 Barkley Memorial Center,University of Nebraska–Lincoln, Lincoln, NE 68583-0738, USA. Tel:402-472-8249. Fax: 402-472-7697. E-mail: khux1@unl.edu

Bra

in I

nj D

ownl

oade

d fr

om in

form

ahea

lthca

re.c

om b

y L

ibra

ry o

f H

ealth

Sci

-Uni

v of

Il o

n 05

/01/

13Fo

r pe

rson

al u

se o

nly.

normal ageing or the beginning stage of Alzheimer’s disease

or other forms of dementia [14–17]; others purport to improve

memory, attention and/or processing skills in individuals who

have brain damage from acquired or traumatic insults or who

demonstrate impaired functioning related to learning dis-

abilities or attention deficit disorders [18]; still others claim to

help individuals without histories of neurological or cognitive

impairments hone specific skills [19]. Although companies

touting such programmes state that scientific theory and

research support their claims, uncertainty exists about

whether engaging in brain games actually enhances cognitive

functioning [3, 15, 20].

One Web-based program—Lumosity: Reclaim Your

Brain� [21]—purports to help people both with and without

neurological impairments bolster their cognitive performance

by regular engagement in various games and mental exercises.

Initial investigations of the efficacy of using Lumosity� Brain

Games to remediate cognitive challenges secondary to TBI

have been positive. For example, Kesler et al. [22] adminis-

tered Lumosity� Brain Games to children with cancer-related

brain injury and reported significant improvements in

participants’ processing speeds, mental flexibility, memory

and pre-frontal cortex activation following intervention.

Despite these positive findings, researchers and TBI profes-

sionals know little about the generalization of improved

functions to untrained activities or situations occurring

outside of structured therapy sessions.

Programmes such as APT-3 and Lumosity� Brain Games

offer new avenues for structuring attention intervention for

survivors of TBI. The purpose of the research reported herein

was to investigate whether improved performance by adult

survivors of severe TBI on APT-3 and/or Lumosity� activities

generalized to improved performance on comparable,

untrained but similarly-structured attention tasks as well as

to ecologically-plausible attention tasks included as part of a

standardized assessment tool.

Method

Participants

Four adult males with severe TBI served as study participants.

The participants ranged in age from 36–50 years. All had

sustained severe TBIs as indicated by a period of coma

exceeding 1 day or a period of post-traumatic amnesia

exceeding 1 week [23, 24], were a minimum of 3 years post-

injury, spoke English as a primary language, had normal or

corrected-to-normal hearing and vision and did not have any

history of learning disabilities or neurological damage other

than that associated with TBI. None of the participants had

aphasia based on attainment of an Aphasia Quotient score of

93.8 or greater on the Western Aphasia Battery–Revised

(WAB-R) [25]. Participants passed a motor screening and a

mathematical computation screening to ensure adequate skills

to perform the experimental tasks. Demographic, injury-

related and screening test information about each participant

appears in Table I.

Design

The study used a single-subject A-B-A-C-A (i.e. pre-testing,

intervention phase 1, post-testing 1, intervention phase 2,

post-testing 2) treatment design repeated across participants.

Two participants received APT-3 treatment during interven-

tion phase 1, while the other two participants received

Lumosity� treatment; all participants engaged in the alternate

treatment programme during intervention phase 2. Thus, each

participant received both treatments throughout the course of

the two, 1-month intervention phases.

Materials

Formal and informal measures of attention

The researchers used the Test of Everyday Attention (TEA)

[26] and a researcher-generated, repeatable probe to measure

each participant’s attention performance. The TEA is a

standardized measure that targets everyday attention demands

in ecologically-plausible scenarios to assess visual selective

attention/speed (i.e. map search and telephone search without

distraction sub-tests), attentional switching (i.e. visual eleva-

tor sub-test), sustained attention (i.e. lottery and telephone

search while counting sub-tests) and auditory-verbal working

memory (i.e. elevator counting with reversal and elevator

counting with distraction sub-tests). It was administered at

pre-testing, post-testing 1 and post-testing 2. The researcher-

generated probe measures were adapted from the

Neurological Assessment Battery (NAB) Numbers and

Letters Test Parts B, C and D [27]. They were administered

at the same times as the TEA as well as on a weekly basis

during intervention phases 1 and 2. The researchers designed

the probe tasks to emulate the attention targets of the APT-3

and Lumosity� programmes (e.g. sustained attention, select-

ive attention, divided attention). The probes included four

sub-tests: (a) marking the X’s in eight rows of numbers and

letters as quickly as possible; (b) counting the number of X’s

in eight rows of numbers and letters as quickly as possible and

writing down the sum for each line; (c) adding the sum of

numbers in eight rows of letters and numbers from left to

right and writing down the sum of each line; and (d) marking

each X in eight rows of numbers and letters while simultan-

eously adding and then writing down the sum of the numbers

in the row. The researchers developed 10 versions of

comparable mathematical equations and letter combinations

Table I. Participants’ background and aphasia quotient information.

Participant Age (years) Time post-injury (years) Cause of injury Length of coma (days) Length of PTA (months) WAB-R Aphasia quotient

OE 36 3 Penetrating blow 2 40.25 99.6NG 50 35 Fall 115 43.50 98.0KX 36 9 Car accident 84 42.80 97.6KS 49 23 Car accident unknown 41.00 99.6

2 S. Zickefoose et al. Brain Inj, Early Online: 1–10

Bra

in I

nj D

ownl

oade

d fr

om in

form

ahea

lthca

re.c

om b

y L

ibra

ry o

f H

ealth

Sci

-Uni

v of

Il o

n 05

/01/

13Fo

r pe

rson

al u

se o

nly.

as those listed in the NAB to prevent specific-task learning.

They measured participant performance based on the accur-

acy of responses.

Perceptual rating scales

The researchers designed a measure to allow participants to

inform them about their perceptions regarding the two

intervention programmes used over the course of treatment.

Specifically, the researchers asked participants the following

two questions at the end of each intervention phase: (a) ‘How

much are you enjoying these activities?’ and (b) ‘Would you

complete these activities on your own?’ Participants

responded by pointing to a number between 1–5 presented

on a Likert-type scale, with higher numbers corresponding

with more positive responses than lower numbers.

Intervention materials

Each individual participated in two intervention programmes

over the course of the research project: the APT-3 and the

Lumosity� Brain Games. The researchers acquired a proto-

type version of the APT-3 from one of the programme authors

for use in this research project (Sohlberg, personal commu-

nication, 17 May 2010). To access Lumosity�, the researchers

purchased a subscription.

The APT-3 programme defines five domains of attention:

(a) sustained attention, (b) selective attention, (c) working

attention, (d) suppression and (e) alternating attention. The

researchers grouped each variation of APT-3 tasks within each

attention domain into a hierarchy of complexity based on the

skills required to complete the tasks, the level of distraction

and the rate of presentation. They used this hierarchy to

determine the sequence of tasks to present to participants as

they progressed through the intervention programme. The

sustained attention domain included a total of 20 levels; the

selective attention domain included 114 levels; the working

memory domain included 13 levels; and the suppression and

alternating attention domain included 16 levels.

Lumosity� is available via an Internet website (i.e. http://

www.lumosity.com) that provides access to brain games

designed to improve cognitive processing speed, flexibility,

attention, memory and problem-solving skills. Game com-

plexity increases and decreases systematically based on an

individual’s performance data. Multiple forms of each game

level are available to prevent task learning with continued

practice.

The researchers selected five attention-oriented

Lumosity� games—Birdwatching, Monster Garden, Playing

Koi, Rotation Matrix and Top Chimp—for use in this research.

Each selected game followed three principles in that they

provided: (a) repeated opportunities for diverse practice, (b)

hierarchical organization of experimental stimuli with grad-

able levels of difficulty and (c) human–computer interactions

through multi-media.

Procedures

The researchers alternately assigned participants to begin

either with the APT-3 or Lumosity� Brain Games interven-

tion and then performed pre-testing to establish each

participant’s baseline level of attention functioning.

Following this, participants engaged in 20 treatment sessions

within a 1-month period, with each session lasting �30

minutes; some daily variation occurred in session length

depending on how quickly a participant performed the tasks

selected for that day. When using APT-3, the participants

performed four tasks during each session—one corresponding

with sustained attention, one with selective attention, one with

working attention and one either with suppression or

alternating attention. When using Lumosity�, the researchers

systematically alternated tasks so that a participant performed

each of the five selected games a minimum of two times per

four sessions. After completing the first month of intervention

(i.e. intervention phase 1), participants performed post-testing

1 and the perceptual rating form for the appropriate

intervention programme; they then proceeded through inter-

vention phase 2 with the alternate intervention programme,

followed by completion of post-testing 2 and the appropriate

perceptual rating form.

Four trained research assistants alternately administered

the daily intervention sessions. The researchers trained the

assistants by demonstrating the data collection procedures

both for the APT-3 and the Lumosity� programmes. In

addition, the research assistants had written step-by-step

programme instructions to guide them through the interven-

tion sessions.

Data analysis

Procedural integrity and inter-judge reliability

The researchers ensured procedural integrity by performing

the following procedures with each research assistant involved

in data collection: (a) providing one-on-one training to

familiarize the assistant with each intervention programme

and explain the set-up and data collection procedures, (b)

demonstrating the set-up and data collection procedures for

both intervention programmes, (c) observing the assistant

performing all set-up and data collection procedures and (d)

providing a written checklist of steps to be performed prior to,

during and following each intervention session.

All scoring of response accuracy for the Lumosity�intervention sessions was handled through the Web-based

program; hence, research assistants only were responsible for

copying scores displayed on the computer monitor. In

contrast, inter-judge reliability was an issue when participants

performed the APT-3 intervention, because the prototype

APT-3 programme did not accurately score participants’

responses within activities. To determine inter-judge reliabil-

ity, the primary researcher and a research assistant independ-

ently collected data during an APT-3 intervention session with

one of the four study participants. Point-to-point reliability for

each APT-3 activity was computed using the following

formula:

number of agreements

number of agreementsþ disagreements� 100:

Inter-judge reliability ranged from 80–100% across the

four administered activities, with the overall reliability

equalling 93%.

DOI: 10.3109/02699052.2013.775484 Remediating attention deficits 3

Bra

in I

nj D

ownl

oade

d fr

om in

form

ahea

lthca

re.c

om b

y L

ibra

ry o

f H

ealth

Sci

-Uni

v of

Il o

n 05

/01/

13Fo

r pe

rson

al u

se o

nly.

Dependent variables

The researchers monitored participants’ session-to-session

performance on each APT-3 and Lumosity� task. Graphic

displays and individual growth curve modelling procedures

provided a means of visualizing and quantifying an individ-

ual’s progression through the two intervention phases of the

research. The researchers also monitored participants’ per-

ceived enjoyment of the intervention activities and the

likelihood that they would continue performing similar

activities independently if the option were available to

them. This monitoring was done through responses to

Likert-type questions.

Participants’ scores on probe measures and scaled scores or

adjusted scaled scores, as appropriate, on the pre-testing,

post-testing 1 and post-testing 2 administrations of the TEA

were dependent variables used for analysis of generalization

effects. Graphic displays, computation of trend lines and

comparison of trend line slopes with a slope of 0 provided a

means of judging generalization of improved attention

performance based on the probe measures. The researchers

used graphic displays of TEA scaled score changes to further

evaluate generalization of improved attending behaviour.

Results

The research results appear in three sections. The first section

presents data about each participant’s progression through the

APT-3 and Lumosity� programmes. Included with this are

data about the participants’ perceptions of their enjoyment

and willingness to continue performing the activities beyond

the conclusion of the research project. The second section

includes data about participants’ performances on the probe

measures administered throughout the course of the interven-

tion programmes. Section three includes the pre-testing, post-

testing 1 and post-testing 2 data from the standardized

measure of attention functioning—that is, the TEA.

Progression through intervention programmes

APT-3

Participants progressed at their own speed through multiple

levels of the APT-3 hierarchy of activities within the sustained

attention, selective attention, working attention and suppres-

sion and alternating attention domains. Figure 1 displays each

participant’s progression within each domain. Note that

abrupt changes in hierarchy level within the selective

attention domain occurred for participants at various points

during the course of intervention; these were due to

researcher-determined readjustments needed in the difficulty

level of the presented tasks.

The researchers computed trend lines associated with each

participants’ progression through the APT-3 programme

(Table II). Statistical analysis confirmed that the slopes of

the trend lines differed significantly from a line with a slope of

0.00 for all four attention domains across all four participants,

thus suggesting that participants progressed to more challen-

ging levels throughout the course of the intervention.

Participants not desiring to continue with the programme

0

2

4

6

8

10

12

14

16

18

20

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Leve

l

Session

Sustained attention

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

2

0

4

6

8

10

12

Leve

l

Session

Working attention

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 200

2

4

6

8

10

12

14

16

Leve

l

Session

Suppression & alternating attention

OE

NG

KX

KS

OE

NG

KX

KS

0

20

40

60

80

100

Leve

l

Session

Selective attention

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Figure 1. Participants’ progressions through increasing levels of difficulty on APT-3 tasks within each attention domain.

4 S. Zickefoose et al. Brain Inj, Early Online: 1–10

Bra

in I

nj D

ownl

oade

d fr

om in

form

ahea

lthca

re.c

om b

y L

ibra

ry o

f H

ealth

Sci

-Uni

v of

Il o

n 05

/01/

13Fo

r pe

rson

al u

se o

nly.

reported that they perceived the tasks as not being functional in

nature and lacking representation of real-life situations.

Participants responded to Likert-type questions at the end

on the APT-3 intervention phase to convey their enjoyment of

and willingness to continue performing APT-3 activities

beyond the conclusion of the research study (Table III). Likert

scores reflected strong enjoyment and interest in continuing

APT-3 by two of the four participants and equally strong

rejection of APT-3 by the other two participants.

Lumosity�

Figure 2 displays each participant’s progression through the

five selected Lumosity� games. As with the APT-3 data, the

researchers computed trend lines associated with each

participants’ progression through the Lumosity� activities

(Table IV). Statistical analysis confirmed that the trend line

slopes differed significantly from a line with a slope of 0.00

for four of the five game activities across all four participants,

thus suggesting that participants progressed to more challen-

ging levels throughout the course of the intervention. For one

Lumosity� game (i.e. Rotation Matrix), only OE progressed

in a manner in which his trend line differed significantly from

a line with a slope of 0.00.

Table III contains participants’ self-reported perceptions

about their enjoyment of performing Lumosity� activities

and willingness to continue performing the games independ-

ently. All four participants indicated that they enjoyed the

Lumosity� games. However, only two of the four expressed a

strong desire to perform the games outside the realm of the

research; the other two provided neutral Likert responses to

this question. Again, the participants’ verbal comments about

continuing to perform the Lumosity� activities reflected

skepticism about the applicability of the activities to improv-

ing attention skills in real-life situations.

Probe measures

Participants performed a total of nine probes over the course

of the research project: one during pre-testing, three during

intervention phase 1, one during post-testing 1, three during

intervention phase 2 and one during post-testing 2. Each

probe included four tasks: (a) cancellation of X’s appearing in

rows, (b) counting the number of X’s appearing in rows, (c)

summing of digits appearing in rows and (d) simultaneous

cancellation of X’s and summing of digits appearing in rows.

The third and fourth probe tasks were of particular import-

ance because participants had to perform more challenging

activities than they did for the first and second tasks; in

addition, the fourth task required divided attention. Because

individual changes over time were of interest rather than

comparisons across participants, the researchers analysed data

from each participant separately.

Participant OE performed the four probe tasks at a level at

or near ceiling at the time of pre-testing. He then maintained

high accuracy throughout all subsequent probe administra-

tions, with accuracy mean scores exceeding 85% in all

instances (i.e. task 1: M¼ 97.22%, SD¼ 5.51; task 2:

M¼ 93.06%, SD¼ 9.08; task 3: M¼ 94.44%, SD¼ 9.08;

task 4 cancellation: M¼ 87.50%, SD¼ 12.50; task 4 summa-

tion: M¼ 94.44%, SD¼ 9.08). Because of his consistency and

high initial accuracy, comparison of trend line slopes

associated with accuracy improvements on probe tasks

across administrations to a line with a slope of 0.00 yielded

no significant results. Still, the slope of his accuracy score

trend line for the summation portion of probe task 4

approached significance when compared with a line with a

slope of 0.00 (t¼ 2.198; p¼ 0.0639), thus suggesting a trend

towards improvement in this aspect of the divided attention

probe task despite the small margin available for improve-

ment. OE’s improved probe task accuracy occurred primarily

Table II. Participants’ trend line, t-value and p-value data for attention domains targeted in APT-3.

Sustained attention Selective attention Working attentionSuppression &

alternating attention

OETrend line y¼ 0.76xþ 3.04 y¼ 4.83x� 2.53 y¼ 0.07xþ 4.36 y¼ 0.06xþ 3.04t-value t¼ 10.778 t¼ 13.042 t¼ 2.931 t¼ 3.571p-value p50.0001 p50.0001 p50.0089 p¼ 0.0022

NGTrend line y¼ 0.88xþ 2.53 y¼ 7.29x� 6.21 y¼ 0.09xþ 3.56 y¼ 0.09xþ 2.29t-value t¼ 18.010 t¼ 8.295 t¼ 7.758 t¼ 2.995p-value p50.0001 p50.0001 p50.0001 p¼ 0.0078

KXTrend line y¼ 0.85xþ 4.79 y¼ 4.94xþ 16.14 y¼ 0.14xþ 4.09 y¼ 0.09xþ 2.67t-value t¼ 15.031 t¼ 17.689 t¼ 5.327 t¼ 3.915p-value p50.0001 p5.0001 p50.0001 p¼ 0.0010

KSTrend line y¼ 0.76xþ 3.04 y¼ 4.83x� 2.53 y¼ 0.07xþ 4.36 y¼ 0.06xþ 3.04t-value t¼ 10.778 t¼ 13.042 t¼ 2.931 t¼ 3.571p-value p50.0001 p50.0001 p50.0089 p¼ 0.0022

Table III. Likert scale ratings of participants’ enjoyment of andwillingness to continue performing APT-3 and Lumosity� tasksindependently.

APT-3 Lumosity�

Participant EnjoymentWillingnessto continue Enjoyment

Willingnessto continue

OE 5 4 5 5NG 1 1 4 3KX 5 5 4 3KS 1 1 5 5

DOI: 10.3109/02699052.2013.775484 Remediating attention deficits 5

Bra

in I

nj D

ownl

oade

d fr

om in

form

ahea

lthca

re.c

om b

y L

ibra

ry o

f H

ealth

Sci

-Uni

v of

Il o

n 05

/01/

13Fo

r pe

rson

al u

se o

nly.

during the first intervention phase when he was using the

APT-3 programme.

Participant NG demonstrated considerably greater accur-

acy variability than OE in performing the probe tasks across

the course of the research. Across administrations, NG

performed probe task 1 with greater accuracy (M¼ 90.28%,

SD¼ 15.02) than any of the other probe tasks (task 2:

M¼ 61.11%, SD¼ 24.56; task 3: M¼ 75.00%, SD¼ 13.98;

task 4 cancellation: M¼ 66.67%, SD¼ 17.68; task 4 summa-

tion: M¼ 75.00%; SD¼ 13.98). No progression of improved

accuracy occurred over the course of the project, as

demonstrated by the finding of no significant differences

0

1

2

3

4

5

6

7

8

9

10

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

Leve

l

Session

Birdwatching0

1

2

3

4

5

6

Leve

l

Session

Monster Garden

0

1

2

3

4

5

Leve

l

Session

Playing Koi0

1

2

3

4

5

6

7

Leve

lSession

Top Chimp

OE

NG

KX

KS

OE

NG

KX

KS

OE

NG

KX

KS

0

2

4

6

8

10

12

Level

Session

Rotation Matrix

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

Figure 2. Participants’ progressions through increasing levels of difficulty on Lumosity� Brain Game activities.

Table IV. Participants’ trend line, t-value and p-value data for Lumosity� Brain Game activities.

Birdwatching Monster Garden Playing Koi Rotation Matrix Top Chimp

OETrend line y¼ 0.30xþ 2.03 y¼ 0.09xþ 1.8 y¼ 0.15xþ 1.90 y¼ 0.02xþ 7.08 y¼ 0.35xþ 1.22t-value t¼ 13.543 t¼ 5.464 t¼ 7.288 t¼ 0.514 t¼ 12.725p-value p50.0001 p50.0001 p50.0001 p¼ 0.6147 p50.0001

NGTrend line y¼ 0.18xþ 1.14 y¼ 0.04xþ 1.59 y¼ 0.18xþ 0.76 y¼ 0.01xþ 6.19 y¼ 0.20xþ 2.40t-value t¼ 13.045 t¼ 2.611 t¼ 8.047 t¼ 0.292 t¼ 5.292p-value p50.0001 p¼ 0.0197 p50.0001 p¼ 0.7749 p¼ 0.0001

KXTrend line y¼ 0.12xþ 1.13 y¼ 0.04xþ 1.49 y¼ 0.14xþ 1.47 y¼ 0.06xþ 6.29 y¼ 0.19xþ 1.91t-value t¼ 9.335 t¼ 3.346 t¼ 7.885 t¼ 2.214 t¼ 5.456p-value p50.0001 p¼ 0.0036 p50.0001 p¼ 0.0439 p50.0001

KSTrend line y¼ 0.20xþ 0.76 y¼ 0.04xþ 1.49 y¼ 0.07xþ 2.12 y¼ 0.02xþ 6.10 y¼ 0.16xþ 2.21t-value t¼ 10.808 t¼ 3.346 t¼ 3.878 t¼ 0.449 t¼ 5.758p-value p50.0001 p¼ 0.0036 p¼ 0.0012 p¼ 0.6599 p50.0001

6 S. Zickefoose et al. Brain Inj, Early Online: 1–10

Bra

in I

nj D

ownl

oade

d fr

om in

form

ahea

lthca

re.c

om b

y L

ibra

ry o

f H

ealth

Sci

-Uni

v of

Il o

n 05

/01/

13Fo

r pe

rson

al u

se o

nly.

between the slopes of trend lines associated with each probe

task and a line with a slope of 0.00.

Participant KX also displayed variability in his perform-

ance accuracy on the four probe tasks across the course of the

research project. In contrast to participants OE and NG, KX’s

greatest accuracy and least variability occurred when per-

forming the summation portion of task 4 (M¼ 87.50%,

SD¼ 10.83) rather than when performing the other three

probe tasks (task 1: M¼ 79.17%, SD¼ 21.65; task 2:

M¼ 83.33%, SD¼ 15.31; task 3: M¼ 68.06%, SD¼ 14.13;

task 4 cancellation: M¼ 70.83%, SD¼ 17.68). Comparison of

a line with a slope of 0.00 to the slopes of KX’s performance

accuracy trend lines on each probe task across administrations

yielded a significant difference for task 2 only (t¼ 2.393;

p¼ 0.0479). Improvement in performing task 2 occurred

primarily during the first intervention phase when KX was

using the APT-3 programme.

Participant KS’s accuracy in performing all probe tasks

ranged either from 50–100% or 62.5–100% across the

multiple administrations. His greatest variability in perform-

ance occurred on tasks 1 (M¼ 87.50%, SD¼ 15.31) and 3

(M¼ 80.56%, SD¼ 16.67). These were also the tasks for

which KS received his highest mean accuracy scores; mean

accuracy for task 2 and for the cancellation and summation

aspects of task 4 ranged from a low of 73.61% (task 4

summation) to a high of 77.78% (task 2). KS demonstrated

upward trends in his performance accuracy for tasks 1 and 4

across the course of the research project. Comparison of the

slope of his trend lines associated with these progressions

with a line with a slope of 0.00 approached but did not reach

significance for task 1 (t¼ 2.167; p¼ 0.0670) and for the

cancellation portion of task 4 (t¼ –2.200; p¼ 0.0637). The

trends toward improved performance on the probe tasks

occurred primarily during the first intervention phase when

KS was using the Lumosity� programme.

TEA scores

Participants performed the TEA on three occasions: (a) before

the start of the first intervention, (b) after completing the first

intervention and before initiating the second intervention and

(c) after completing the second intervention. Because of

repeated exposure to the test, practice effects may have

occurred. As such, the researchers used the adjusted scaled

score equivalents to obtained raw scores provided in the TEA

manual to evaluate participants’ performances on the second

and third test administrations. Adjusted scaled score equiva-

lents guard against possible practice effects and allow for

interpretation of performance data when attempting to

identify real improvement among multiple TEA administra-

tions. Computation involves comparing the change in a

person’s score between second and third testing to the mean

practice effect for a given sub-test reported in the TEA

manual. If the change is larger than the mean, improvement

has occurred; however, if the change is smaller than the mean,

deterioration has occurred. The raw scores and scaled or

adjusted scaled scores, as appropriate, for each participant

appear in Table V. The mean scaled or adjusted scaled score

equivalent on each of the TEA sub-tests is 10 with a standard

deviation of 3. As with the probe measures, individual

changes over time were of interest rather than comparisons

across participants; hence, the researchers analysed data from

each participant separately.

OE’s scaled score equivalents on the TEA sub-tests at the

time of pre-testing ranged from 0–4. This corresponded with

performance �2 SD below the mean on all sub-tests. OE’s

adjusted scaled scores improved on six of the nine sub-tests

post-APT-3 intervention and his scores either remained the

same or further improved on four sub-tests post-Lumosity�training. Overall, OE’s adjusted scaled scores improved from

his pre-testing performance on seven of the nine sub-tests over

the combined APT-3 and Lumosity� intervention periods. At

the time of the second post-testing, adjusted scaled scores for

six of the nine sub-tests remained �2 SD below the mean; two

adjusted scaled scores were between 1–2 SD below the mean;

and one adjusted scaled score was within 1 SD above the mean.

NG’s pre-testing scaled score equivalents did not exceed 6

on any of the TEA sub-tests and he received a scaled score of

0 on four of the nine sub-tests. This corresponded with per-

formance between 1–2 SD below the mean on three sub-tests,

Table V. Participants’ scaled or adjusted scaled score results on TEA sub-tests.

Participant Visual selective attention/speed Sustained attention Attentional switching Auditory-verbal working memory

OEPre-testa 4 0 1 0 1 4 4 2Post-APT-3b 5þ 0 7þ 2þ 4þ 13þ 4 2Post-Lumosity�b 6þ 0 6þ 2þ 1 13þ 4 3þ

NGPre-testa 0 0 0 2 6 4 6 5Post- Lumosity�b 5þ 0 0 0� 5� 5þ 5� 7þ

Post -APT-3b 5þ 0 0 0� 3� 7þ 4� 2�

KXPre-testa 4 4 3 6 8 11 10 8Post-APT-3b 5þ 0� 0� 3� 11þ 9� 11þ 7�

Post-Lumosity�b 3� 0� 2� 3� 3� 4� 7� 7�

KSPre-testa 3 2 2 2 5 5 6 7Post- Lumosity�b 3 0� 0� 4þ 6þ 7þ 4� 10þ

Post- APT-3b 3 0� 0� 3þ 2� 9þ 4� 8þ

ascaled score: M¼ 10 (SD¼ 3); badjusted scaled score: M¼ 10 (SD¼ 3).þ indicates increase from baseline performance; � indicates decrease from baseline performance.

DOI: 10.3109/02699052.2013.775484 Remediating attention deficits 7

Bra

in I

nj D

ownl

oade

d fr

om in

form

ahea

lthca

re.c

om b

y L

ibra

ry o

f H

ealth

Sci

-Uni

v of

Il o

n 05

/01/

13Fo

r pe

rson

al u

se o

nly.

between 2–3 SD below the mean on two sub-tests and53 SD

below the mean on four sub-tests. After completing the

Lumosity� intervention, NG’s adjusted scaled scores

increased on three of the nine subtests, remained consistent

on three other sub-tests and decreased on the three remaining

sub-tests. After the APT-3 intervention, his adjusted scaled

scores remained consistent on one and improved further on

another of the sub-tests on which he had demonstrated earlier

improvement; his adjusted scaled score performance

decreased or remained at 0 on all remaining sub-tests.

Overall, NG’s adjusted scaled scores improved on two of the

nine sub-tests over the combined Lumosity� and APT-3

intervention periods. At the time of post-testing 2, four of the

nine adjusted scaled scores were at 0 (i.e. 53 SD below

mean), three remained at or less than 2 SD below the mean

and two were between 1–2 SD below the mean.

Four of KX’s scaled score equivalents on TEA sub-tests

corresponded with performance52 SD below the mean at the

time of pre-testing; one scaled score equivalent was between

1–2 SD below the mean; and four were between 1 SD below

and 1 SD above the mean. KX’s adjusted scaled scores

improved on three of the nine sub-tests following completion

of the APT-3 intervention, but his scores decreased on the

remaining six sub-tests. Following the Lumosity� interven-

tion, all but one of KX’s adjusted standard scores decreased or

remained the same as they were following the APT-3

intervention; all scores were lower than at pre-testing.

KS’s scaled score equivalents on TEA sub-tests ranged

from 0–7 at the time of pre-testing. This corresponded with

scores ranging from 1 to53 SD below the mean. Following

intervention with the Lumosity� programme, KS’s adjusted

scaled scores improved on four of the nine TEA sub-tests;

however, his adjusted standard scores on three other sub-tests

decreased. Following completion of the APT-3 intervention,

KS’s adjusted scaled score for one sub-test increased again,

but this gain was offset by his decreased score on another sub-

test. Overall, KS’s adjusted scaled scores improved on three

of the nine sub-tests and declined on four of the nine sub-tests

over the combined Lumosity� and APT-3 interventions. At

the time of the second post-testing, three of the nine adjusted

scaled scores were 0 (i.e. 53 SD below the mean), four

remained52 SD below the mean and two were within 1 SD

below the mean.

Results summary across participants

All four participants demonstrated statistically significant

progress in reaching new levels of difficulty on intervention

tasks over the course of the APT-3 and Lumosity� interven-

tion phases. Specifically, participant KX improved signifi-

cantly on all APT-3 and Lumosity� activities and the other

three participants improved significantly on all but the

Rotation Matrix game of Lumosity�. Likert rating scale

data regarding participants’ perceived enjoyment of the

intervention programmes indicated that all four enjoyed

engaging in the Lumosity� Brain Games, whereas only two

of the four enjoyed engaging in the APT-3 tasks. Similarly, all

four were either neutral or positive about continuing further

with the Lumosity� activities, but only two of the four felt

this way about the APT-3 activities.

Generalization of improved attention evidenced through

probe measure performance changes across participants

appeared predominantly during the first intervention period,

regardless of which intervention programme a given partici-

pant received during that phase. Specifically, early general-

ization was evident for one aspect of the probe measure for

participant KX; the performances of two other participants

(i.e. OE and KS) approached significance either for one or

two aspects of the probe task. OE’s high performance

accuracy at the time of pre-testing suggests that the probe

task may have been inadequate for measuring generalization

of improvement in his attention functioning. Generalization of

improved attention based on repeated performances of the

TEA yielded mixed findings. Participant OE appeared to

generalize his improved attending behaviour attained over the

course of both intervention programmes to several TEA sub-

tests; the TEA sub-test scores of the remaining three

participants were too inconsistent to warrant any such claim

associated with the separate or combined intervention

programmes.

Discussion

Variability across participants

Participation in computer-based intervention programmes is

an attractive alternative to traditional treatment approaches

for many individuals with TBI because of features such as low

cost, wide-spread availability, provision of immediate and

unbiased feedback and built-in entertainment [1, 2].

Application of two such programmes in the study reported

herein resulted in two of four adults with severe TBI

demonstrating improved attending behaviour as measured

either through a researcher-generated probe or through a

standardized attention assessment tool. This is encouraging

given that the individuals had severe attention deficits and

were multiple years post-injury. People with less severe initial

injuries, less persistent attention challenges and in more acute

stages of recovery may benefit more. At the same time,

cautious interpretation of findings is necessary, because the

observed gains in participants’ probe scores early in the

intervention process may have reflected practice effects rather

than actual improvement in attention functioning. In particu-

lar, probe repetition may have provided a means of improving

participants’ understanding of the task directions and/or

awareness of task requirements. Furthermore, the results are

discouraging in that participation in the programmes showed

only limited generalization for the two best-performing

participants and no apparent generalized benefits for the

other two participants. These findings call into question the

claims of some researchers and manufacturers of computer-

based cognitive intervention programmes who purport that

the products are helpful to virtually all people engaging in the

activities on a consistent basis [4, 19]. The results also

confirm the suspicions of many researchers who have noted

the ease of demonstrating improvement on specific interven-

tion tasks but the challenge of converting those gains to

everyday functioning [3, 20, 28].

Each participant in the current study demonstrated a unique

pattern of performance in progressing through the intervention

activities and performing the generalization tasks.

8 S. Zickefoose et al. Brain Inj, Early Online: 1–10

Bra

in I

nj D

ownl

oade

d fr

om in

form

ahea

lthca

re.c

om b

y L

ibra

ry o

f H

ealth

Sci

-Uni

v of

Il o

n 05

/01/

13Fo

r pe

rson

al u

se o

nly.

Several factors reflecting the heterogeneity of the TBI

population may have contributed to the observed variability

including individual differences in the extent of neurological

damage, the length of time post-injury and the severity of

attention deficits at the initiation of treatment [8]. In particular,

the participant who appeared to benefit most from the

intervention (i.e. OE) differed from the other participants in

that he was relatively young, had sustained his injury only 3

years prior to study initiation, experienced trauma from the

penetration of a high-pressure pipe through his skull rather

than from a closed head injury and had higher pre-intervention

scores on the TEA and probe measures than the other

participants. In all likelihood, a combination of these factors

contributed to his success in improving his attention and

generalizing those improvements to other tasks. The wide

range across participants post-injury is a limitation of this

study. Additional investigations with homogeneous participant

groups regarding time post-injury are warranted to evaluate the

effects of computer-based intervention programmes during

specific stages of recovery.

Other factors contributing to the obtained results concern

the day-to-day variability of many TBI survivors and the

motivation of individual participants to perform well on the

generalization tasks. In particular, the declining scores

obtained by participant KX on repeated TEA administrations

are difficult to explain in terms other than personal incon-

sistency. No reason exists to assume that participation in the

intervention activities would result in a decline in his

attending skills and no medication, lifestyle or psychological

changes were noted during the course of the research that

would account for his deterioration. As such, frustration with

the difficulty of certain tasks, limited motivation to perform

the testing activities or extreme day-to-day variability warrant

consideration as possible factors influencing his performance.

Factors relating to aspects of the chosen intervention

programmes may also have contributed to the variability

observed across participants and the limited generalization of

improvements. In particular, the researchers determined the

hierarchy for presenting APT-3 tasks and selected the games

for inclusion in the Lumosity� intervention. These decisions

were based on clinical judgement and logical assumptions

about task difficulty and complexity; as such, they are

certainly subject to question. Inclusion of different tasks or a

different sequence of tasks might have yielded better results.

Finally, factors relating to the length and intensity of

treatment may have contributed to the lack of generalization

noted. Specifically, the 30-minute sessions, 5 days per week,

for 4 weeks with each intervention programme may not have

been optimal. Longer or more intense treatment or sessions

distributed over a longer period of time may have fostered

better generalization of improved attending behaviour to

novel activities. In particular, KS and OE’s performances

approaching but not reaching significance on the

probe measures suggest that, had they been given the

opportunity to continue longer, their gains might have

shown greater generalization. This notion is supported by

the reported improvements noted by other researchers [11,

29], who have administered APT interventions with longer

sessions and over longer time periods than was done in the

current study.

The fact that current study participants demonstrated

greater improvement on TEA sub-tests following the first

intervention phase than the second, regardless of the order of

programme presentation, calls into question the idea that

additional intervention time would be of value regarding

generalization to non-experimental tasks, however. The

benefits gained from programme participation appeared

early rather than late in the intervention process. Hence,

assuming that additional treatment time and/or intensity

would facilitate greater generalization of functional changes

in attending behaviour might be erroneous. Indeed, some

researchers (e.g. [9]) have documented no improvement in

brain injury survivors’ attention scores following an extended

period of APT intervention. Ultimately, researchers need to

investigate further the appropriate intensity and duration of

various attention intervention programmes to maximize

positive outcomes for individual survivors.

Preferences for APT-3 vs Lumosity�

Commitment to engaging in attention training activities on a

routine basis is critical to the success of any individual. As

such, having access to a programme with appealing graphics,

animations and sound tracks and that presents activities in a

format bolstering interest and enjoyment can be important. In

the current study, the presence or absence of these types of

features appeared to influence participants’ attitudes toward

the Lumosity� and APT-3 programmes and their desires

regarding continued engagement as reflected through their

Likert-scale ratings. However, the fact that study participants

were not unanimous in their preferences for one programme

over the other confirms the need to consider individual

differences regarding activity difficulty, formatting and pres-

entation options. Important considerations include determin-

ing whether an individual prefers: (a) competing with a

computerized opponent or working independently, (b) parti-

cipating in a game or a drill-and-practice format, (c) having

multiple levels beyond their current ability or being successful

in mastering the highest level available, (d) engaging in

activities of short or long duration, (e) receiving computer-

provided or clinician-provided feedback, (f) having adjust-

ments in difficulty level occur within or between activities

and (g) having considerable or minimal redundancy and

repetition within tasks. Of note, all participants complained

that the intervention activities of both programmes did not

reflect real-life tasks to a satisfactory degree. They appeared

more willing to overlook this lack of functional application—

or at least better tolerate it—when engaged in Lumosity�than APT-3. Perhaps this was because of their perception that

the former activities were mental games with entertainment

value, but the latter were therapy tasks.

Conclusions

The use of tools such as Lumosity� Brain Games and APT-3

to remediate the attention deficits of individuals with TBI is

an area of research requiring further investigation. Indeed,

exploration of the clinical value of brain fitness programmes

is in its infancy regarding the types of individuals most likely

to benefit, the optimum activities for inclusion, and the best

methods of presenting programmes and evaluating outcomes.

DOI: 10.3109/02699052.2013.775484 Remediating attention deficits 9

Bra

in I

nj D

ownl

oade

d fr

om in

form

ahea

lthca

re.c

om b

y L

ibra

ry o

f H

ealth

Sci

-Uni

v of

Il o

n 05

/01/

13Fo

r pe

rson

al u

se o

nly.

Although the results presented herein raise questions regard-

ing the efficacy of programmes such as these, further research

is needed to determine the factors that influence whether

particular individuals will or will not benefit. Certainly,

engagement in brain training and brain fitness programmes is

unlikely to have negative ramifications regarding cognitive

outcomes and may provide enjoyable alternatives to other

leisure activities. Still, consumers and professionals should be

wary of poorly-substantiated, universal claims that such

programmes provide a means of boosting the cognitive

functioning of survivors of TBI.

Declaration of interest

The authors report no conflicts of interest. The authors alone

are responsible for the content and writing of the paper.

References

1. Sullivan K, Goudy L, Prauka M, Quinn J, French L, Teixeira K.Computer-based brain-fitness programs for traumatic brain injuryrehabilitation. Poster presented at the annual convention of theAmerican Speech-Language-Hearing Association, San Diego, CA.November 2011.

2. Pierce SM, Mayer NH, Whyte J. Computer-assisted exercisesystems in traumatic brain injury: Cases and commentary. Journalof Head Trauma Rehabilitation 2001;16:406–413.

3. Owen AM, Hampshire A, Grahn JA, Stenton R, Dajani S, BurnsAS, Howard RJ, Ballard CG. Putting brain training to the test.Nature 2010;465:775–778.

4. Gontkovsky ST, McDonald MB, Clark PG, Ruwe WD. Currentdirections in computer-assisted cognitive rehabilitation.NeuroRehabilitation 2002;17:195–199.

5. Sohlberg MM, Mateer C. Attention Process Training APT-3: Adirect attention training program for persons with acquired braininjury. Youngsville, NC: Lash & Associates Publishing/TrainingInc.; 2011.

6. Sohlberg MM, Mateer CA. Attention Process Training (APT).Puyallup, WA: Association for Neuropsychological Research andDevelopment; 1986.

7. Sohlberg MM, Johnson L, Paule L, Raskin SA, Mateer CA.Attention Process Training II: a program to address attentionaldeficits for persons with mild cognitive dysfunction [rehabilitationmaterials]. Puyallup, WA: Association for NeuropsychologicalResearch & Development; 1994.

8. Sohlberg MM, Mateer CA. Cognitive rehabilitation: an integrativeneuropsychological approach. New York: The Guilford Press; 2001.

9. Park N, Proulx GB, Towers WM. Evaluation of the attentionprocess training programme. Neuropsychological Rehabilitation1999;9:117–130.

10. Sohlberg MM, Mateer CA. Effectiveness of Attention-TrainingProgram. Journal of Clinical and Experimental Neuropsychology1987;9:117–130.

11. Sohlberg MM, McLaughlin KA, Pavese A, Heidrich A, Posner MI.Evaluation of Attention Process Training and brain injury education

in persons with acquired brain injury. Journal of Clinical andExperimental Neuropsychology 2000;22:656–676.

12. World Health Organization (WHO). International classificationof functioning, disability and health. Geneva: World HealthOrganization; 2001.

13. Boman I-L, Lindstedt M, Hemmingsson H, Bartfai A. Cognitivetraining in home environment. Brain Injury 2004;18:985–995.

14. Dakim, Inc. Dakim� Brain Fitness: clinically demonstrated toimprove memory. Santa Monica, CA: Lumos Labs, Inc.; 2010.Available online at: http://www.dakim.com/why-it-works/clinical-trials/, accessed 1 January 2012.

15. Fuyuno I. Brain craze. Nature 2007;447:18–20.16. Kawashima R, Okita K, Yamazaki R, Tajima N, Yoshida H, Taira

M, Iwata K, Sasaki T, Maeyama K, Usui N, et al. Reading aloudand arithmetic calculation improve frontal function of people withdementia. Journals of Gerontology, Series A, Biological Sciencesand Medical Sciences 2005;60:380–384.

17. Smith GE, Housen P, Yaffe K, Ruff R, Kennison RF, Mahncke HW,Zelinski EM. A cognitive training program based on principles ofbrain plasticity: Results from the Improvement in Memory withPlasticity-based Adaptive Cognitive Training (IMPACT) Study.Journal of the American Geriatric Society 2009;57:594–603.

18. Klingberg T, Fernell E, Olesen PJ, Johnson M, Gustafsson P,Dahlstrom K, Gillberg CG, Forssberg H, Westerberg H.Computerized training of working memory in children withADHS—a randomized, controlled trial. Journal of the AmericanAcademy of Child and Adolescent Psychiatry 2005;44:177–186.

19. Hardy J, Scanlon M. The science behind Lumosity. San Francisco,CA: Lumos Labs; 2009.

20. George DR, Whitehouse PJ. Marketplace of memory: what thebrain fitness technology industry says about us and how we can dobetter. The Gerontologist 2011;55:590–596.

21. Lumos Labs. Lumosity: Reclaim Your Brain�. San Francisco, CA:Dakim, Inc.; 2010. Available online at: http://www.lumosity.com,accessed April 3, 2010.

22. Kesler SR, Lacayo NJ, Jo B. A pilot study of an online cognitiverehabilitation program for executive function skills in children withcancer-related brain injury. Brain Injury 2010;25:101–112.

23. Fortuny LA, Briggs M, Newcombe F, Ratcliff G, Thomas C.Measuring the duration of post traumatic amnesia. Journal ofNeurology, Neurosurgery and Psychiatry 1980;43:377–379.

24. Russell WR, Smith A. Post-traumatic amnesia in closed headinjury. Archives of Neurology 1961;5:16–29.

25. Kertesz A. The Western Aphasia Battery – Revised [WAB-R]. SanAntonio, TX: Harcourt Assessment, Inc.; 2006.

26. Robertson IH, Ward T, Ridgway V, Nimmo-Smith I. The Test ofEveryday Attention. Bury St. Edmunds: Thames Valley TestCompany; 1994.

27. Stern RA, White T. Neuropsychological Assessment Battery. Lutz,FL: Psychological Assessment Resources, Inc.; 2003.

28. Papp KV, Walsh SJ, Snyder PJ. Immediate and delayed effects ofcognitive interventions in healthy elderly: A review of currentliterature and future directions. Alzheimer’s and Dementia 2009;5:50–60.

29. Palmese CA, Raskin SA. The rehabilitation of attention individualswith mild traumatic brain injury, using the APT-II programme.Brain Injury 2000;14:535–548.

10 S. Zickefoose et al. Brain Inj, Early Online: 1–10

Bra

in I

nj D

ownl

oade

d fr

om in

form

ahea

lthca

re.c

om b

y L

ibra

ry o

f H

ealth

Sci

-Uni

v of

Il o

n 05

/01/

13Fo

r pe

rson

al u

se o

nly.