Embed Size (px)
Citation preview
The Testing Industry
Its Evolution and Future Directions
Louis Woodruff
NCTA Conference
San Antonio Texas
August 7, 2009
Texas is as large as all of New England
New York, Pennsylvania, Ohio
and Illinois combined
San Antonio TexasSan Antonio Texas
Texas includes 267,339 square miles.
7.4% of the nation's total area
There are more than 70,000 miles
of highways in Texas
The world’s largest parking lot
is located at DFW Airport
The Airport is larger than Manhattan
Texas is home to Dell and Compaq
Central Texas is often referred to as
the Silicon Valley of the South
The name “Texas” comes from an
Indian word meaning friends or allies
Texas was an independent nation
from 1836 to 1845
Texas is the only State
to enter the nation by treaty
instead of annexation
The last battle of the Civil War
was fought in Texas
It was a Confederate victory
That same year…1865…forty years
before the Wright brother’s historic flight
a Texan flew a coil-spring-powered airplane
The craft soared to tree-top heights
before crashing into a henhouse
killing several chickens and
scaring many children
Rodeo is the official Texas State sport
World's 1st Rodeo was in Pecos Texas
July 4, 1883
Just two years after the 1st Rodeo
Dr Pepper was invented in Waco
There is no period after the Dr in Dr Pepper
75% of the world's Snickers bars
are made in Waco
More popcorn is sold in Dallas, per capita,
than anywhere in the U.S.
The average American eats
42 quarts of popcorn annually!!
The average American eats
42 quarts of popcorn annually!!
The first word spoken from the moon
was
"Houston"
on July 20, 1969
The armadillo is the official State mammal
In Texas, for $150 you can become
a licensed dead animal hauler
In Texas it's illegal to put graffiti
on someone else's cow
TACTP
Test
NCTA
Exam
s
CCTC
There are 18 million people in Texas
and 16 million cattle
In “Texan” the plural of y’all . . . is all y’all
Texas is a national leader in the alignment
of elementary and secondary
curricula and assessment
Texas Association of Collegiate Testing Personnel
Founded 1988
Administration of educational and psychological testing andassessment programs at Texas colleges and universities
Promote, implement, and evaluate professional standards
Disseminate information to professional testing personnel
Promote continuing education and foster public understanding
professional testing personnel
The Testing Industry
Testing Professional
Not Your College Major
Not Your Original Plan
How Big is it?
The Testing Industry
• in 1997 - $253,000,000
• in 1955 - $7,000,000
• in 2007 - $700,000,000
Recently Reported Size
of the Testing Market
The Testing Industry
True
FalseX
$700,000,000
Recently Reported Size
of the Testing Market
The Testing Industry
Educational Testing Service
Recently Reported Size
of the Testing Market
$700,000,000 - Reported Annual Cash Flow
$50,000,000 - Contract in California
The Testing Industry
$700,000,000 ??
Likely more than triple that size . . .
. . . and growing ! ! !
Recently Reported Size
of the Testing Market
The Testing Industry
Multi-Billion Dollar Industry
Large Diversified Market
Supporting Technologies
Future Growth PotentialProfit O
pportunitie
s ?
Profit Opportu
nities ?
Low Margins
Losses
Limited Capital
Complexand
CostlyDevelopmen
t
ExpensiveDemanding
InfraStructure
DiminishingEconomies
of Scale
Low Margins
Losses
Limited Capital
Complexand
CostlyDevelopmen
t
ExpensiveDemanding
InfraStructure
DiminishingEconomies
of ScaleHOW DID WE
GET INTO
THIS SITUATION ?
What role has Academia (the colleges)
played in the evolution of the Industry?
What were the early forces that shaped
the modern standardized testing market?
What market forces & business dynamics
are driving the Testing Industry today?
Are there lessons to be learned from the
past that can be applied in the present
to shape a better future for our Industry?
In 1845 Boston schools applied“uniform” testing for all studentsthrough oral examinations.
In 1874 the superintendentin Portland Maine, createda uniform curriculum with “standardized” tests.
He published the scores.
Soon after thathe was removed!
A well intentioned action, ahead of its time,may be throttled and pounded into extinction.
An important lesson for those seeking change!
OK…but are there any business lessons from the past for the testing industry?
There is a lesson from the past
for today’s Testing Industry . . .
to be learned from the
history of the #2 pencil !
Slates and chalk continued to beused after the pencil was inventeddue to high costs for the manualproduction of pencils
In 1866 a patent was grantedfor a wood-plane machineenabling mass production . . .
$ $ $
. . . the “new technology” (the pencil) was soonembraced in the marketplace
Within 4 years over20 million pencilswere sold
This enabled the unitcost of the pencil tobe lowered . . .
By 1903 the eraserwas addedand the cost dropped toa penny
$ $ $
$ $ $
. . . this is the same situation the testing industry has repeatedly confronted . . .
$ $ $
$ $ $
Having the technology is not enough.
Through mass
production and
robust distribution
channels, manufacturing
has to provide the market
the “right” unit cost !
Having what the
market needs is
not enough.
$ $ $
$ $ $
FundingSources
ProductionCapability
MarketNeed
EnablingTechnology
DistributionChannels
ViableProductLow Unit
Price BusinessMandate
we should first look at the forces that
gave birth to today’s testing industry?
Before we look at how the testing industry
has handled this business mandate . . .
Today’s testing industry
grew out of Academia in
pursuit of scientific missions!
Edward L Thorndike applied the scientific method to the constructionof tests to measure academicperformance.
Lewis Terman at Stanford created the Intelligence Quotient (IQ) by dividing mental age by chronological age.
The College Entrance Examination Board published subject syllabi for schools to teach and colleges to test.
Within two decades multiple
standardized tests were in use.
These tests were developed by
Academics within the Universities.
Those Universities supported the
costs to develop these tests as well
as the cost to bring them to market.
Researchers at University of Iowadeveloped the Iowa Test of Basic Skills.
The Stanford Achievement Tests were created to assess subject knowledge.
The Scholastic AptitudeTest was introduced.
The CaliforniaAchievement Test was published.
Universities and the CollegeBoard were operating their own commercial testing business.
However, hand scoring of answer sheets
and manual processing of score reports
kept costs high and limited use.
By the mid-1930’s the fledgling industry
was reaching its capacity for growth.
The testing industry, birthed through Academia,
Academia’s
Business Challenge
Measure
Academic
AchievementThe Universities Scientifically Based
Academic Assessments
was facing the same challenge faced by the #2 pencil !
Still Needed:
Enabling Technology
Production Capability
Distribution Channels
It was the business sector
that developed the
Enabling Technology
required to drive things forward
$ $
$ $
In 1938 IBM created the first electronicgraphite/optical test scoring machine.
Technology
In 1938 IBM created the first electronicgraphite/optical test scoring machine.
While this financially helped Academia’s testing initiative, the required business mandate remained incomplete.
Enhanced Production Capability and improved Distribution Channels werestill needed to complete “The Model”.
The military funded development of new assessments to support recruitment and the war effort.
This placed a burden on the nation’s colleges who had no means for assessing the readiness of these non-traditional students.
Following the war, the GI Bill provided money for returning soldiers to further their education and attend college.
The Chauncey Farm
American Councilon Education
The Carnegie Foundation
The decentralized testing services could not handle the demands of such an admissions challenge.
A group of influential Universities approached the College Board to level the admissions playing field.
The College Board asked its VPto develop a testing operation to meet this new national demand.
ACT was founded as a second non-profit testing organization to meet the new market demands.
The testing industry, centered in these large non-profit institutions, both flourished and profited !
FundingSources
ProductionCapability
MarketNeed
EnablingTechnology
DistributionChannels
ViableProduct
Low UnitPrice
Increasing Demandfor Assessment
Private and Public Funding
Valid and Reliable Tests Automated Scanning Scoring and Reporting
Measures performance relative to a
representative national sample of
examinees called “norming sample”
Test Item DatabaseTest Item Database
Norm-Referenced Test
Test ItemStatistics
Test FormEquating
Test ItemRotation
Test Design
Validity &Reliability
% of Reusable Test Items
Reconstructed Test Forms
Large National Volume
FundingSources
ProductionCapability
MarketNeed
EnablingTechnology
DistributionChannels
ViableProduct
Low UnitPrice
DistributionChannels
Labor forTest Delivery
InternationalPartners
Partnerships for low cost services and facilities
FundingSources
ProductionCapability
MarketNeed
EnablingTechnology
DistributionChannels
ViableProduct
Low UnitPrice
National Norm Referenced
Paper & Pencil Testing
What changed the testing industry
from a non-profit service focus
to a for-profit commercial focus?
$
$
$$
$
$
$
$
$
$$
$
$
$
$ $
$
So then why are the vendors
in today’s Testing Industry not
making substantial profits?
Service Issues
Delayed Reporting
Shipping Problems
Processing Delays
Increasing Costs
Scoring Errors
How did the industry get into this situation?
Is there a way out of it?
CBT significantly improved
the test product . . . but it
limited production capabilities, required
a new distribution infrastructure and
raised operational and development costs
How did the industry get into this situation?
raising client expectationsbeyond what the vendors could profitably deliver
The delivery advantage of using schools(inexpensive classrooms for large group testing)
lost its business value for CBT.
CBT altered the business dynamics in the testing industry by reducing profit margins, expandingcompetition, and raising client expectationsbeyond what the vendors could profitably deliver.
Store-front commercial outlets as a new distribution channel.
http://www
Even internet delivery did not resolve things
Internet Internet Delivery Delivery
Additional competitors emerged
Cost savings have been minimal
Test development and overall operating costs remain higher than traditional P&P testing
FundingSources
ProductionCapability
MarketNeed
EnablingTechnology
DistributionChannels
ViableProductLow Unit
Price
CBT-IBT disrupted the
dynamics of the P&P business model
CBT
Just eight years ago it appeared
that things were about to change!
The “No-Child Left Behind” legislation
gave the Testing Industry hope for a
return to the profitable P&P model.
The public schools could not support
CBT for such large volumes, leaving
P&P delivery as the most viable option.
An expanded paper & pencil
market could provide the
foundation to support the
CBT side of the business !
The Industry leaders scrambled to
win the State contracts to provide
this federally mandated testing,
expecting significant future profits.
Low cost P&P delivery Less costly P&P test development High test volumes Economies of scale
What turned the NCLB golden goose
into a turkey for the Testing Industry?
Individual State specs
Lower per test volumes
State owned test items
Short reporting window
Delivery by the client
Low economies of scale
Increased competition
All of this
impacts
all of us!
Measure performance relative to a
representative national sample of
examinees called “norming sample”
Norm-Referenced Tests
Criterion-Referenced Tests
Measure performance relative
to mastery of content at certain
pre-defined levels of proficiency
At the same time, CBT & IBT continue
to stress the industry with increasing
client demands, costly operations,
and expanding competition.
NCLB testing, even with over 33,000,000
tests projected for this year, continues
to be a losing venture for many vendors.
Is there some new technology that
will again catapult the Industry
forward
into a new and promising paradigm?
Is there a silver bullet that can give the
Testing Industry the profitability needed
to remain stable and effective?
Four Requirements to Promote
this New Test Delivery Paradigm
Competition must drive down the cost of Laptops.
Wireless Technology must make affordable advances
in security and control.
Demand for hand-held full function computers
must reach mass production levels.
Education must integrate personal hand-held
wireless computers into the standard curriculum.
Test Delivery Unit #4
Laptop Server + IDD’s
Test Delivery Unit #3
Laptop Server + IDD’s
Test Delivery Unit #2
Laptop Server + IDD’s
Test Delivery Unit #1
Laptop Server + IDD’s
Laptop Server Hub
Host Server
Laptop #1
Laptop #2
Laptop #3
Laptop #4
Test Server #1 (laptop)
Testing Room #4
Test Server #4 (laptop)
Testing Room #3
Test Server #1 (laptop)
Testing Room #2
Test Server #1 (laptop)
Testing Room #1
Test Server #1 (laptop)
A Possible Model Data Source
Host and Laptop Servers
stored in movable cabinet
Laptop servers & Individual
Display Devices transported
in mobile Test Delivery Units
After testing in classrooms
laptop servers returned to
Host for test data upload.
Test data download
via the internet
A Test Delivery Paradigm ShiftA Test Delivery Paradigm Shift
Old Model in a New DimensionOld Model in a New Dimension
Returning the Test DeliveryReturning the Test Delivery
Business AdvantageBusiness Advantage
to Partnershipsto Partnerships
with Schoolswith Schools
Can we really know how our Industry
will evolve over the next 20 years?
What changes may lie ahead ? What changes may lie ahead ?
How might technology shape the journey ?How might technology shape the journey ?
Confirming Examinee Identity Confirming Examinee Identity
Continuously During Testing Continuously During Testing
Confirming Examinee Identity Confirming Examinee Identity
Continuously During Testing Continuously During Testing
ComputerizedComputerized PaperPaper
Immediate Immediate
On-SiteOn-Site
Answer SheetAnswer Sheet
andand
Essay ScoringEssay Scoring
Immediate Immediate
On-SiteOn-Site
Answer SheetAnswer Sheet
andand
Essay ScoringEssay Scoring
ComputerizedComputerized
“ “Test Coat” Test Coat”
for for
continuouscontinuous
electronic electronic
monitoringmonitoring
ComputerizedComputerized
“ “Test Coat” Test Coat”
for for
continuouscontinuous
electronic electronic
monitoringmonitoring
Quantum Computing EnabledQuantum Computing Enabled
Photon State EncodingPhoton State Encoding
Quantum Computing EnabledQuantum Computing Enabled
Photon State EncodingPhoton State Encoding
Current data encryption algorithms rely
on computational schema for protection
and can be deciphered, given sufficient
computing power, knowledge and time.
Current data encryption algorithms rely
on computational schema for protection
and can be deciphered, given sufficient
computing power, knowledge and time.
The Heisenberg Uncertainty PrincipleThe Heisenberg Uncertainty Principle
It is impossible to discover both
the momentum and position of a
particle at any given instant in time.
It is impossible to discover both
the momentum and position of a
particle at any given instant in time.
Therefore, data encryption using a
particle in motion as a key could
never be deciphered.
Therefore, data encryption using a
particle in motion as a key could
never be deciphered.
Quantum Computing EnabledQuantum Computing Enabled
Photon State EncodingPhoton State Encoding
(particle motion as the encoding key)(particle motion as the encoding key)
Data SecurityData Security
No Longer a Concern !No Longer a Concern !
Data SecurityData Security
No Longer a Concern !No Longer a Concern !
Projection of a laser generated3-D image suspended in space
A future test center with
individualized testing
environments supporting
holographic images as
interactive test stimuli.
A future test center with
individualized testing
environments supporting
holographic images as
interactive test stimuli.
A Virtual Testing EnvironmentA Virtual Testing EnvironmentA Virtual Testing EnvironmentA Virtual Testing Environment
In the future we may be able to
reach total standardization through
the management of each examinee’s
perceptions during the testing event.
In the future we may be able to
reach total standardization through
the management of each examinee’s
perceptions during the testing event.
Will future advances in brain mapping
and Psychopharmacology enable
selected modifications of the brain’s
perceptions to achieve absolute virtual
standardization - from the inside out?
Will future advances in brain mapping
and Psychopharmacology enable
selected modifications of the brain’s
perceptions to achieve absolute virtual
standardization - from the inside out?
Virtual Testing EnvironmentVirtual Testing EnvironmentVirtual Testing EnvironmentVirtual Testing Environment
Transmitting an Ideal, Standardized,
Virtual, Personal Testing Space,
Integrated with the Test
Transmitting an Ideal, Standardized,
Virtual, Personal Testing Space,
Integrated with the Test
StandardizationStandardizationas an as an
Altered State ofAltered State ofConsciousness !Consciousness !
StandardizationStandardizationas an as an
Altered State ofAltered State ofConsciousness !Consciousness !
the road aheadmay look lonelythe road aheadmay look lonely
and at times wemay feel unsureand at times wemay feel unsure
not knowingwhich turn to takenot knowingwhich turn to take
We will all need help
to navigate through
the unknowns ahead
ITS ALWAYS BETTER WITH A FRIEND !
ITS ALWAYS BETTER WITH A FRIEND !
Louis Woodruff
NCTA Conference
San Antonio Texas
August 7, 2009
Contact:
215-968-8467
The Testing Industry
Its Evolution and
Future Directions
