Beyond TRL: A Revised Model of Technology Development and ... · Beyond TRL: A Revised Model of Technology Development and Considerations for Programmatic Analysis Prof. Zoe Szajnfarber

Beyond TRL: A Revised Model of Technology Development and Considerations for

Programmatic Analysis Prof. Zoe Szajnfarber

Assistant Professor of Engineering Management & Systems Engineering

and Space Policy, The George Washington University

Research Affiliate, MIT Engineering Systems Division

NASA Cost Symposium

NASA Langley – August 14, 2014

On Technology Cost Estimating

• Technology development– or the focused long term development of – is a major part of R&D at NASA.

• Technology Cost Estimating is a relatively unexamined field in the academic literature on cost estimating

• NASAs recent efforts to fund technology cost estimating research have been helpful in understanding how technology develops (Cole et al 2013, 2014)

• Our focus is not on technology cost estimating: we study the process of technology development itself

• However, we hope our research can provide insight for the cost and scheduling community

© 2014 SzajnLab

© 2014 SzajnLab

21995 1990 1985

History of Shifts in R&D Strategy

(1986-1992)

R&T in Office of Aeronautics

and Space Technology

(OAST) = Basic, cross-

cutting research (~250M)

(1992-1994)

Aeronautics

to Office of

Aeronautics

(1992-1994)

Space to Office

of Advanced

Concepts

(OSAT, ~200M)

(1996-1999)

Cross-Enterprise

Technology

Development

(under Code S)

(1996-1998)

Office of Space

Science

(~100M)

(1999-2001)

Office of

Aerospace

Technology

(OAT) ~$40M NRA

~$100M

cross-cutting

(2001-2004)

“Pioneering

Revolutionary

Technology” ~$100M

Advanced

~$180M

IT/Comm.

(1998-2007) NIAC

(2004+)

Exploration

Techn ology

Development

program

~75% cut in

2005

Basic

P

roje

ct-

specific

?

(Based on data collected for NASA R&T Study and NRC study of NIAC)

NEED: To control the system better, we need to understand it better.

© 2014 SzajnLab

Guiding Research Questions

NEED: To control the system better, we need to understand it better.

1. How do new capabilities traverse the innovation system as they are matured and infused into flight projects?

• Empirically grounded models of the innovation process

• Considers technical, social and political factors

• Can this process be predicted/estimated?

2. To what extent can the process be improved through feasible management interventions?

• Exploring organization configuration as a design lever

• Design for evolvability/tinkerability

• Improved incentive systems, based on valid preference structures.

• Balanced technology investment strategies that acknowledge key attributes of space innovation ecosystem

© 2014 SzajnLab

NASA Innovation Landscape

Political-level context

Agency-level planning

Project-level Development & Implementation

Technology-level Research & Development

Scientific and Technical State-of-the-art

© 2014 SzajnLab

NASA (Space Science) Innovation Landscape

Political-level context

New mission

concepts

Prioritization: E.g., Decadal

Survey

Selection: E.g., 1

Flagship; 4 Ex

Concept Studies

Phase A: Concept & Tech Development

Phase B: Prelim design

NRA

Center IRAD

Branch

Center IRAD

Branch

Center IRAD

Branch

Center IRAD

Branch

Center IRAD

Branch

Center IRAD

Branch

Center IRAD

Branch

Center IRAD

Branch

Center IRAD

Branch

NRA NRA NRA NRA

Center IRAD

Branch

Center IRAD

Branch

NRA

Scientific and Technical State-of-the-art

© 2014 SzajnLab

Current Conceptualization: Stage-Gates

Concepts

Basic R&D Applied R&D Project-specific

Tech Dev.

Shelved

capabilities Shelved

capabilities

Shelved

concepts

Flight

Innovation as an Optimization Problem Relative resource allocation problem (how much money in each bucket?) Resources spacing problem (how many buckets? Gate criteria definition problem (how many should be advanced, and by what criteria?)

*Synthesized from NASA strategic planning documents 1990-2006

Maturity (TRL)

© 2014 SzajnLab

Applied R&D

RTOP RTOP

Project-specific

Tech Dev.

Basic R&D

DDF

IRAD

SBIR P1

SBIR P2

Office of

Commercialization

NRA

ROSES

ESTO

MPTO

Pre/

Phase A

Pathfinder

missions

SBIR P2

DDF

IRAD

SBIR P1

Office of

Commercialization

NRA

ROSES

ESTO

MPTO

Pre/

Phase A

Pathfinder

missions

Concepts

Shelved


capabilities

Flight

Shelved

concepts

3

4

5

6

7

8

9

1

9a

Legend New capability

Event

Action

Decision

Contract

New technical concept

First detector

arrays

New Camera Prototype

Space-qualified QWIPs

Chance

encounter

Contacts

Called

Readiness

Communicated

Technical

problems on

project

Congress:

must fly IR

camera!

2 Collaboration

policy

Actual Complexity of Process

Scientist

recruited

0

Maturity (TRL)

© 2014 SzajnLab

Stage-Gate Assumptions

Concepts Basic R&D Applied R&D Project-specific

Tech Dev.

Shelved


capabilities

Shelved

concepts

Flight

Maturity (TRL)

*Synthesized from NASA strategic planning documents 1990-2006

Underlying assumptions:

(1) Technologies mature from left to right over time;

(2) Stages are mutually exclusive (at a given time);

(3) Shelving is an active process, controlled by decision makers;

(4) Shelf life is passive and a function of technical obsolescence.

© 2014 SzajnLab

19

90

20

00

20

10

198

2

198

3

198

4

198

5

198

6

198

7

198

8

198

9

199

1

199

2

199

3

199

4

199

5

199

6

199

7

199

8

199

9

200

1

200

2

200

3

200

4

200

5

200

6

200

7

200

8

200

9

DDF

CETDP

DDF

DDF

DDF IRAD IRAD IRAD

Con-X Con-X

CTD

IPP Seed

Astro-H

PIXIE

AXAF Astro EAstro E2

RTOPS

DDF

SR&T SR&T

DDF DDF

P1

SBIR P2

NMSC

BASIS

BASIS

NDE

P1

SBIR P2

SWIFT

DDF

APPRA

DDFDDFDDF

SR&T

DDF DDF

DDF

IRAD

IRAD IRAD IRAD

APPRA

Cat “3” PhA

AXP GEMS

P1

SBIR P2

DDF

AXAF-NRA

AXAF “-S” Astro-E Astro-E2 Astro-H

Con-X NRA

Con-X IXO

NRA NRANRANRA

NeXT

NRA NRA NRA

CETDPCCT

DDF DDF IRAD IRAD

NRA

DDF DDF IRAD IRAD

IRAD

IRAD

Switchbacks in Maturity

Applied R&D

Project-

specific Tech

Dev.

Basic R&D

Time

vs. Observation:

switchbacks

Matu

rity

(T

RL)

© 2014 SzajnLab

19

90

20

00

20

10

198

2

198

3

198

4

198

5

198

6

198

7

198

8

198

9

199

1

199

2

199

3

199

4

199

5

199

6

199

7

199

8

199

9

200

1

200

2

200

3

200

4

200

5

200

6

200

7

200

8

200

9

DDF

CETDP

DDF

DDF

DDF IRAD IRAD IRAD

Con-X Con-X

CTD

IPP Seed

Astro-H

PIXIE

AXAF Astro EAstro E2

RTOPS

DDF

SR&T SR&T

DDF DDF

P1

SBIR P2

NMSC

BASIS

BASIS

NDE

P1

SBIR P2

SWIFT

DDF

APPRA

DDFDDFDDF

SR&T

DDF DDF

DDF

IRAD

IRAD IRAD IRAD

APPRA

Cat “3” PhA

AXP GEMS

P1

SBIR P2

DDF

AXAF-NRA

AXAF “-S” Astro-E Astro-E2 Astro-H

Con-X NRA

Con-X IXO

NRA NRANRANRA

NeXT

NRA NRA NRA

CETDPCCT

DDF DDF IRAD IRAD

NRA

DDF DDF IRAD IRAD

IRAD

IRAD

Applied R&D

Project-

specific Tech

Dev.

Basic R&D

Time

vs. Observation:

switchbacks

Matu

rity

(T

RL)

Switchbacks in Maturity

DDF DDF

DDF

DDF IRAD IRAD IRAD

CTD

DDF DDF DDF

P1 P1

Con-X IXO

© 2014 SzajnLab

Passive Gates, Active Shelves

• Expectation (assumptions #3 and 4):

3. Rejection at Gate => Shelving

4. Similar shelf lives for similar technologies

• Observation:

Case Rejected

+ Shelf

Rejected

+ !Shelf

!Rejected

+ Shelf

Duration on

Shelf

Tech A 1 1 1 8 /1yrs

Tech B 0 2 1 5 yrs

Tech C 0 3 0 N/A

Tech D 0 2 1 2 yrs

Tech E 1 Multiple 1 2 / 5 yrs

Tech F 0 multiple 0 N/A

Szajnfarber, Z., and Weigel, A. L. (2012). "Managing Complex Technology Innovation: the need to move

beyond stages and gates" International Journal of Space Technology Management and Innovation, 2(1), 30-48

Need: More nuanced understanding of underlying processes

© 2014 SzajnLab

Building Theory from Case Studies

Process

Data

~100 hrs interviews

~150 archival

documents

~2 months informal

observation

Within-case

Analytical Chronologies

(Pettigrew 1990)

1 1997

2 10/1/1998 1998

3 1999

4 9/1/1999 1999

5 10/1/1999 1999

6 10/1/1999 1999

7 9/1/2000 2000

8 10/1/2000 2000

9 10/1/2000 2000

10 late 2000

11 2000

2001d

y

t

re

Interview se 12 early 2001

with KB ng

to

ey

ke

13 6/7/2001 2001

14 4/6/2001 2001

“sense-making” Cross-case theory

building

Structured Visual Map (per Langley 1999)

Characteristic Epochs

Transition inducing Shocks

Technology

Exploration

Architectural

Exploration

Treading

Water and

Branching Out

Exploitation

DevO

p

TechB

Context

Join (-)

Context

Funding (+/-)

DevOp

Approved

DevO

pN

eed

TechA

TechA

Con

text

Fundin

g (+

/-)

DevOpNeed

TechA

Technology

graveyard

Flight

Gestation

Period

DevOpNeed

TechA

Con

text

TechB

TechB

Path Initiation Path Termination

Epoch-Shock Model

SR&T, int with

KB and KJ

MOXE ongoing, LOBSTER just s tarting, us ing gas proportional counters to do a l l sky

monitoring. KB gets connected with KJ (LOBSTER) and SH (interested in Gamma-ray

telescope)

DDF proposal

Awarded: Proposal to explore use of "wel l" microstructure recently patented by Solberg,

Pi tts and Walsh for astronomica l gas detectors . Appl ication: a l l sky monitors ; "The micro-

wel l detector replaces the thin wires of the traditional multi -wire proportional counter

with metal electrodes mounted on oppos i te s ides of a polymer fi lm. The polymer i s then

micro-machined to expose the anodes to the cathodes , establ ishing a two-dimens ional

array of anode-cathode pairs that act as gas proportional counters . The anodes and

cathodes provide an orthogonal coordinate readout with equal resolution in each

dimens ion."

Interview

with KB

KB s truggl ing with making detector work, someone says PDJ good at that - tel l s him to

clean i t, they s tart working together

DDF reportChange: Found that idea wasn't feas ible, instead focused on TFT readouts - >

col laboration with PSU

SR&T

proposal ,

interview

with KB

Rejected: track immaging technology for gamma-ray telescopes . Us ing TFTs col laboration

with PSU

DDF proposal

Awarded: proposal to explore use of sol id-s tate instead of imaging gas micro-wel l

detector. "Construction of a sui table thin window for such a gas detector remains a

technica l chal lenge. We therefore propose to investigate an a l ternative development

path: a sol id-s tate substi tute for the gas in the microwel l detector"

DDF report

Bui l t TFT and integrated i t with micro-wel l detector: S. D. Hunter, et a l . 1999, “Imaging

Micro-wel l Proportional Counters Fabricated with Masked UV Laser Ablation,” Proc. 5th

Int. Conf. on Pos i tion-Sens i tive Detectors , to appear in Nucl . Inst. Meth.

K. Black, et a l . 2000, “Imaging Micro-wel l Detectors for X and g-ray Appl ications ,”

SPIE. 4140-33, in press .

J.R. Huang, et a l . 2000, "Active-Matrix Pixel i zed Wel l Detectors on Polymeric

Substrates , " Proc. National Aerospace and Electronics Conf., Oct 10-12, 2000, Dayton

OH, NAECON 2000. (best paper)

DDF proposaldirect continuation of DDF 1999, but focus on pourous dielectric and colaboration with

Adelphi more than PSU. PSU continued with SR&T

SR&T

proposal ,

interview

with KB

Awarded: same as above. Funding for 2000-2002; never managed to get the TFTs working

though

interview

with KB

memory of conversation with PDJ: thought these detectors were important but needed to

justi fy investment with future appl ications . Pursuing Lobster and gamma-ray miss ion.

Thought polarimeter could be good (science director asking us to solve that problem

every year!). Did some back of the envelop ca lculations and proved to ourselves that i t

wouldn't work for polarimetry... so we dropped that

(Martoff et.

a l . 2000)

NIM A - demonstrated an innovative technique for di ffus ion suppress ion in multi -wire

dri ft chambers with electronegative gas additives

DDF report Adelphi got SBIR through FY2003 to pursue purous dia lectric for thin fi lm window

"It’s kinda funny. That group has been working on polarimeters for a whi le and ha

been doing i t with s trip-readout (one dimentional read out) and they’d never rea l l

gotten anywhere. In their papers i t a lways sa id that i f we had a pixi lated readout i

would be great. And I had never pa id any attention … Then they submitted this natu

paper and somebody here – who I knew – was asked to review the paper… and becau

of my expertise they gave i t to me to take a look… Assuming i t was yet another nothi

result, I took i t home and forgot about i t unti l i t was bed time... when I got around

reading i t. That was a mistake because i t got me so exci ted that I couldn’t s leep! Th

had fantastic results reading out these detectors in a pixel i zed fashion. We could ma

polarimetry work!"

(Costa et a l

2001)

Demonstrate polarimeter based on photoelectric effect - use micropattern gas counters

read-out in pixel i zed way. (Neon-based gas , GEM, MPGC digi ta l read-out)

APRA

proposal

Acepted: develop polarimeter based on Costa (result not yet publ ished?) Increased

active area by factor of 50 (KB says i t's wasn't funded, but found reference in 2006 APRA

that says i t was)

Incident# Date Year Data Source Description

Event Database (Van de Ven et al

1990; 2000)

Case Funding Personnel Technology

Co

mp

on

ent

Ex

plo

rati

on

CADR#1 4xCenter team + Inst - Tech parallel component paths

CZT#2 3xCenter + 3xNASA +

Balloon

team +4xTech

+Inst multiple technique strategies

Pol#3 Brainstorm + 2xCenter +

3xNASA team + Tech multiple readout strategies

Si#4 NASA + Project team + 3xInst +

Tech - 3xObs

multiple materials and techniques

tried

Si#5 2xCenter + 2xNASA +

Sounding Rocket + Project team + Tech


tried

Si#6 2xCenter + NASA + SR

+2xProject no change

multiple readout strategies and

techniques tried

TES#7 Branch +3xCenter + 2xNASA

+ SR + Project team + Tech

Exploration of new materials and

techniques


Co

mp

on

ent

Ex

plo

rati

on



Balloon

team +4xTech





Tech - 3xObs


tried




tried




techniques tried




techniques


Co

mp

on

ent

Ex

plo

rati

on



Balloon

team +4xTech





Tech - 3xObs


tried




tried




techniques tried




techniques


Co

mp

on

ent

Ex

plo

rati

on



Balloon

team +4xTech





Tech - 3xObs


tried




tried




techniques tried




techniques

© 2014 SzajnLab

Treading Water and

Branch Out

Epoch-Shock Model: Track View

• System exhibits epochs of persistent stable (and identifiable) behaviors punctuated by transition inducing shocks

– Epochs are illustrated as boxes, and roughly map to stages

– Shocks induce transitions following arrows from one box to another

Treading

Water and

Branch Out

Technology

Exploration

Architectural

Exploration Exploitation

Technology

Exploration

Architectural


Technology

graveyard

Gestation

Technology

graveyard

Flight

Basic

R&DApplied

R&D

Project-

specific

Shelf ShelfShelf

Flight

Gestation


Szajnfarber, Z., and Weigel, A. L. (2013). "A process model of technology innovation in governmental

agencies: insights from NASA's science directorate" Acta Astronautica, 84(3-4), 56-68

© 2014 SzajnLab

Treading Water and

Branch Out


• System exhibits epochs of persistent stable (and identifiable) behaviorspunctuated by transition inducing shocks

Technology

Exploration

Technology

graveyard

Initiation Path Termination Basic R&D STAGE • Low TRL

• <$100K• Cen ter-level

Technology Exploration EPOCH • Patchwork of funding sources• Small core team; ad hoc collaborations• Multiple parallel technology paths


Tec

hn

olo

gy

Ex

plo

rati

on

CADR#1 4xCenterteam + Inst -

Techparallel component paths

CZT#23xCenter + 3xNASA +

Balloon

team +4xTech

+Instmultiple technique strategies

Pol#3Brainstorm + 2xCenter +

3xNASAteam + Tech multiple readout strategies

Si#4 NASA + Projectteam + 3xInst +

Tech - 3xObs

multiple materials and

techniques tried

Si#52xCenter + 2xNASA +

Sounding Rocket + Projectteam + Tech

multiple materials and

techniques tried

Si#62xCenter + NASA + SR

+2xProjectno change


techniques tried

TES#7Branch +3xCenter +

2xNASA + SR + Projectteam + Tech


techniques

Architectural Exploration Exploitation

Gestation

Flight atio

nor

plEx

y og

olTe

chn

Case PersonnelFunding Technology

CADR#1 4xCenter team + Inst -Tech parallel component paths

CZT#2 3xCenter + 3xNASA + Balloon

team +4xTech +Inst multiple technique strategies

Pol#3 Brainstorm + 2xCenter +3xNASA team + Tech multiple readout strategies

Si#4 NASA + Project team + 3xInst + Tech - 3xObs

multiple materials and techniques tried

Si#5 2xCenter + 2xNASA + Sounding Rocket + Project team + Tech multiple materials and

techniques tried

Si#6 2xCenter + NASA + SR +2xProject no change multiple readout strategies and

techniques tried

TES#7 Branch +3xCenter + 2xNASA + SR + Project team + Tech Exploration of new materials and

techniques

© 2014 SzajnLab


• System exhibits epochs of persistent stable (and identifiable) behaviorspunctuated by transition inducing shocks

– Epochs are illustrated as boxes, and roughly map to stages

– Shocks induce transitions following arrows from one box to another

– Innovation pathways start in gestation and move through the system.

Technology

Exploration

Architectural

Exploration

Gestation

Technology

graveyard

Flight Exploitation


Readiness

Communicated

Technical

problems on

project

Congress:

must fly IR

camera!

Chance

encounterContacts

Called

QWIPs detector arraysCollaboration

policy

© 2014 SzajnLab

Epoch-Shock Model: Paths Traveled

• Overlay of ALL the transitions from the pathways studied

– Bi-directional and heavy flow between Technology and Architectural exploration.

– Flow through Exploitation forks between Treading Water and Flight

Technology

Exploration

(8)

Architectural

Exploration

(11)

Exploitation

(11)

Treading

Water

(4)


Gestation

(5)

Technology

graveyard

Flight (4)

© 2014 SzajnLab

Epoch-Shock Model: Paths Traveled

• Overlay of ALL the transitions from the pathways studied

– Colors differentiate different types of shocks, some of which are more controllable by management interventions

– Combined shocks are possible (e.g., red + blue = purple)

Architectural

Exploration

(11)

Exploitation

(11)

Treading

Water

(4)


Gestation

(5)

Technology

graveyard

Flight (4)

Technology

Exploration

(8)

Technology

Missions

Context

Actions

© 2014 SzajnLab

Implications:

Stage-Gate-based management strategies suppress

important dynamics. The Epoch-Shock view

provides a basis for feasible, productive intervention.

© 2014 SzajnLab

Treading Water and Branch Out

Treading

Water and

Branch Out

Technology

Exploration

Architectural


Technology

Exploration

Architectural

ExplorationExploitation

Technology graveyard

Gestation

Technology graveyard

Flight

Why Stage-Gates Can’t Work

Stage-Gate

Current control mechanisms

1. Proportionally morefunding for basic R&D toincrease pool of early-stage concepts.

2. Used gate decisions tocontrol % progression tonext stage.

Epoch-Shock

Gestation


Basic

R&DApplied

R&D

Project-

specific

Shelf ShelfShelf

Flight

Assessment based on Epoch-Shock model

1. Resources can’t be earmarked for “earlystage/basic.” In practice that funding stream issplit between basic concepts and others that aretreading water and branching out.

2. Actively controllable gates don’t exist. Winnowinghappens based on the co-timing of a technicalbreakthrough (unpredictable) and the nextrelevant mission call (semi-cyclical).

© 2014 SzajnLab

Rethinking the Management Problem

• Basic insight:

– As long as innovation occurs at multiple technical levelssimultaneously, and innovating teams can choose to drawresources from multiple institutional levels

– Current management strategies can’t work as intended!

• Epoch-Shock formulation provides a basis for rethinking themanagement problem:

– Some shocks can be harnessed as management levers: exploringpredictability and influenceability.

– The work environment can be designed, to encourage desirableinteractions and collaborations: exploring incentive systems andorganizational/architectural interactions

© 2014 SzajnLab

Implications for Cost & Schedule Analysts

•A key part of TRL analysis depends on the stage gate model of innovation

–Thinking in terms of the epoch shock model may help point analysts to more complex nuances that they need to study and evaluate.

•A key part of estimating an individual technology depends on the depends on the broader tech ecosystem

–Our cases showed that funding for these projects came from a variety of funding sources at multiple levels

•The process of technology development takes much longer than expected

© 2014 SzajnLab

Thanks for your attention. Comments welcome.

Professor Zoe Szajnfarber

E-mail: [email protected]

Web: www.seas.gwu.edu/~zszajnfa

Thanks to our sponsors:

MS: Ademir Vrolijk

PhD: Isabel Bignon

PhD: Alex Burg

PhD: Jason Crusan

PhD: Zachary Pirtle

PhD: Amy Cox

PhD: Sam Marquart

Documents

Beyond TRL: A Revised Model of Technology Development and ... · Beyond TRL: A Revised Model of Technology Development and Considerations for Programmatic Analysis Prof. Zoe Szajnfarber