Transdisciplinary Framework for Developing Training & Education in the U.S. Army

8/12/2019 Transdisciplinary Framework for Developing Training & Education in the U.S. Army

1/22

Riccio, G., Flanagan, S., Darwin, M., & Cornell d'Echert, B. (2010). Preparation for Full Spectrum Operations. In: Riccio, G., Diedrich, F., &

Cortes, M. (Eds.).An Initiative in Outcomes-Based Training and Education: Implications for an Integrated Approach to Values-Based

Requirements (Chapter 1). Fort Meade, MD: U.S. Army Asymmetric Warfare Group. [Cover art by Wordle.net represents word frequency in

ext.]


2/22

7 Riccio et al.

Asymmetric Warfare Group

Chapter 1. Preparation for Full Spectrum Operations

Gary Riccio, Scott Flanagan, Morgan Darwin, & Blaise Cornell dEchert

The Wexford Group International

The mission of the Asymmetric Warfare Group (AWG) is to provide operational advisoryassistance in support of Army and Joint Force Commanders to enhance the combat effectiveness

of the operating force and enable the defeat of asymmetric threats. The AWG executes missions,

responsibilities, and functions required to enhance the capabilities of U.S. units by making them

more adept at quickly identifying and attacking enemy vulnerabilities, and by preparing them for

a broader spectrum of threats (Cornell-dEchert, 2009a,b; Shaw & Cortes, 2009). The methods

used by the AWG to promulgate best practices in asymmetric warfare range from seminars to

field-based training. In March of 2008, the AWG initiated a scientific and technical effort to help

define and develop Outcomes-Based Training and Education (OBTE) (Darwin, 2008a,b). The

objectives of this research include:

Identify the unique and essential attributes of OBTE, as an innovation in training, withsufficient depth and clarity to enable it to be validated, verified, replicated, and

generalized. Develop measures for instructor and trainee behavior that reflect OBTE principles in the

context of basic and advanced rifle marksmanship in initial entry training.

Develop measures for the intangible effects of OBTE in basic training and beyond totheir potential impact on Full Spectrum Operations.

Demonstrate how evidence can be gathered for the effects of OBTE using measures thatcan be traced to scientific, operational, and institutional principles and practices.

Identify organizational factors affecting OBTE implementation (including life-cyclemanagement) in Army institutions of training and education.

Identify the extent to which initiatives analogous to OBTE can be developed to achievesimilar impact on other programs of training and education.

Definition and measurement of OBTE is timely because approaches related to OBTE are beingintroduced into a variety of programs of instruction in the Army (Cavallaro, 2008; Cox, 2008;

Currey, 2008; Ferguson, 2008; Foster, 2009; Markin, 2008; Perry & McEnery, 2009; Sellers,2008, Tice, 2008). Given this interest, the intent is to develop OBTE into an integrated set of

instructional capabilities that meet the needs of Full Spectrum Operations. Adoption of any such

capabilities in the institutional Army ultimately would require compatibility with the approach of

the United States Army Training and Doctrine Command (TRADOC) to instructional systems

design and associated processes such as analysis, design, development, implementation andevaluation (TRADOC, 1999).

At present, TRADOC Regulation 350-70 and several key pamphlets associated with this

regulation are being revised to facilitate innovation in training and education. This is important

because there is a need to maintain rigorous systems engineering while expediting adaptation tothe changing needs for instruction in the Army (Swain, 2005). For these reasons, the present work

was influenced by state-of-the-art approaches in the DoD and Industry to the use of systems

engineering in the integration and development of capabilities (Chairman of the Joint Chiefs of

Staff, 2004; Chrissis, Konrad, & Shrum, 2003; Sage & Rouse, 1999; Schaeffer, 2005). In

particular, our adherence to the practices of the Capability Maturity Model Integration for

Services (CMMI Product Team, 2009) increases the likelihood of upward compatibility with

future guidance about Army Training and Education Development (ATED) as well as viability of

the instructional capability over its life cycle.


3/22

Preparation for Full Spectrum Operations 8


This report of our research for the AWG is presented in a format consistent with the Service

System Development (SSD) process area of CMMI for Services. Accordingly, section 1.1 and 1.2

respectively summarize activities of the AWG to develop and analyze stakeholder requirements

(Specific Goal 1 of SSD) and to develop a service system that meets such requirements (Specific

Goal 2 of SSD). Chapters 2 and 3 describe in some detail the practical and scientific

considerations in developing and analyzing stakeholder requirements. Chapters 4 and 5 develop aunique scientific foundation for OBTE based on our collaborative inquiry with the progenitors

and stakeholders of OBTE. The remainder of the report addresses the scientific and technical

effort to verify and validate the service system (Specific Goal 3 of SSD). Section 1.3 describes

the preparation for verification and validation (Specific Practice 3.1 of SSD). Chapters 2, 7, and

12 also are relevant to Specific Practice 3.1. Chapter 6 describes performance of peer reviews

(Specific Practice 3.2 of SSD). Chapters 7, 8, 9, and 13 describe verification of selected service

system components (Specific Practice 3.3 of SSD). Chapter 10 describes the initial exploration of

methods for validation of the service system (Specific Practice 3.4 of SSD). Chapters 4 and 5 also

are relevant to Specific Practice 3.4.

The interleaving of the chapters in this monograph with respect to service system development

reflects the spiral development approach we utilized in this investigation (see Prologue). Thisapproach is a best practice in capabilities development and continuous process improvement

(Chairman of the Joint Chiefs of Staff, 2004; Chrissis, Konrad, & Shrum, 2003; Sage & Rouse,

1999; Schaeffer, 2005). Implications for continuous process improvement are summarized in

Chapters 11, 14 and the Epilogue. The implications are as significant for Army training andeducation, in general, as they are for OBTE in particular. The reason is that the opportunities for

improvements in the implementation of OBTE reveal precisely where there are shortfalls and

gaps in current training with respect to the Army doctrinal requirements and the associated needs

of Full Spectrum Operations.

1.1 Requirements of Full Spectrum Operations

Full Spectrum Operations (FSO) are characterized by dynamic adjustments in objectives, roles,responsibilities, tactics, techniques, and procedures (French, 2002; Kilcullen, 2006; Meigs, 2003;

Petreaus, 2006). Such insights about contemporary conflict emphasize that most if not allindividual and collective tasks critically depend on an understanding of context and the

relationships among elements of the operational context. Appreciation of contextual relationships

reveals higher-order invariants across combat and non-combat operations and, thus, facilitates

transition between them. This is important given that, in counterinsurgency operations, transitions

between combat and non-combat situations can occur over time scales that are short relative toones physiological reactions to such situations. In such environments, an outward orientation can

help direct attention away from physiological distractions of potential life-and-death situations

(cf., Kolditz, 2007; Lerner & Keltner, 2001; Lerner & Tiedens, 2006; Lukey & Tepe, 2008). In

principle, it can help Soldiers remain grounded and persevere in the trans-extremis environments

of FSO. An outward orientation is fostered by skills of perception and action that involve habitualorientation to ones capabilities for interaction with the surroundings (cf., J. Gibson, 1977, 1979)

as well as by the associated understanding of ones efficacy (cf., Bandura, 1977, 1997).

There is a consistent thread through Army doctrinal publications that calls for individual cunning

and collective agility in unpredictable and changing environments (Riccio, Sullivan, Klein, Salter,

& Kinnison, 2004; Shaw & Cortes, 2009). The Institutional Army is expected to produce Soldiers

who have the personal attributes and values necessitated by such agility.


4/22

9 Riccio et al.


Soldiers and units at every level must be flexible and adaptive. Often, stability

operations require leaders with the mental and physical agility to shift from

noncombat to combat operations and back again. (Headquarters Department of the

Army [HQDA], 2001, p. 9-5).

Simultaneously, individual Soldiers, NCOs, warrant officers, officers, and the

civilian work force are responsible for training themselves through personal self-development. Training is a continuous, lifelong endeavor that produces competent,

confident, disciplined, and adaptive Soldiers and leaders with the warrior ethos in our

Army. Commanders have the ultimate responsibility to train Soldiers and develop

leaders who can adjust to change with confidence and exploit new situations,

technology, and developments to their advantage. (HQDA, 2002, p. 1-5).

Commanders ensure that their subordinates know how to think instead of what to

think. They develop their subordinates confidence and empower them to make

independent, situational-based decisions. The goal is to develop subordinates who

have an agile and adaptive mindset. (HQDA, 2003, p. 2-12).

In 2003-2004, it was clear that senior Army leaders recognized training gaps and shortfalls withrespect to the Global War on Terror (dEchert, 2009; Shaw & Cortes, 2009). The AWG training

advisory teams debriefed units returning from Operation Iraqi Freedom (OIF) and Operation

Enduring Freedom (OEF). Without exception, leaders at all levels emphasized that the key to

mission success was proficiency in basic warfighting skills such as moving, shooting,communicating, and medical skills. Proficiency at these fundamentals allowed units to respond

quickly to new conditions and non-standard missions. Senior leaders were especially critical of

certain aspects of pre-deployment training (Flanagan, 2005; French, 2002).

Based on constant interaction with active duty units in theater, the AWG also recognized a needfor training that incorporates situations in which Soldiers and leaders are encouraged to think, act,

and make decisions as a training event unfolds. The impact of situated training is critically

interdependent with a Soldiers mastery of basic skills (Flanagan, 2005; Darwin, 2008a,b). A keyinsight, however, is that the definition of basic Soldier skills implicit in the AWGs use of this

term is broader than its common usage. In the present context, basic Soldier skills should not beconfused with the lowest level of detail, or even perhaps the most basic level of detail, in a typical

task analysis of individual and collective tasks. A different perspective is suggested by the

following observation:

Their honed skills will be validated and sometimes raised to a higher level. Theirthinking skills are challenged; this will prompt review and contemplation from

individual Soldiers and leaders, a comprehensive assessment of teamwork learned,

and new ideas, innovations, and techniques discovered. (Flanagan, 2005, p. 1)

From this perspective, a situation provides a context for linkage of tasks and understanding theconcurrent and downstream consequences of individual actionsthe levels situation awareness

required for adaptability (J. Gibson, 1977; E. Gibson, 1988; cf., Endsley, 1995). Without a basic

level of skill, however, it is difficult to appreciate the opportunities a situation affords and the

implications of ones action in that situation. And, without such an appreciation, it is difficult to

develop a deeper understanding of a skill and reach a higher level of skill. Put simply,

understanding ones own skill includes knowledge about what one can do with it; and ones

understanding of a situation is at least partially dependent on what one can do in that situation (J.

Gibson, 1977; 1979).


5/22



The implication is that training should address basic skills of perception and action that are

critical to proficiency in moving, shooting, and communicating. This requires a finer grain of

analysis, a level of analysis that also cuts across the distinctions among these tasks. This insight is

a source of innovation in Army training. It is not radical insofar as it is consistent with relatively

neglected implications of a theoretical framework that is familiar in the Army (Gagne, 1962,

1985)training and education for military tasks can benefit from consideration of basic scientificunderstanding of purposeful perception and action (Gagne & J. Gibson, 1947) as well as

motivation and emotion. In chapters 3-5, we extend this foundation in meaningful perception and

situated action to philosophically consistent views of social and cultural factors affecting self-

development and, in particular, the development of values.

The AWG analysis of stakeholder requirements for Army training can be summarized in three

conjectures:

Conjecture 1: Instruction can be designed, developed and implemented with respect to basicskills that lead to better understanding of a situation including relationships among elementswithin a relevant situation, that motivate and guide engagement with that situation, and that

afford opportunities to influence the situation instead of merely responding to it. There is aneed to identify this level of analysis for instruction and learning that fosters the development

of basic Soldier skills necessary for success in Full Spectrum Operations.

Conjecture 2: Instruction can be designed and implemented in verifiable ways that apply toall levels of leadership and leader development for a curriculum or program of instruction. A

general approach to Army training and education can be developed that applies to all

interactions between an instructor and a learner, as well as to self-development, without

sacrificing validity with respect to Full Spectrum Operations. There is a need for a verifiable

and valid approach that can be utilized by instructors and learners together or by learners ontheir own across all types of learning events.

Conjecture 3: Soldiers do not rise to the occasion in combat; they fall back to their highestlevel of training and education. The highest level of training and education should relate

directly and unambiguously to the most important requirements in Army Doctrine, such asArmy Values and Warrior Ethos. There currently are no doctrinally codified standards for

such values-based requirements. There is a need for values-based standards that can be

applied to any and all learning events.

1.2 Outcomes-Based Training and Education (OBTE)

In response to feedback from active duty commanders and in anticipation of ongoing challenges,

the AWG developed a set of briefings, seminars, workshops, and field-based training courses to

enhance combat skills and to better prepare Soldiers for Full Spectrum Operations (dEchert,

2009; Flanagan, 2005; Shaw & Cortes, 2009). A combat skills course was conceived as a modelfor noncommissioned officers (NCO) to set up and conduct training (Flanagan, 2005). The

approach to this course was heavily influenced by decades of lessons learned by special units in

the military for which training is a persistent priority and core value. The intent of such an

approach is "through mentorship and coaching, demonstrate a method of training which is safe,

effective, combat relevant, and encourages a continuous thought process which will demand

accountability" (McNamara, 2008, p. 2). This led to the Combat Applications Training Course

(CATC), a course that demonstrates how to address context (Conjecture 1) and learning to learn

(Conjecture 2) with respect to marksmanship but in ways that are general in principle. The


6/22

11 Riccio et al.


specific context of marksmanship allowed the AWG to follow through with instructors and their

chain of command to examine viability of the approach in existing programs of instruction in the

Generating Force (Conjecture 3).

Packaged with broader forums for education of instructors and leaders, and generalized to skills

other than marksmanship, the approach currently is implemented as a set of instructional

capabilities referred to as OBTE (Darwin, 2008b; Cornell-dEchert, 2009a,b). OBTEsystematically seeks to engage an organization to shape instruction and influence the overall

development of an individual, as a Soldier, with respect to broad outcomes that transcend course-

specific objectives. OBTE strives to develop Soldiers who can think and behave adaptively amid

uncertainty, Soldiers who can learn in any situation, and who continuously improve as individuals

and as members of a unit (see e.g., Chapters 2, 3, and 15).

1.2.1 Exemplar of OBTE: Combat Applications Training Course

As a means of demonstrating OBTE and providing context for understanding the principles, the

AWG adapted a field-based course in rifle marksmanship, referred to as the Combat Applications

Training Course (CATC) (Darwin, 2008a). It was originally developed as a course inmarksmanship as provided, for example, to the 82nd

Airborne Division (Aug-Dec 2006) and the

101stAirborne Division (Feb-Jun 2007). In CATC, over a period of one week, instruction

progresses from introductory levels (e.g., weapons system function, components and

maintenance, shooting from a prone position) to more advanced and complex situations that

combine moving, shooting, and communicating (e.g., highly mobile engagement of multiple

targets from multiple positions). The instructional content covers elements of both Basic Rifle

Marksmanship (BRM) and Advanced Rifle Marksmanship (ARM). It includes situated exercises

that are complex and challenging due to the introduction of teamwork and activities focused on

problem solving. While the course incorporates some lecture and discussion sessions, the

majority of the course is experiential in nature (cf., Reed, 1996; Riccio, 1993).

CATC has evolved into a course with the primary intent of educating leaders and instructors

about OBTE (Darwin, 2008b). The earliest examples of this train the trainer course were at FortBenning and Fort Jackson in 2007-2008. The course attempts to promulgate a developmental

approach to instruction primarily by leveraging the credibility and influence of highly

accomplished instructors as role models for instructional strategies that exemplify OBTE

(Bandura, 1977; Riccio et al., 2004). General features of this developmental approach include

transition from structured command-oriented tasks to less formal self-guided activities, as well asa gradual increase in task complexity and challenges (Darwin, 2008a,b; cf., Tobias & Duffy,

2009).

An illustrative example of OBTE in marksmanship training is a gradual change in range control

during the course of training, providing less directive communication, and applying less

conventional controls to demonstrate how marksmanship training can be achieved safely while

encouraging individual accountability. Rather than the instructor assuming responsibility for

safety alone, the students are expected to assume that responsibility along with instructors. This is

important because, from the perspective of OBTE, Soldiers ultimately will be accountable in the

field. Such accountability should be promoted as early as possible in the development of Soldiers.

In the application of OBTE to marksmanship, students have accountability for behavior such as

operationally appropriate weapons orientation (down and ready) and clearing (self).

In addition to this focus on individual accountability, any implementation of OBTE should

emphasize understanding the why and developing problem-solving skills. For instance, during

breaks from experiential exercises, After Action Reviews (AAR) are used by instructors to


7/22



introduce and elaborate on background fundamentals and changes to the training environment. In

marksmanship training, for example, focus is placed on understanding ballistics and why

adaptations are needed (e.g., what happens when firing up or down a hill). Many events are

applied as problems for students to solve (Darwin, 2008a,b; Savory & Duffy, 1995; Vandergriff,

2006). An example is the use of situations in which students effectively manage resources such as

time and ammunition. Critically, in such situations, students are not told solutions that they then

practice. Rather, the students construct solutions under the guidance of an instructor.

1.2.2 OBTE as a Multifaceted Instructional System

An essential attribute to OBTE, as an innovation in instruction, is the critical role for leadership

throughout the organization responsible for instruction. The view of the AWG about this role is

described well in the following passage (quoted in Shaw & Cortes, 2009):

If the transformed Army will require leaders who can operate independently in the

absence of close supervision, the current leader development experience of company

command will have to change. Consequently, the author asks for senior leaders not todo more, but to do less and thus give subordinates more freedom to innovate. (Wong,

2002, p. v)

There is much implied in this recommendation. One important and difficult implication is that

leaders at all levels should create a climate of trust in their subordinates. This should not be

confused with blind trust. The implied trust is in providing subordinates with opportunities forinitiative in which they are accountable for their decisions. At the same time, a deeper level of

trust is engendered by leaders who demonstrate a willingness to share accountability for any

unintended consequences of the initiative they encourage in their subordinates. Such sharing of

accountability becomes productive and collaborative if there is a serious commitment to AAR.

Such collaborative reflection identifies steps that can be taken to improve training by minimizingundesirable consequences of a new approach and to sustain the good training that results.

Another important implication of leadership in instructional programs is the demonstration ofcommon or convergent objectives amid division of labor throughout the command chain. Leaders

can do this, for example, in walkthroughs conducted in the context of peer review (Chrissis et al.,2003; CMMI Product Team, 2009). Senior leaders who do this provide a model for their

subordinate leaders, both senior and subordinate leaders provide a model for instructors, and

instructors provide a model for their students. It reveals that there is a common stake in learning

and, in the spirit of peer review, an interest in individual adaptation to maximize opportunities for

learning, whatever the situation. Such engagement by leaders is essential from the perspective ofOBTE (Haskins, 2009; Schwitters, 2009; Shaw & Cortes, 2009; Vandergriff, 2007; Appendix C).

The AWGs intent is to develop OBTE into a mature service system that can be transitioned to

the institutional Army. It recognizes that leadership is an essential attribute of the approach. Thus,

a series of workshops, seminars, and short courses were developed for leaders throughout thecommand chain for a program of instruction (Darwin, 2008a,b,). These engagements with leaders

are viewed, together with field-based courses for instructors, as an integrated set of instructional

capabilities for an organization (Cornell-dEchert, 2009a,b). Validation of OBTE in a particular

setting will reflect the influence of all such components of the instructional service system

irrespective of the extent to which they reflect OBTE. Thus, leadership influenced by OBTE

should have a constructively additive or multiplicative effect on instruction influenced by OBTE.

If only one of these components of an instructional system is influenced by OBTE, their influence

will tend to cancel or interfere with each other.


8/22

13 Riccio et al.


1.3 An Appraisal of Instruction with Respect to OBTE

1.3.1 A Systems Engineering Framework for Integration and Development of OBTE

To date, OBTE has been explored in a variety of programs of instruction in the Army (Cox, 2008;

Cavallaro, 2008; Currey, 2008; Ferguson, 2008; Markin, 2008; Sellers, 2008, Tice, 2008; see alsoVandergriff, 2006, 2007). Growing interest in OBTE recently was exemplified by the attendance

at a workshop organized by the AWG and the Johns Hopkins University. Workshop participants

represented twenty-six locations across HQDA, FORSCOM, Joint, Inter-agency, and allied

organizations (Devens, 2009). Given this widespread interest, it is important to understand the

issues pertaining to implementation of OBTE and how it can be improved as an integrated set ofinstructional capabilities (Cornell-dEchert, 2009a,b). This will enable the AWG and others to

help stakeholders translate their interest into action, whether for initial implementation or process

improvement.

A rigorous framework within which to describe and guide the steps toward transition of OBTE is

provided by CMMI (Chrissis, Konrad, & Shrum, 2003; Schaeffer, 2005). Use of CMMI should

not be interpreted as the imposition of onerous processes onto a service system (e.g., OBTE) thatis intended to foster innovation. To the contrary, CMMI enables initiative by ensuring that there

is accountability within a rigorous yet flexible framework for understanding the evolving and

interrelated activities of an organization (Ahern, Clouse, & Turner, 2008; Garcia & Turner,

2006). Rigorous description of intent and execution provides desirable constraints on innovation,

while flexibility of the framework allows an organization to make sense of the results of

innovation (Weick, 1995).

CMMI is used to describe the current research and its implications for continuing integration and

development of OBTE as an instructional service system (CMMI Product Team, 2009). From the

perspective of CMMI, the AWG has been performing and managing OBTE at a level that is

commensurate with capability level 2.

A capability level 2 process is characterized as a managed process. A managed

process is a performed (capability level 1) process that has the basic infrastructure in

place to support the process. It is planned and executed in accordance with policy;

employs skilled people who have adequate resources to produce controlled outputs;

involves relevant stakeholders; is monitored, controlled, and reviewed; and is

evaluated for adherence to its process description (CMMI Product Team, 2009, p. 23-

24).

The intent of the current research is to enable the implementation of OBTE at a level

commensurate with capability level 3; that is, to enable its institutionalization (CMMI generic

goal, GG 3). The generic practices at this level are to establish a defined process (GP 3.1) and to

collect improvement information (GP 3.2). A premise of the investigation is that the behavioraland social sciences are indispensable to definition and measurement for a service system at

capability level 3.

A capability level 3 process is characterized as a defined process. A defined process

is a managed (capability level 2) process that is tailored from the organizations set ofstandard processes according to the organizations tailoring guidelines and

contributes work products, measures, and other process improvement information to

the organizational process assets A defined process clearly states the purpose,


9/22



inputs, entry criteria, activities, roles, measures, verification steps, outputs, and exit

criteria (CMMI Product Team, 2009, p. 24).

The appropriate guidance for OBTE at capability level 3 is the Service System Development

process area within CMMI for Services (CMMI Product Team, 2009, pp. 437-462). The specific

goals for this process area are: develop and analyze stakeholder requirements (SG 1), develop

service system (SG 2), and verify and validate service system (SG 3). The activities associatedwith SG 2, reviewed above, are treated in more detail elsewhere (Darwin, 2008a,b; Cornell-

dEchert, 2009a,b). The activities associated with SG 1 and SG 3 relate directly to the current

research and, in the context of spiral development, have explicit implications for SG 2.

The specific practices associated with SG1 of the Service System Development process area are:

develop stakeholder requirements (SP 1.1), develop service system requirements (SP 1.2),

analyze and validate requirements (SP 1.3). The specific practices associated with SG 3 are:

preparation for verification and validation (SP 3.1), performance of peer reviews (SP 3.2),

verification of selected service system components (SP 3.3), and validation of the service system

(SP 3.4).

1.3.2 Preparation for Validation and Verification

The purpose of the investigation was to verify and validate OBTE to confirm that it satisfies its

intended requirements and that it will satisfy end-user expectations during actual service delivery.

More specifically, our objective is to understand the implementation of OBTE and its effects onthought and behavior, and where possible, to get some clues about its impact on instruction and

learning. To accomplish this objective, we utilized multiple sources of evidence from a variety of

techniques. While no single data collection method alone will yield comprehensive results, a

combination of approaches typically yields deeper understanding (e.g., CMMI Product Team,

2009; Jackson, Woods, Durkee, OMalley, Diedrich, Aten, et al., 2008; Kirkpatrick, 1994;Tashakkori & Teddlie, 1998). In particular, our inquiry involved askingvia self-report survey and

watchingvia observation (Cresswell, 1998; Ellingson, 2009; Miles & Huberman, 1994). The

sources of evidence addressed in Section II of this report are listed in Table 2. Section IIIprovides additional evidence that builds on this foundation for further development of OBTE.

We began by analyzing results from web-based surveys that were solicited by the AWG from

Soldiers who had participated in CATC. The open-ended responses to these items provided

evidence regarding impressions of participants about the usefulness and potential application of

the course. This form of peer review is consistent with Kirkpatricks Level 1 of program

evaluation in which reactions of participants are collected (Kirkpatrick, 1994). Such reactions areindicative of the motivation a student might experience in a course, and they can be suggestive

about whether the course achieves its intent, but they do not necessarily indicate why. For this

reason, consistent with peer review in CMMI, attention was focused on the potential use of such

surveys to identify defects or to make recommendations about changes to improve OBTE (CMMI

Product Team, 2009, p. 457).

An essential attribute of CATC as a train-the-trainer course, and of OBTE in general, is its

emphasis on instructor-student interactions. The behavior of instructors as role models, mentors,

facilitators and advisors is assumed to be critical to the development of individuals. For this

reason, an established methodology was employed to identify such behavior, to define its

relationship to OBTE principles and related scientific research, and to develop measures for

verification of OBTE in practice (MacMillan, Entin, Morley, & Bennett, in press; Chapter 2). The

objective of the verification was to provide data for a formative assessment of OBTE and


10/22

15 Riccio et al.


feedback to instructors about the efficacy of their instructional behavior (Bransford et al., 2000;

CMMI Product Team, 2009, p. 460).

Table 2. Questions addressed by the various sources of evidence addressed in Section II.

Type of evidence and source Questions addressed

Peer Review

Self report data from Soldiers who participatedin CATC

Source: Web-based survey conducted afterCATC participation

Was CATC perceived as valuable overall?

Was CATC perceived as valuable inincreasing broader skills targeted by OBTE?

What suggestions did Soldiers have forimproving CATC?

Kirkpatrick Levels 1

Verification (Primary)

Observed behavior for DS who had takenCATC, acting as instructors in marksmanship

course

Source:Observational rating form duringmarksmanship course

To what extent did DS exhibit the OBTEbehavior that was trained in the CATC?

Kirkpatrick Levels 2 (DS)

Verification (Secondary)

Observed behavior for Privates inmarksmanship course taught by DSs who hadtaken CATC

Source: Observational rating form duringmarksmanship course

To what extent did Privates exhibit highmarksmanship performance?

To what extent did Privates exhibit OBTEbehavior?

Kirkpatrick Level 2 (Privates)

Verification (Secondary)

Communication between DS and Privates

Source:Communications rating form duringmarksmanship course

Were communication patterns consistent

with OBTE principles?Kirkpatrick Level 2 (DS)

Validation (Secondary)

Correlation between OBTE behavior of DS andperformance of Privates

SourceObservational rating forms for DS andPrivates during marksmanship course

Was a higher level of OBTE behavior in DSassociated with higher marksmanshipperformance by Privates?

Kirkpatrick Level 3 (DS)

Validation (Primary)

Survey of deployed Soldiers who had taken the

initial iterations of CATC

Source:Survey following return fromdeployment

Did deployed Soldiers find their OBTEtraining to be valuable upon deployment?

Kirkpatrick Level 3 (toward Level 4)

The same methodology was employed to develop measures for student behavior that would be

sensitive to the effects of OBTE. These measures are consistent with Kirkpatricks Level 2 of

program evaluation that attempts to assess learning. This level is complicated in general because

it often addresses nested learning objectives (Kirkpatrick, 1994). In the present context, measures


11/22



are required that relate both to course-specific learning objectives as well as the longer-term

developmental outcomes that are the reason for OBTE (Cornell-dEchert, 2009b; Darwin, 2009).

Measures to assess learning of both instructors and students are equally important for verification

of OBTE (Bransford et al., 2000; Cornell-dEchert, 2009a,b).

Verification is central to the development of OBTE into a defined process. Thus, the focus and

timing of the investigation was dictated by opportunities to develop and apply OBTE measures ina program of instruction in which one could expect substantive variation in behavior of

instructors and students. An opportunity emerged from a systematic engagement of the AWG at

Fort Benning and Fort Jackson. The AWG started providing OBTE services at these posts in

August 2007 and continued through the end of this investigation. The engagement at Fort Jackson

included the Center of Excellence for Basic Combat Training, Drill Sergeant School cadre,

Victory University, and Basic Combat Training drill sergeants. The engagement at Fort Benning

included instructors in noncommissioned officer education system, instructors in the officer

education system, drill sergeants in 198thInfantry Training Brigade (one station unit training),

and drill sergeants in the 192nd

Basic Combat Training Brigade.

The engagements at Fort Benning and Fort Jackson included briefings, seminars, and workshops

as well as CATC. There was due diligence to ensure that the command chain for instructorsunderstood the intent and approach of OBTE. This generally included first-hand experience as a

participant in CATC or in an abbreviated (three-day) version of this field course referred to as the

Senior Leader Course. The buy-in of senior leaders was considered critical to establishing a

command climate favorable to OBTE. At Fort Jackson, the Commander of the Basic TrainingCenter of Excellence and post Command Sergeant Major championed OBTE (Schwitters, 2009).

The Training Brigade Commanders at Fort Benning also made OBTE a priority both in its

implementation and in exercising initiative and leadership its continuing development (Haskins,

2009). Empowering such leaders with information about best practices in OBTE is considered to

be a critical component of OBTE.

By April 2008 the command climate at Fort Jackson and Fort Benning was sufficiently favorable,

and sufficient numbers of Drill Sergeant (DS) had participated in CATC, to conduct an appraisalof OBTE in the context of Initial Entry Training. Data were collected on DS and students

(Privates) at these posts to verify that instructor behavior consistent with OBTE could beidentified and measured in situ. The relationships between instructor behavior and student

behavior are relevant to Kirkpatricks Level 3, with respect to DS not Privates, insofar as these

data are indicative of DS competency in their job as instructors or as influencers of Privates

(Kirkpatrick, 1994). As such, these data also relate to validation in CMMI-SVC, however, only

secondarily so because the focus of OBTE is on longer-term developmental outcomes aboutwhich one can only get suggestive evidence at the time of instruction.

Our primary method of addressing validation of OBTE focused on outcomes more likely to

reflect long-term developmental impact. In the context of Kirkpatricks level 3 and level 4 of

program evaluation, the most relevant feedback is about performance of Soldiers, as such, intheater. Level 4 emphasizes the contributions of training to the organizational mission and

objectives (Kirkpatrick, 1994). Toward that end, we developed a post-deployment survey. The

survey was administered to members of the 82nd

Airborne Division who, prior to deployment, had

taken an early version of CATC (i.e., before the AWG attempted to implement it more explicitly

as a train-the-trainer course). This particular survey provided further self-report evidence

regarding the influence of the course on preparedness of individuals for combat operations and as

trainers in their own units during deployment. These results therefore highlight the potential of


12/22

17 Riccio et al.


OBTE as an approach, as well as the challenges of the approach, for achieving long-term

developmental outcomes in the context of training for proficiency in a particular skill.

Collectively, these various sources of evidence provide a unique look into the potential of OBTE

with respect to assessment of instructor behavior, influence on student behavior, and the

downstream consequences in theater. This multi-faceted appraisal was possible because of the

rigorous definition and methods for measurement of OBTE (e.g., Chapter 2) coupled with clearstatements of intent and purpose (Cornell-dEchert, 2009a, b; Darwin, 2008a, b).

1.4 References

Ahern, D., Clouse, A. & Turner, R. (2008). CMMI distilled: A practical introduction tointegrated process improvement. Upper Salle River, NJ: Addison-Wesley.

Bandura, A. (1977). Social learning theory. New York: General Learning Press.

Bandura, A (1997). Self-efficacy: The exercise of control. New York, NY: Freeman and

Company.

Bransford, J. D., Brown, A. L., & Cocking, R. R. (2000).How people learn: Brain, mind,

experience, & school.Washington, DC: National Academy Press.

Cavallaro, G. (2008). The art of making better drill sergeants.Army Times, Jun 23.Chairman of the Joint Chiefs of Staff (2004).Joint capabilities integration and development

system.Retrieved November, 2004, from Defense Technical Information Center,

http://www.dtic.mil/cjcs_directives/cdata/unlimit/3170_01.pdf.

Chrissis, M.B., Konrad, M. & Shrum, S. (2003). CMMI: Guidelines for process integration and

product improvement.Boston, MA: Pearson Education.

CMMI Product Team (2009). CMMI for services, version 1.2. (CMU/SEI-TR-2009-001; ESC-

TR-2009-001). Pittsburgh, PA: Carnegie Mellon University.

Cox, M. (2008). Zeroed in on marksmanship: Army overhauls weapons training and qualification.

Army Times. May 5, 2008,14-16.

Cornell-dEchert, B. (2009a).An introduction to outcomes-based training and education. Fort

Meade, MD: Asymmetric Warfare Group.

Cornell-dEchert, B. (2009b). Outcomes-

based training and education: Implementation guide.

Fort Meade, MD: Asymmetric Warfare Group.

Cresswell, J.W. (1998). Qualitative inquiry and research design: Choosing among five traditions.

Thousand Oaks, CA: Sage.

Currey, C.J. (2008). Outcomes-based training: Whats next?Initial Entry Training Journal, 1, 1-

3.

Darwin, M. (2008a).Asymmetric Warfare Group combat applications training course (CATC)

senior leader discussion(Briefing, January 23, 2008). Fort Meade, MD: Asymmetric Warfare

Group.

Darwin, M. (2008b). Outcomes-based training and education: fostering adaptability in full

spectrum operations(Briefing, December 2008). Fort Meade, MD: Asymmetric Warfare

Group.

Darwin, M. (2009, March).Developing outcomes. Presentation at the US Army Asymmetric

Warfare Group workshop on Outcomes-based Training and Education, Applied Physics

Laboratory, Johns Hopkins University, Laurel, MD.

Devens, M. (2009, March).Introductory remarks. Presentation at the US Army Asymmetric

Warfare Group workshop on Outcomes-based Training and Education, Applied Physics

Laboratory, Johns Hopkins University, Laurel, MD.

Ellingson, L. (2009).Engaging crystallization in qualitative research: An introduction. Los

Angeles, CA: Sage.


13/22



Endsley, M.R. (1995). Toward a theory of situation awareness in dynamic systems.Human

Factors, 37, 32-64.

Ferguson, M.E. (2008). Outcomes based training and education: Targeting the intangibles. The

NCO Journal, October, 2008, 16-21.

Flanagan, S. (2005). Combat skills. Unpublished manuscript (Nov. 16, 2005).

Foster, C. (2009). No approved solutions in asymmetric warfare: Nurturing adaptive leaders in

an outcomes-based training environment.Assembly, July/August, 28-29.French, D., (2002). Wake up and smell there is something wrong.News From the Front, Sept-

Oct 2002.26-29. Ft. Leavenworth, KS: Center for Army Lessons Learned.

Gagne, R. (1962). Military training and principles of learning.American Psychologist, 17, 263-

276.

Gagne, R. (1985). The Conditions of Learning (4th ed.). New York: Holt, Rinehart & Winston.

Gagne, R., & Gibson, J. (1947). Research on the recognition of aircraft. In: J. Gibson (Ed.)

Motion picture training and research. Washington, DC: U.S. Government Printing Office.

Garcia, S, & Turner, R. (2006). CMMI(R) survival guide: Just enough process improvement.

Upper Salle River, NJ: Addison-Wesley.

Gibson, E.J. (1988). Exploratory behavior in the development of perceiving, acting, and theacquiring of knowledge.Annual Review of Psychology, 39, Pages 1-42.

Gibson, J.J. (1977). The theory of affordances. In R. Shaw & J. Bransford (eds.),Perceiving,acting and knowing: toward an ecological psychology. Hillsdale, NJ: Erlbaum.

Gibson, J.J. (1979). The ecological approach to visual perception. Boston: Houghton Mifflin.

Haskins, C. (2009, March).Development of outcomes based training. Presentation at the US

Army Asymmetric Warfare Group workshop on Outcomes-based Training and Education,Applied Physics Laboratory, Johns Hopkins University, Laurel MD.

Headquarters Department of the Army (2008). Operations. Field Manual No. 3-0. Washington,

DC: Headquarters Department of the Army.

Headquarters Department of the Army (2002). Training the Force. (Revised 2008, Training for

Full Spectrum Operations). Field Manual No. 7-0. Washington, DC: HeadquartersDepartment of the Army.

Headquarters Department of the Army (2003).Battle focused training. Field Manual No. 7-1.

Washington, DC: Headquarters Department of the Army.Jackson, C., Woods, H., Durkee, K., OMalley, T., Diedrich, F., Aten, T., Lawrence, D., & Ayers,

J. (2008). Tools for assessment of operator contribution to system performance.Proceedingsof the Undersea Human Systems Integration Symposium, Bremerton, WA.

Kilcullen, D. (2006). Twenty-eight articles: Fundamentals of company level counterinsurgency.

Retrieved August, 2006, from the Joint Information Operations Center,

http://www.au.af.mil/info-ops/iosphere/iosphere_summer06_kilcullen.pdf.

Kirkpatrick, D., (1994).Evaluating training programs. San Francisco, CA: Berrett-KoehlerPublishers, Inc.

Lerner, J., & Keltner, D., (2001). Fear, anger, and risk.Journal of Personality and Social

Psychology, 81(1), 146-159.

Lerner, J., & Tiedens, L. (2006). Portrait of the angry decision maker: How appraisal tendencies

shape angers influence on cognition.Journal of Behavioral Decision Making (Special Issueon Emotion and Decision Making),19, 115-137.

Lukey, B. & Tepe, V. (Eds.) (2008).Biobehavioral resilience to stress. Boca Raton, FL: CRC

Press.

MacMillan, J., Entin, E. B., Morley, R., & Bennett, W. (in press). Measuring team performance in

complex dynamic environments: The SPOTLITE method.Military Psychology.

Marken, W. (2008). Merging doctrine: Outcomes-based training and mission essential task lists

compared.Initial Entry Training Journal, 2, 1-3.


14/22

19 Riccio et al.


McNamara, P. (2008). T.A.P.S. Tactical application of practical shooting: Recognize the void in

your tactical training. New York: iUniverse.

Meigs, M. C. (2003). Unorthodox thoughts about asymmetric warfare.Parameters, 33(2),4-18.

Mutafelija, B., & Stromberg, H. (2003). Systematic process improvement using ISO 9001:2000

and CMMI. Boston, MA: Artech House.

Perry, R. & McEnery, K. (2009). Army reconnaissance course: Defining the aim point for

reconnaissance leader training.Armor, July-August, 14-20.Petraeus, D.H. (2006). Learning counterinsurgency: Observations from Soldiering in Iraq.

Military Review, Jan-Feb, 2-12.

Reed, E. (1996). The necessity of experience.New Haven: Yale University.

Riccio, G. (1993). Information in movement variability about the qualitative dynamics of posture

and orientation. In: K. Newell (Ed.). Variability and Motor Control(pp. 317-357),

Champaign, IL: Human Kinetics.

Riccio, G., Sullivan, R., Klein, G., Salter, M., & Kinnison, H. (2004). Warrior ethos: Analysis of

the concept and initial development of applications. ARI Research Report 1827. Arlington,

VA: US Army Research Institute for the Behavioral and Social Sciences.

Sage, A.P., & Rouse, W.B. (Eds.) (1999).Handbook of systems engineering and management.New York, NY: Wiley.

Savory, J.R. & Duffy, T.M. (1995). Problem based learning: An instructional model and itsconstructivist frameworkEducational Technology, 35, 31-38.

Schaeffer, M. (2005).DoD and CMMI. 5th Annual CMMI Technology Conference and User

Group. Denver, CO 14-17 November 2005.

http://www.dtic.mil/ndia/2005cmmi/tuesday/schaeffer.pdfSchwitters, J. (2009, March). Command imperative for change.Presentation at the US Army

Asymmetric Warfare Group workshop on Outcomes-based Training and Education, Applied

Physics Laboratory, Johns Hopkins University, Laurel, MD.

Sellers, T.J. (2008). Basic officer leader course: The so what in junior officer education today.

Infantry, September-October,2008, 6-8.Shaw, R. & Cortes, M. (2009, March).Necessity for change to defeat asymmetric threats and

prepare for full-spectrum operations.Presentation at the US Army Asymmetric Warfare

Group workshop on Outcomes-based Training and Education, Applied Physics Laboratory,Johns Hopkins University, Laurel, MD.

Swain, R. (2005). Changes in instructional system design (ISD): Improving training productdelivery to United States Army Soldiers. USAWC Strategic Research Project. Carlisle

Barracks, PA: U.S. Army War College.

Tashakkori, A., & Teddlie, C. (1998).Mixed methodology: Combining qualitative and

quantitative approaches. Thousand Oaks, CA: Sage.

Tice, J. (2008). Soldier training is in for a big overhaul.Army Times, April 8, 2008.Tobias, S. & Duffy, T. (eds.) (2009). Constructivist instruction: Success or failure.New York:

Routledge:

United States Army Training and Doctrine Command (1999). Systems approach to training

management, processes, and products.TRADOC Regulation 350-70 (under revision). Fort

Monroe, VA: Training and Doctrine Command.Vandergriff, D. (2006). Adaptive leaders course, Part 1: Old dogs teaching new tricks.Army,

November 2006,59-66.

Vandergriff, D. (2007). From swift to Swiss: Tactical decision games and their place in military

education and performance improvement.Performance Improvement, 45(2), 30-39.

Weick, K.E. (1995). Sensemaking in organizations. London: Sage.

Wong, L. (2002). Stifled innovation: Developing tomorrows leaders today. Carlisle, PA: U.S.

Army War College.


15/22

ix Riccio, Diedrich, & Cortes (Eds.)


TABLE OF CONTENTS

.

page

Prologue: A Programmatic View of the Inquiry into Outcomes-Based Training & Education....... 1Historicity of our Research on OBTE..........................................................................................1The Approach and Lessons Learned from the Research..............................................................3Documentation of the Research ...................................................................................................4

Section I. Development of Stakeholder Requirements for OBTE..............................................6 Chapter 1. Preparation for Full Spectrum Operations ......................................................................7

1.1 Requirements of Full Spectrum Operations...........................................................................81.2 Outcomes-Based Training and Education (OBTE)..............................................................10

1.2.1 Exemplar of OBTE: Combat Applications Training Course........................................111.2.2 OBTE as a Multifaceted Instructional System .............................................................12

1.3 An Appraisal of Instruction with Respect to OBTE ............................................................131.3.1 A Systems Engineering Framework for Integration and Development of OBTE........131.3.2 Preparation for Validation and Verification .................................................................14

1.4 References ............................................................................................................................17Chapter 2. Formative Measures for Instructors..............................................................................20

2.1 Development of Formative Measures ..................................................................................202.1.1 The COMPASS Methodology......................................................................................202.1.2 Development of Measures for OBTE ...........................................................................21

2.2 Description of Formative Measures .....................................................................................212.2.1 Results of the COMPASS Process................................................................................212.2.2 Elaboration on the Description of Measures.................................................................23

2.3 OBTE Performance Measures: Planning for Training.........................................................232.3.1 Define Outcomes ..........................................................................................................232.3.2 Create a Positive Learning Environment......................................................................252.3.3 Create the Parameters of Learning................................................................................27

2.4 OBTE Performance Indicators: Training Execution............................................................282.4.1 Communicate the Parameters of Learning....................................................................282.4.2 Training Emphasizes Broad Combat or Mission Success ............................................292.4.3 Customize Instruction When Possible Based on Constraints/Conditions ....................312.4.4 Facilitates Learning of Concepts ..................................................................................322.4.5 Creates a positive learning environment.......................................................................342.4.6 Instructors Utilize Measures of Effectiveness & Self-Evaluation................................362.4.7 Uses scenarios to facilitate learning..............................................................................382.4.8 Instructors exhibit intangible attributes in own actions................................................402.4.9 Hotwashes and Mini-AAR............................................................................................42

2.5 Uses of the Measures ...........................................................................................................432.5.1 Formative Measures for Instructors..............................................................................442.5.2 Quality Assurance and Instructor Education ................................................................442.5.3 Continuous Improvement of Assessments....................................................................452.5.4 Program Evaluation and Organizational Change..........................................................46

2.6 References ............................................................................................................................46


16/22

Table of Contents x


Chapter 3. Principles and Practices of Outcomes Based Training & Education............................503.1 Multifaceted Inquiry.............................................................................................................50

3.1.1 Interaction with Progenitors of OBTE..........................................................................513.1.2 AWG Documents on OBTE .........................................................................................523.1.3 Collaborative Reflection on Participant Observation in CATC ...................................523.1.4 Interaction with Stakeholders .......................................................................................53

3.2 Essential Characteristics of OBTE.......................................................................................533.2.1 The Meaning of Developmental is a Critical Difference..............................................533.2.2 The Definition of Outcomes is a Critical Difference....................................................563.2.3 The Emphasis on Values and Causally Potent Intangibles is a Critical Difference .....583.2.4 The Meaning of Experience is a Critical Difference....................................................613.2.5 The Emphasis on Instructor-Student Interactions is a Critical Difference ...................623.2.6 The Emphasis on Learning to Learn is a Critical Difference .......................................633.2.7 The Emphasis on Collaborative Design and Development is a Critical Difference.....65

3.3 Toward a Grounded Theory for OBTE................................................................................663.3.1 Need for an Integrated Interdisciplinary Framework ...................................................663.3.2 Formative Measures of Instructor Behavior as Evolving Best Practices of OBTE......67

3.4 Emerging Best Practices in OBTE for a Community-Centered Environment.....................683.4.1 Leadership and Enculturation of Soldiers.....................................................................683.4.2 Robust and Adaptable Plan...........................................................................................703.4.3 Instructors as Role Models ...........................................................................................703.4.4 Collaborative Identification of Outcomes and Measures .............................................71

3.5 Emerging Best Practices in OBTE for a Knowledge-Centered Environment .....................713.5.1 Integrated Understanding of Basic Soldier Skills in Full Spectrum Operations ..........723.5.2 Task Relevance of Planned Instructional Events..........................................................723.5.3 Reveal Operational Relevance of Training...................................................................733.5.4 Incorporate Stress into Instructional Events .................................................................733.5.5 Identify General Lessons Learned and Extrapolate to New Situations ........................74

3.6 Emerging Best Practices in OBTE for an Assessment-Centered Environment...................743.6.1 Collaborative Reflection and Problem Solving ............................................................753.6.2 Communication.............................................................................................................753.6.3 Nature and Extent of Guidance.....................................................................................763.6.4 Establish a Pervasive Mindset of Collaborative Reflection..........................................76

3.7 Emerging Best Practices in OBTE for a Learner-Centered Environment ...........................773.7.1 Soldier Motivation and Development of Intangibles....................................................773.7.2 Plan for Development of the Individual .......................................................................783.7.3 Get Students to Take Ownership ..................................................................................783.7.4 Collaborative Reflection as a Means to Develop Self Efficacy....................................79

3.8 References ............................................................................................................................79Chapter 4. Grounded Theory for Values-Based Training & Education.........................................86

4.1 Exploration of Holistic and Functionalistic Underpinnings for OBTE ...............................864.1.1 Fundamental Units of Analysis.....................................................................................874.1.2 Nested Time Scales and Adaptability...........................................................................884.1.3 Adaptability and Ambiguity .........................................................................................904.1.4 Mechanistic Analogies and Predominant Experimental Paradigms .............................92

4.2 Three Pillars for the Scientific Foundation of OBTE ..........................................................93 4.2.1 Ecological Psychology..................................................................................................934.2.2 Self-Efficacy Theory.....................................................................................................974.2.3 Positive psychology......................................................................................................98

4.3 A More Integrated Scientific Infrastructure.......................................................................101


17/22

xi Riccio, Diedrich, & Cortes (Eds.)


4.3.1 Self Determination Theory .........................................................................................1014.3.2 Situated Learning Theory ...........................................................................................1034.3.3 Existential Psychology................................................................................................105

4.4 Building on the Scientific Infrastructure for OBTE...........................................................1094.4.1 Triadic Frameworks....................................................................................................1094.4.2 Further Development ..................................................................................................112

4.5 References ..........................................................................................................................112Chapter 5. Passion and Reason in Values-Based Learning & Development ...............................118

5.1 The Nested Self ..................................................................................................................1185.1.1 An Alternative to Individual versus Collective ..........................................................1185.1.2 Cognition and Reality .................................................................................................119

5.2 Conscious Experience and the Dynamics of Thinking ......................................................1225.3 Emotion, Information, and Engagement ............................................................................125

5.3.1 Ecological Perspective on Emotion ............................................................................1255.3.2 Emotion as Engagement .............................................................................................1265.3.3 Implications for Training and Education....................................................................129

5.4 Emotion, Decision-Making, and Inter-Temporal Choice...................................................1295.4.1 Toward a More Integrated Theory..............................................................................1295.4.2 Emotion and Decision-Making...................................................................................1305.4.3 Emotion and Nested Time Scales ...............................................................................1315.4.4 Neuroeconomics and Inter-Temporal Reasoning .......................................................1325.5.5 Inter-Temporal Reasoning and Adaptive Dynamical Systems...................................133

5.5 Beyond Science..................................................................................................................1345.5.1 Existentialism..............................................................................................................1345.5.2 The Soldier-Scholar as an Emergent Property of a Collective Pursuit.......................135

5.6 References ..........................................................................................................................137Section II. Verification and Validation of OBTE as a Service System ..................................142 Chapter 6. Initial Impressions of Participation in CATC .............................................................1436.1 Methods..............................................................................................................................143

6.1.1 Participants..................................................................................................................1436.1.2 Procedure ....................................................................................................................1436.1.3 Analyses......................................................................................................................144

6.2 Results ................................................................................................................................1446.3 Implications for Service System Development: Peer Review ...........................................1466.4 References ..........................................................................................................................147

Chapter 7. Local Development of Measures of Effectiveness .....................................................1497.1 What do Instructors Believe Soldiers Should Learn in Initial Entry Training?.................1497.2 Measure Development Process ..........................................................................................1507.3 What do OBTE-Trained DS Believe is Important to Assess in BRM/ARM? ................... 1517.4 Implications........................................................................................................................1567.5 Conclusions........................................................................................................................1587.6 References ..........................................................................................................................159


18/22

Table of Contents xii


Chapter 8. Observations of Behavior and Communication in Rifle Marksmanship Training .....1608.1 Methods..............................................................................................................................160

8.1.1 Participants..................................................................................................................1608.1.2 Procedure ....................................................................................................................1608.1.3 Analyses......................................................................................................................161

8.2 Results ................................................................................................................................1638.2.1 Behavior of DS ...........................................................................................................1638.2.2 Behavior and Performance of Privates .......................................................................1658.2.3 Patterns of Communication ........................................................................................1688.2.4 Potential Influence of Instructor Behavior on Performance of Privates.....................170

8.3 Implications for Service System Development..................................................................1718.3.1 Verification of OBTE .................................................................................................1718.3.2 Validation of OBTE....................................................................................................172

8.4 References ..........................................................................................................................173Chapter 9. Impact on Rifle Marksmanship Training....................................................................174

9.1 Behavioral Data Collection During Basic Rifle Marksmanship ........................................1749.1.1 Method........................................................................................................................1749.1.2 Assessment..................................................................................................................1759.1.3 Results An Overview...............................................................................................1779.1.4 Evidence for Influence of OBTE................................................................................1789.1.5 Behavior of Drill Sergeants after Exposure to OBTE ................................................1809.1.6 Behavior of Privates....................................................................................................1829.1.7 Patterns of Communication ........................................................................................1869.1.8 Summary.....................................................................................................................186

9.2 Attitudes Toward an OBTE in Basic Training...................................................................1879.2.1 Method........................................................................................................................1879.2.2 Results.........................................................................................................................187

9.4 References ..........................................................................................................................191Chapter 10. Influence of CATC in an Operational Setting ..........................................................19210.1 Methods............................................................................................................................192

10.1.1 Participants................................................................................................................19210.1.2 Procedure ..................................................................................................................19210.1.3 Analyses....................................................................................................................193

10.2 Results ..............................................................................................................................19310.2.1 Downstream Impact on Marksmanship ....................................................................19310.2.2 Downstream Impact on Training in the Units ..........................................................19410.2.3 Downstream Impact on Self Efficacy.......................................................................195

10.3 Implications for Service System Development: Validation.............................................19610.4 References ........................................................................................................................197

Chapter 11. Implications for Service System Development.........................................................19811.1 Lessons Learned about Transfer of OBTE.......................................................................19811.2 Implications for Service System Development................................................................199

11.2.1 Further Development and Analysis of Stakeholder Requirements for OBTE..........19911.2.2 Further Development of OBTE as a Service System ...............................................199 11.2.3 Further Verification and Validation of OBTE..........................................................201

11.3 References ........................................................................................................................203


19/22

xiii Riccio, Diedrich, & Cortes (Eds.)


Section III. Further Development of OBTE as a Service System ..........................................206Chapter 12. Development of General Measures for Students ......................................................207

12.1 Intent ................................................................................................................................20712.2 Performance Measure Development Process...................................................................207

12.2.1 Phase One: Define Performance Indicators (PI).......................................................20712.2.2 Phase Two: Translate PI into performance measures...............................................20812.2.3 Phase Three: Measure refinement.............................................................................20812.2.4 Phase Four: Retranslation of Measures ....................................................................208

12.3 Product of Measure Development....................................................................................20912.3.1 Learner Perception of the Instructor and Course......................................................20912.3.2 Learner Engagement.................................................................................................21112.3.3 Student Relationship with Teacher ...........................................................................21212.3.4 Student Results .........................................................................................................21412.3.5 Self-Report Measures ...............................................................................................216

12.4 Conclusion........................................................................................................................21712.5 References ........................................................................................................................217

Chapter 13. Adapting OBTE in a Classroom Environment .........................................................21913.1 Intent ................................................................................................................................21913.2 Observing OBTE in the Classroom Environment............................................................219

13.2.1. Participants...............................................................................................................21913.2.2. Procedure .................................................................................................................22013.2.3. Measures ..................................................................................................................220

13.3 Utility of OBTE Measures in a Classroom Environment ................................................22013.3.1 Generality of Measures.............................................................................................22013.3.2. Implications for Improvement of Measures.............................................................22113.3.3 Implications for improvement of course design .......................................................222

13.4 Use of 360 Reviews for Collaborative Reflection..........................................................22313.4.1 The Role of a 360 Review in OBTE .......................................................................22313.4.2 Narrative of a Participant Observer ..........................................................................22513.5 Learning, cognitive load and motivation..........................................................................22813.5.1 The NASA Task Load Index as a subjective measure of student workload.............22813.5.2 Results.......................................................................................................................22913.5.3 Implications ..............................................................................................................230

13.6 Conclusions......................................................................................................................23013.7 References ........................................................................................................................231

Chapter 14. Organizational Climate and Creation of Durable Change ........................................23314.1 The Need ..........................................................................................................................23314.2 Initial Indications of Possible Resistance to Change .......................................................23414.3 Models and Considerations for Sustainable Change........................................................235

14.3.1 The Change Transition Period ..................................................................................23514.3.2 Organizational Culture..............................................................................................23714.3.3 Clarity of Mission and Shared Understanding..........................................................23714.3.4 Relevant Observations During the Current Investigation.........................................23814.3.5 Organizational Support and Incentives.....................................................................238

14.4 Conclusions......................................................................................................................23914.5 References ........................................................................................................................239


20/22

Table of Contents xiv


Chapter 15. Five ways OBTE can enable the Army Leader Development Strategy....................24215.1 Background ......................................................................................................................24215.2 An Emerging Consensus..................................................................................................244

15.2.1 What Part to Balance?...............................................................................................24415.2.2 Improving Training, by Design ................................................................................24515.2.3 Increased Use of dL and Dependence on Self-Development ...................................24615.2.4 Future Orientation, Unknown Requirements............................................................24715.2.5 The Quality Instructor Challenge .............................................................................24715.2.6 Purpose and Design are Key.....................................................................................24815.2.7 A Natural Advantage ................................................................................................24915.2.8 Task Specialization or Generalized Competency.....................................................249

15.3 Conclusion........................................................................................................................25115.4 References ........................................................................................................................252

Epilogue. Integration of Leader Development, Education, Training, and Self-Development.....254Toward Values-Based Standards for Army Doctrinal Requirements ......................................254Nested Standards and Quality Assurance.................................................................................256Needs and Opportunities for Staff & Faculty Development.................................................... 259

A Role for Science and Measurement .................................................................................259Toward Best Practices in Instructor Education....................................................................260Critical Considerations for Further Scientific Investigation ....................................................263

The Necessity of Long-Term Studies ..................................................................................263False Dichotomy of Objective-Subjective...........................................................................264Clarity About What Is Evaluated.........................................................................................265Next Steps ............................................................................................................................266

References ................................................................................................................................268Section IV. Appendices...............................................................................................................270 Appendix A. OBTE Principles & Practices: Instructor Measures................................................271

A.1 Genesis of Formative Measures for Instructors ................................................................271A.2 Principles of Outcomes-Based Training & Education......................................................272A.3 Guide to Using Measures of Instructor Behavior..............................................................276A.4 Complete Menu of Instructor Measures............................................................................279

Appendix B. OBTE Principles & Practices: Student Measures...................................................318B.1 Guide to Using Measures of Student Behavior ................................................................. 318B.2 Complete Menu of Student Measures ...............................................................................319

Appendix C: A Commanders View of Outcomes-Based Training and Education..................... 340Summary ..................................................................................................................................340

Definition.............................................................................................................................340Description...........................................................................................................................340 Elements of OBTE. ..................................................................................................................341Developing the Outcomes....................................................................................................341Developing the Training Plan ..............................................................................................341Conducting Training............................................................................................................342How Training is Assessed....................................................................................................344

Conclusion................................................................................................................................344


21/22

xv Riccio, Diedrich, & Cortes (Eds.)


Appendix D: Warrior Ethos..........................................................................................................345Analysis of the Concept and Initial Development of Applications..........................................345

Current Understanding of Warrior Ethos.............................................................................345Purpose.................................................................................................................................348 Approach..............................................................................................................................348 Expansion of the Definition of Warrior Ethos.....................................................................348The Tenets of Warrior Ethos ...............................................................................................349Clarifying the Definition of Warrior Ethos..........................................................................351Warrior Attributes Derived from the Tenets of Warrior Ethos ...........................................353

References ................................................................................................................................355Supplementary Work Product from Warrior Ethos Project .....................................................355

Appendix E: Indicators of Warrior Ethos.....................................................................................356Methods....................................................................................................................................356

Participants...........................................................................................................................356 Instruments and Facilities ....................................................................................................356Pr