Prologue. Programmatic Inquiry Into Outcomes Based Training & Education

garyText BoxRiccio, G. (2010). A Programmatic Inquiry into Outcomes Based Training & Education. In: Riccio, G., Diedrich, F., & Cortes, M. (Eds.). An Initiative in Outcomes-Based Training and Education: Implications for an Integrated Approach to Values-Based Requirements (Prologue). Fort Meade, MD: U.S. Army Asymmetric Warfare Group. [Cover art by Wordle.net represents word frequency in text.]

1 Riccio

Asymmetric Warfare Group

Prologue: A Programmatic View of the Inquiry into Outcomes-Based Training & Education

Gary Riccio

The Wexford Group International

Historicity of our Research on OBTE This unique investigation resulted from an unusual confluence of events that brought key personnel from the Asymmetric Warfare Group (AWG) together with key personnel in the science team that ultimately was engaged by the AWG. In September 2007, Fred Diedrich and I were involved in a project with the Fort Benning research unit of the Army Research Institute for the Behavioral and Social Sciences (ARI). One task in that project was directed toward identification of formative measures for instructors in the Basic Noncommissioned Officer Course (BNCOC) at Fort Benning. In particular, we were focusing on the Small Arms Proficiency Training Module (SAPT) of BNCOC. Scott Flanagan was invited to participate in the measure development process known as COMPASS (described in Chapter 2). Scott was working on Outcomes Based Training and Education (OBTE) with the AWG at that time. The most important criteria imposed on the COMPASS process for SAPT were the Pentathlete Characteristics (i.e., Warrior Leader, an Ambassador, a Critical and Creative Thinker, a Leader Developer, and a Resource Manager). This introduced an exigency to map the measures of observable instructional behavior to something more abstract, to something more like cultural values. While the Pentathlete Characteristics in particular ceased to become a priority over the course of the project, we consider this kind of mapping to be an important general source of external validity for a set of instructional measures (Sidman, Riccio, Semmens, et al., 2009). In a prior project, we addressed a set of values-based concepts, those embodied in Warrior Ethos (Riccio, Sullivan, Klein, Salter, & Kinnison, 2004; see Appendix D). We found that it is possible to identify relationships between abstract values and concrete behavior of Soldiers in an operational context or training context. These relationships led to a deeper understanding of Warrior Ethos in terms of scientifically traceable concepts and in terms of specific actionable recommendations for the planning and execution of training. This is important because there is a natural skepticism about the meaning of values-based terminology that changes from time to time. In general, we suspect that persistence of a relatively small number of core values can be identified amid such changes in terminology through their common connections to a scientifically and philosophically meaningful foundation of enduring concepts (see Chapters 3, 4, and 5). Given the original intent of the ARI project, it was natural to consider common and convergent themes across projects in which we had first-hand exposure to various programs of instruction in the institutional Army. Our involvement with such programs was through research, thus we were more likely to be exposed to programs in which change was taking place or was being considered. A significant convergence occurred during this project between the SAPT module of BNCOC and an initiative of the AWG to introduce a different approach to Army training and education (Outcomes-Based Training & Education [OBTE]) through its initial application to marksmanship training (see Chapter 1). The AWG initiative in OBTE became important to consider because our initial assessment of instruction in BNCOC, using the measures developed for SAPT module, showed gaps between what BNCOC leadership expected instructors and students to be doing and what was actually occurring in SAPT. At the same time, OBTE was

Prologue 2


receiving increasing attention in several programs of instruction (see Chapters 1 and 11). Both our team and BNCOC leadership began to inquire into the lessons learned by the AWG. By December of 2007, BNCOC leadership began to inquire into the possibility of BNCOC instructors attending the AWGs marksmanship course. The intent of this course is to familiarize trainers with OBTE. At the same time, Scott Flanagan brought information back to the AWG about the COMPASS process and its potential value in bringing scientific rigorousness to OBTE. By January, LTC Michael Richardson (Baker Squadron Commander at the AWG) began to inquire into the possibility of developing measures for OBTE by employing the COMPASS process and the team that ARI was using to develop measures for BNCOC. After some deliberation about intent and purpose, in April 2008, the team began a three-month effort with the AWG to define and measure the practice of OBTE and to begin the development of theoretical underpinnings for the approach. LTC Richardson monitored the effort with the assistance of CSM Michael Cortes, with the approval of COL Robert Shaw, and with visibility to many key personnel in the AWG. Scott Flanagan, Morgan Darwin and Blaise Cornell dEchert (CD) were key informants about OBTE to the science team on behalf of the AWG. The essential activities and findings of the initial phase of the investigation are described in Chapters 2, 6, 7, 8, 10, and 11. At the end of October 2008, the science team was re-engaged for further study of OBTE. In addition to data collection, COL Shaw wanted the team to produce a monograph that would reveal the current and potential scholarly breadth and depth of OBTE. By this time, LTC Richardson had transitioned to a new assignment outside of the AWG and was replaced by LTC Richard Thewes. CSM Cortes became a critical adviser and technical monitor across this change of command. CD and Morgan continued in their influential roles as informants about the history, cultural implications, and ongoing exploration of OBTE in the Institutional Army. The essential activities and findings of the second phase of the AWGs internal scientific inquiry are reported in Chapters 9, 12, 13, and 14. The engagement of the science team by the AWG ended in June 2009 as planned, consistent with the AWGs Operations Order for OBTE. Chapters 3, 4, 5, 15, and the Epilogue were written over several months following the end of the investigation. These chapters reflect the continuing dialog among a variety of stakeholders about the vision, purpose, decentralized adaptation, and programmatic implications of OBTE. The science team was co-led by Fred Diedrich and me. Fred was responsible for all data collection on the ground and for managing the associated personnel, the most extensive part of the investigation (see e.g., Chapters 8, 9, and 13). I was responsible for development of the grounded theory for OBTE (see e.g., Chapter 3, 4, and 5) through continuous interactions with the progenitors of OBTE about the assumptions, history, and expectations of the approach as well as about their ongoing dialog with stakeholders. Fred also was primarily responsible for ensuring that observations of OBTE were as concrete and verifiable as possible by emphasize its manifestations and effects in the behavior of instructors and students. I was primarily responsible for conducting due diligence on the conscious experience of instructors and students in OBTE to achieve a deeper level of understanding of the causes and effects of good instruction. We believe that the balance between behavior and conscious experience is one of the most noteworthy achievements in this investigation. It helped us identify and stay focused on behavior that is meaningful and on conscious experience that is grounded in reality. Amid the division of labor between Fred and me, there was frequent substantial communication between us on all aspects of the project. The purpose and outcome of this association was a reciprocal influence between the evolving theory of OBTE and the findings about the practice of OBTE to move us systematically toward a theory of practice. In particular, our interactions with

3 Riccio


progenitors of OBTE directly influenced the development of measures utilized in the investigation (see Chapters 2, 7, and 12) as well as our understanding of the limits of the investigation and associated gaps in validation of OBTE (see e.g., Chapter 11 and Epilogue). The findings from the AWGs multifaceted investigation, in turn, influenced the concurrent dialectic with OBTE progenitors and stakeholders by providing increasingly clear and relevant questions centered on concrete issues and opportunities in the ongoing implementation of OBTE. The current level of understanding of OBTE would not have been possible without the empirical components of investigation and the theory development that it enabled.

The Approach and Lessons Learned from the Research The research design is consistent with a state-of-the-art Service System Development (SSD) process area of the Capability Maturity Model Integration for Services (CMMI-SVC) and also reflects best practices in scientific support of social and programmatic decision-making (Chapters 1 and 11). Accordingly, the investigation addressed development and analysis of stakeholder requirements (Specific Goal 1 of SSD), development of OBTE as a service system that meets such requirements (Specific Goal 2 of SSD), and verification and validation of OBTE as a service system (Specific Goal 3 of SSD). Specific practices associated with SG 3 include performance of peer reviews, verification of selected service system components, and validation of the service system. Peer review was addressed primarily in the context of surveys administered by the AWG. Verification was conducted primarily through field-based observations in Initial Entry Training (IET) at Fort Jackson, Fort Benning, and Fort Sill that employed measures specifically developed to enable formative assessment of instructor and student behavior. The empirical approach to validation focused on surveys of Soldiers who had been deployed to a combat environment after being exposed to an early version of a field course in marksmanship that demonstrated instructional principles of OBTE. Validation also was addressed through development of grounded theory for OBTE based on established lines of scholarship in the behavioral and social sciences. Collectively, the evidence suggests considerable potential for OBTE in the Army, especially if it becomes explicitly more directed toward staff and faculty development and quality assurance (see Epilogue). It is noteworthy that our findings suggest that OBTE can motivate individuals to take ownership of their own learning and development, and that it can increase self-efficacy for teaching and developing others. The approach motivates a greater interest in instructor-student interactions and how these interactions affect progress toward developmental outcomes while satisfying course-specific learning objectives. While the findings show that there is both an impact of OBTE and considerable room for improvement in Initial Entry Training, the measures of OBTE provide actionable information about the gaps and shortfalls. The findings and associated research have implications for all the factors that influence institutional learning including student diversity and background, capacity for self development, instructional design and development, instructor education, reward and recognition, individual assessment and program evaluation, leadership in an institution of learning, leader development, and the relevant science of measurement. Given our approach to development of OBTE as an integrated instructional service system, it is critical to understand what instructors should do and what they believe they should influence in particular programs of instruction. Toward that end, we developed a set of measures for instructor behavior that exemplifies OBTE and that can be applied to any program of instruction (Chapters 2, Appendix A). We also developed measures for student behavior that complement the instructor measures and that help reveal the efficacy of instructor-student interactions (Chapters 7 and 12, Appendix B). We thus defined OBTE as a verifiable and replicable instructional service system,

Prologue 4


and we explored its links to established theories. We emphasized the utility of these measures in formative assessments for instructors and instructional program evaluation. Moving forward, we stress that definition and measurement of instructor-student interactions and verification of instructional practices is critical to sustain or improve the quality of any instructional service system and to transfer any new approach to particular programs of instruction (Chapters 1 and 11). In essence, definition and measurement of OBTE can help turn conventional wisdom into conventional practice. We believe that tacit knowledge about good training can be made concrete through programmatic scientific inquiry. Of necessity, this also reveals connections with domains of training and education outside the Army. Thus, there is an opportunity to realize the potential for improvement in virtually everything that influences Army training and education (e.g., see Epilogue) by adapting specific practices within and beyond Army training and education and by utilizing the more general scientific underpinnings. The research cited in this monograph provides a theoretical and empirical basis for shared understanding and continuous coordinated improvement across multiple organizations and organizational levels in training and education (see Chapters 14 and 15). The most immediate result of this is the ability to develop and promulgate standards for existing values-based requirements in Army doctrine. The longer-term result is that Army training and education will be aligned more closely and more completely with Army doctrine and the operational exigencies it represents.

Documentation of the Research The purpose of the investigation was to define and measure OBTE to confirm that it satisfies its intended requirements and that it will satisfy end-user expectations during actual implementation of the approach. More specifically, we utilized multiple sources of evidence to understand the implementation of OBTE and its effects on thought and behavior. The structure of the monograph reflects these means and ends (Figure 1). As is evident in this structure, there was not a linear progression through the chapters with respect to development of OBTE as an integrated instructional service system. Consistent with spiral development, verification and validation both informs and is informed by the ongoing refinement of stakeholder requirements and continuous improvement of OBTE as a mature service system. There are reciprocal relationships among these concurrent systems engineering activities. The AWGs intent is to develop OBTE into a mature service system that can be transitioned to the institutional Army. Some of the chapters will have straightforward utility to instructors and their chains of command (e.g., Chapters 2, 12, and the Appendices); that is, they are actionable without undue interpretation. It is not, however, a users manual for Soldiers and instructors. The primary purpose of this document is to support programmatic decision-making (Chapters 1 and 11). In this respect, the most important contribution of this document is that it reveals the depth and breadth of evidence necessary to support decisions about instruction pursuant to requirements in existing Army doctrine. This includes both methods of assessment and grounded theory that are well established in the scientific literature. We also point to scholarship beyond science that may be necessary to understand the inevitable interactions with the unknown that presumably are the reason for emphasis on adaptability and values in preparing Soldiers for Full Spectrum Operations. With grounding in the science that is relevant to values-based adaptability, and recognition that science also has its limits in this regard, this monograph lays the foundation for development of materials specifically for Soldiers and instructors (see e.g., the Epilogue).

5 Riccio


Figure 1. Evolution of the investigation as reflected in the chapters of this monograph.

References Bruny, T., Riccio, G., Sidman, J., Darowski, A., & Diedrich, F. (2006). Enhancing warrior ethos

in initial entry training. Proceedings of the 50th Annual Meeting of the Human Factors and Ergonomics Society, San Francisco, CA.

Freeman, J., Jason, J., Aten, T., Diedrich, F., Cooke, N., Winner, J., Rowe. L., & Riccio, G. (2008). Shared Interpretation of Commander's Intent (SICI). Final Report to the Army Research Institute for the Behavior and Social Sciences, contract number W74V8H-06-C-0004.

Riccio, G., dEchert, B.C., Lerario, M., Pound, D., Bruny, T., & Diedrich, F. (2006). Enhancing Joint Task Force Cognitive Leadership Skills. Report to the Army Research Institute for the Behavioral and Social Sciences, contract number Army contract no. W74V8H-06-P-0186. Vienna, VA: The Wexford Group International.

Riccio, G., Lerario, M., Cornell dEchert, B., Pound, D., Bruny, T., & Diedrich, F. (2006). Training a Joint and expeditionary mindset. Report to the Army Research Institute for the Behavioral and Social Sciences, contract number W74V8H-06-P-0189. Vienna, VA: The Wexford Group International.

Riccio, G., Sullivan, R., Klein, G., Salter, M., & Kinnison, H. (2004). Warrior ethos: Analysis of the concept and initial development of applications. ARI Research Report 1827. Arlington, VA: US Army Research Institute for the Behavioral and Social Sciences.

Sidman, J., Riccio, G., Semmens, R., Geyer, A., Dean, C., & Diedrich, F. (2009). Reshaping Army institutional training: Current training. Final Report to the Army Research Institute for the Behavior and Social Sciences, contract number W74V8H-04-D-0047 DO 0010.

ix Riccio, Diedrich, & Cortes (Eds.)


TABLE OF CONTENTS .

page Prologue: A Programmatic View of the Inquiry into Outcomes-Based Training & Education.......1

Historicity of our Research on OBTE..........................................................................................1The Approach and Lessons Learned from the Research..............................................................3Documentation of the Research ...................................................................................................4

Section I. Development of Stakeholder Requirements for OBTE..............................................6 Chapter 1. Preparation for Full Spectrum Operations ......................................................................7

1.1 Requirements of Full Spectrum Operations ...........................................................................81.2 Outcomes-Based Training and Education (OBTE)..............................................................10

1.2.1 Exemplar of OBTE: Combat Applications Training Course........................................111.2.2 OBTE as a Multifaceted Instructional System .............................................................12

1.3 An Appraisal of Instruction with Respect to OBTE ............................................................131.3.1 A Systems Engineering Framework for Integration and Development of OBTE........131.3.2 Preparation for Validation and Verification .................................................................14

1.4 References ............................................................................................................................17 Chapter 2. Formative Measures for Instructors ..............................................................................20

2.1 Development of Formative Measures ..................................................................................202.1.1 The COMPASS Methodology ......................................................................................202.1.2 Development of Measures for OBTE ...........................................................................21

2.2 Description of Formative Measures .....................................................................................212.2.1 Results of the COMPASS Process................................................................................212.2.2 Elaboration on the Description of Measures.................................................................23

2.3 OBTE Performance Measures: Planning for Training.........................................................232.3.1 Define Outcomes ..........................................................................................................232.3.2 Create a Positive Learning Environment ......................................................................252.3.3 Create the Parameters of Learning................................................................................27

2.4 OBTE Performance Indicators: Training Execution............................................................282.4.1 Communicate the Parameters of Learning....................................................................282.4.2 Training Emphasizes Broad Combat or Mission Success ............................................292.4.3 Customize Instruction When Possible Based on Constraints/Conditions ....................312.4.4 Facilitates Learning of Concepts ..................................................................................322.4.5 Creates a positive learning environment.......................................................................342.4.6 Instructors Utilize Measures of Effectiveness & Self-Evaluation ................................362.4.7 Uses scenarios to facilitate learning..............................................................................382.4.8 Instructors exhibit intangible attributes in own actions ................................................402.4.9 Hotwashes and Mini-AAR............................................................................................42

2.5 Uses of the Measures ...........................................................................................................432.5.1 Formative Measures for Instructors ..............................................................................442.5.2 Quality Assurance and Instructor Education ................................................................442.5.3 Continuous Improvement of Assessments....................................................................452.5.4 Program Evaluation and Organizational Change..........................................................46

2.6 References ............................................................................................................................46

Table of Contents x


Chapter 3. Principles and Practices of Outcomes Based Training & Education............................503.1 Multifaceted Inquiry.............................................................................................................50

3.1.1 Interaction with Progenitors of OBTE..........................................................................513.1.2 AWG Documents on OBTE .........................................................................................523.1.3 Collaborative Reflection on Participant Observation in CATC ...................................523.1.4 Interaction with Stakeholders .......................................................................................53

3.2 Essential Characteristics of OBTE.......................................................................................533.2.1 The Meaning of Developmental is a Critical Difference..............................................533.2.2 The Definition of Outcomes is a Critical Difference....................................................563.2.3 The Emphasis on Values and Causally Potent Intangibles is a Critical Difference .....583.2.4 The Meaning of Experience is a Critical Difference ....................................................613.2.5 The Emphasis on Instructor-Student Interactions is a Critical Difference ...................623.2.6 The Emphasis on Learning to Learn is a Critical Difference .......................................633.2.7 The Emphasis on Collaborative Design and Development is a Critical Difference.....65

3.3 Toward a Grounded Theory for OBTE................................................................................663.3.1 Need for an Integrated Interdisciplinary Framework ...................................................663.3.2 Formative Measures of Instructor Behavior as Evolving Best Practices of OBTE......67

3.4 Emerging Best Practices in OBTE for a Community-Centered Environment.....................683.4.1 Leadership and Enculturation of Soldiers.....................................................................683.4.2 Robust and Adaptable Plan...........................................................................................703.4.3 Instructors as Role Models ...........................................................................................703.4.4 Collaborative Identification of Outcomes and Measures .............................................71

3.5 Emerging Best Practices in OBTE for a Knowledge-Centered Environment .....................713.5.1 Integrated Understanding of Basic Soldier Skills in Full Spectrum Operations ..........723.5.2 Task Relevance of Planned Instructional Events..........................................................723.5.3 Reveal Operational Relevance of Training...................................................................733.5.4 Incorporate Stress into Instructional Events .................................................................733.5.5 Identify General Lessons Learned and Extrapolate to New Situations ........................74

3.6 Emerging Best Practices in OBTE for an Assessment-Centered Environment ...................743.6.1 Collaborative Reflection and Problem Solving ............................................................753.6.2 Communication.............................................................................................................753.6.3 Nature and Extent of Guidance.....................................................................................763.6.4 Establish a Pervasive Mindset of Collaborative Reflection..........................................76

3.7 Emerging Best Practices in OBTE for a Learner-Centered Environment ...........................773.7.1 Soldier Motivation and Development of Intangibles....................................................773.7.2 Plan for Development of the Individual .......................................................................783.7.3 Get Students to Take Ownership ..................................................................................783.7.4 Collaborative Reflection as a Means to Develop Self Efficacy....................................79

3.8 References ............................................................................................................................79 Chapter 4. Grounded Theory for Values-Based Training & Education .........................................86

4.1 Exploration of Holistic and Functionalistic Underpinnings for OBTE ...............................864.1.1 Fundamental Units of Analysis.....................................................................................874.1.2 Nested Time Scales and Adaptability ...........................................................................884.1.3 Adaptability and Ambiguity .........................................................................................904.1.4 Mechanistic Analogies and Predominant Experimental Paradigms .............................92

4.2 Three Pillars for the Scientific Foundation of OBTE ..........................................................934.2.1 Ecological Psychology..................................................................................................934.2.2 Self-Efficacy Theory.....................................................................................................974.2.3 Positive psychology ......................................................................................................98

4.3 A More Integrated Scientific Infrastructure .......................................................................101

xi Riccio, Diedrich, & Cortes (Eds.)


4.3.1 Self Determination Theory .........................................................................................1014.3.2 Situated Learning Theory ...........................................................................................1034.3.3 Existential Psychology................................................................................................105

4.4 Building on the Scientific Infrastructure for OBTE...........................................................1094.4.1 Triadic Frameworks ....................................................................................................1094.4.2 Further Development ..................................................................................................112

4.5 References ..........................................................................................................................112 Chapter 5. Passion and Reason in Values-Based Learning & Development ...............................118

5.1 The Nested Self ..................................................................................................................1185.1.1 An Alternative to Individual versus Collective ..........................................................1185.1.2 Cognition and Reality .................................................................................................119

5.2 Conscious Experience and the Dynamics of Thinking ......................................................1225.3 Emotion, Information, and Engagement ............................................................................125

5.3.1 Ecological Perspective on Emotion ............................................................................1255.3.2 Emotion as Engagement .............................................................................................1265.3.3 Implications for Training and Education ....................................................................129

5.4 Emotion, Decision-Making, and Inter-Temporal Choice...................................................1295.4.1 Toward a More Integrated Theory..............................................................................1295.4.2 Emotion and Decision-Making...................................................................................1305.4.3 Emotion and Nested Time Scales ...............................................................................1315.4.4 Neuroeconomics and Inter-Temporal Reasoning .......................................................1325.5.5 Inter-Temporal Reasoning and Adaptive Dynamical Systems...................................133

5.5 Beyond Science ..................................................................................................................1345.5.1 Existentialism..............................................................................................................1345.5.2 The Soldier-Scholar as an Emergent Property of a Collective Pursuit.......................135

5.6 References ..........................................................................................................................137 Section II. Verification and Validation of OBTE as a Service System ..................................142 Chapter 6. Initial Impressions of Participation in CATC .............................................................143

6.1 Methods..............................................................................................................................1436.1.1 Participants..................................................................................................................1436.1.2 Procedure ....................................................................................................................1436.1.3 Analyses......................................................................................................................144

6.2 Results ................................................................................................................................1446.3 Implications for Service System Development: Peer Review ...........................................1466.4 References ..........................................................................................................................147

Chapter 7. Local Development of Measures of Effectiveness .....................................................149

7.1 What do Instructors Believe Soldiers Should Learn in Initial Entry Training? .................1497.2 Measure Development Process ..........................................................................................1507.3 What do OBTE-Trained DS Believe is Important to Assess in BRM/ARM? ...................1517.4 Implications........................................................................................................................1567.5 Conclusions ........................................................................................................................1587.6 References ..........................................................................................................................159

Table of Contents xii


Chapter 8. Observations of Behavior and Communication in Rifle Marksmanship Training .....1608.1 Methods..............................................................................................................................160

8.1.1 Participants..................................................................................................................1608.1.2 Procedure ....................................................................................................................1608.1.3 Analyses......................................................................................................................161

8.2 Results ................................................................................................................................1638.2.1 Behavior of DS ...........................................................................................................1638.2.2 Behavior and Performance of Privates .......................................................................1658.2.3 Patterns of Communication ........................................................................................1688.2.4 Potential Influence of Instructor Behavior on Performance of Privates .....................170

8.3 Implications for Service System Development..................................................................1718.3.1 Verification of OBTE .................................................................................................1718.3.2 Validation of OBTE....................................................................................................172

8.4 References ..........................................................................................................................173 Chapter 9. Impact on Rifle Marksmanship Training....................................................................174

9.1 Behavioral Data Collection During Basic Rifle Marksmanship ........................................1749.1.1 Method ........................................................................................................................1749.1.2 Assessment..................................................................................................................1759.1.3 Results An Overview ...............................................................................................1779.1.4 Evidence for Influence of OBTE ................................................................................1789.1.5 Behavior of Drill Sergeants after Exposure to OBTE ................................................1809.1.6 Behavior of Privates....................................................................................................1829.1.7 Patterns of Communication ........................................................................................1869.1.8 Summary.....................................................................................................................186

9.2 Attitudes Toward an OBTE in Basic Training...................................................................1879.2.1 Method ........................................................................................................................1879.2.2 Results.........................................................................................................................187

9.4 References ..........................................................................................................................191 Chapter 10. Influence of CATC in an Operational Setting ..........................................................192

10.1 Methods............................................................................................................................19210.1.1 Participants................................................................................................................19210.1.2 Procedure ..................................................................................................................19210.1.3 Analyses....................................................................................................................193

10.2 Results ..............................................................................................................................19310.2.1 Downstream Impact on Marksmanship ....................................................................19310.2.2 Downstream Impact on Training in the Units ..........................................................19410.2.3 Downstream Impact on Self Efficacy .......................................................................195

10.3 Implications for Service System Development: Validation.............................................19610.4 References ........................................................................................................................197

Chapter 11. Implications for Service System Development.........................................................198

11.1 Lessons Learned about Transfer of OBTE.......................................................................19811.2 Implications for Service System Development................................................................199

11.2.1 Further Development and Analysis of Stakeholder Requirements for OBTE..........19911.2.2 Further Development of OBTE as a Service System ...............................................19911.2.3 Further Verification and Validation of OBTE..........................................................201

11.3 References ........................................................................................................................203

xiii Riccio, Diedrich, & Cortes (Eds.)


Section III. Further Development of OBTE as a Service System ..........................................206 Chapter 12. Development of General Measures for Students ......................................................207

12.1 Intent ................................................................................................................................20712.2 Performance Measure Development Process...................................................................207

12.2.1 Phase One: Define Performance Indicators (PI).......................................................20712.2.2 Phase Two: Translate PI into performance measures ...............................................20812.2.3 Phase Three: Measure refinement.............................................................................20812.2.4 Phase Four: Retranslation of Measures ....................................................................208

12.3 Product of Measure Development....................................................................................20912.3.1 Learner Perception of the Instructor and Course ......................................................20912.3.2 Learner Engagement .................................................................................................21112.3.3 Student Relationship with Teacher ...........................................................................21212.3.4 Student Results .........................................................................................................21412.3.5 Self-Report Measures ...............................................................................................216

12.4 Conclusion........................................................................................................................21712.5 References ........................................................................................................................217

Chapter 13. Adapting OBTE in a Classroom Environment .........................................................219

13.1 Intent ................................................................................................................................21913.2 Observing OBTE in the Classroom Environment............................................................219

13.2.1. Participants...............................................................................................................21913.2.2. Procedure .................................................................................................................22013.2.3. Measures ..................................................................................................................220

13.3 Utility of OBTE Measures in a Classroom Environment ................................................22013.3.1 Generality of Measures .............................................................................................22013.3.2. Implications for Improvement of Measures.............................................................22113.3.3 Implications for improvement of course design .......................................................222

13.4 Use of 360 Reviews for Collaborative Reflection..........................................................22313.4.1 The Role of a 360 Review in OBTE .......................................................................22313.4.2 Narrative of a Participant Observer ..........................................................................225

13.5 Learning, cognitive load and motivation..........................................................................22813.5.1 The NASA Task Load Index as a subjective measure of student workload.............22813.5.2 Results.......................................................................................................................22913.5.3 Implications ..............................................................................................................230

13.6 Conclusions ......................................................................................................................23013.7 References ........................................................................................................................231

Chapter 14. Organizational Climate and Creation of Durable Change ........................................233

14.1 The Need ..........................................................................................................................23314.2 Initial Indications of Possible Resistance to Change .......................................................23414.3 Models and Considerations for Sustainable Change........................................................235

14.3.1 The Change Transition Period ..................................................................................23514.3.2 Organizational Culture..............................................................................................23714.3.3 Clarity of Mission and Shared Understanding..........................................................23714.3.4 Relevant Observations During the Current Investigation.........................................23814.3.5 Organizational Support and Incentives.....................................................................238

14.4 Conclusions ......................................................................................................................23914.5 References ........................................................................................................................239

Table of Contents xiv


Chapter 15. Five ways OBTE can enable the Army Leader Development Strategy....................24215.1 Background ......................................................................................................................24215.2 An Emerging Consensus ..................................................................................................244

15.2.1 What Part to Balance?...............................................................................................24415.2.2 Improving Training, by Design ................................................................................24515.2.3 Increased Use of dL and Dependence on Self-Development ...................................24615.2.4 Future Orientation, Unknown Requirements............................................................24715.2.5 The Quality Instructor Challenge .............................................................................24715.2.6 Purpose and Design are Key.....................................................................................24815.2.7 A Natural Advantage ................................................................................................24915.2.8 Task Specialization or Generalized Competency .....................................................249

15.3 Conclusion........................................................................................................................25115.4 References ........................................................................................................................252

Epilogue. Integration of Leader Development, Education, Training, and Self-Development .....254

Toward Values-Based Standards for Army Doctrinal Requirements ......................................254Nested Standards and Quality Assurance.................................................................................256Needs and Opportunities for Staff & Faculty Development ....................................................259

A Role for Science and Measurement .................................................................................259Toward Best Practices in Instructor Education....................................................................260

Critical Considerations for Further Scientific Investigation ....................................................263The Necessity of Long-Term Studies ..................................................................................263False Dichotomy of Objective-Subjective ...........................................................................264Clarity About What Is Evaluated.........................................................................................265Next Steps ............................................................................................................................266

References ................................................................................................................................268 Section IV. Appendices...............................................................................................................270 Appendix A. OBTE Principles & Practices: Instructor Measures................................................271

A.1 Genesis of Formative Measures for Instructors ................................................................271A.2 Principles of Outcomes-Based Training & Education ......................................................272A.3 Guide to Using Measures of Instructor Behavior..............................................................276A.4 Complete Menu of Instructor Measures............................................................................279

Appendix B. OBTE Principles & Practices: Student Measures ...................................................318

B.1 Guide to Using Measures of Student Behavior .................................................................318B.2 Complete Menu of Student Measures ...............................................................................319

Appendix C: A Commanders View of Outcomes-Based Training and Education .....................340

Summary ..................................................................................................................................340Definition .............................................................................................................................340Description...........................................................................................................................340

Elements of OBTE. ..................................................................................................................341Developing the Outcomes....................................................................................................341Developing the Training Plan ..............................................................................................341Conducting Training ............................................................................................................342How Training is Assessed....................................................................................................344

Conclusion................................................................................................................................344

xv Riccio, Diedrich, & Cortes (Eds.)


Appendix D: Warrior Ethos..........................................................................................................345Analysis of the Concept and Initial Development of Applications..........................................345

Current Understanding of Warrior Ethos.............................................................................345Purpose.................................................................................................................................348Approach..............................................................................................................................348Expansion of the Definition of Warrior Ethos.....................................................................348The Tenets of Warrior Ethos ...............................................................................................349Clarifying the Definition of Warrior Ethos..........................................................................351Warrior Attributes Derived from the Tenets of Warrior Ethos ...........................................353

References ................................................................................................................................355Supplementary Work Product from Warrior Ethos Project .....................................................355

Appendix E: Indicators of Warrior Ethos.....................................................................................356

Methods....................................................................................................................................356Participants...........................................................................................................................356Instruments and Facilities ....................................................................................................356Procedure .............................................................................................................................356

Results ......................................................................................................................................358Qualitative Findings.............................................................................................................358Quantitative Findings...........................................................................................................358

Discussion ................................................................................................................................359

Evolution of the investigation as reflected in the chapters of this monograph.