Technical Report 847

The Application of Computers toLearning in the Command and GeneralStaff College: Identification of ComputerOpportunities

(0O- Richard R. Sandoval, Charles T. Thorn, and Mary S. Trainor

I' Cognitive Engineering and Design Research Team

~ June1989

D--: DTIC.

~ United States Army Research Institutefor the Behavioral and Social Sciences

Approved for public release; distribution is unlimited.

89 8 14 004

U.S. ARMY RESEARCH INSTITUTE

FOR THE BEHAVIORAL AND SOCIAL SCIENCES

A Field Operating Agency Under the Jurisdiction

of the Deputy Chief of Staff for Personnel

EDGAR M. JOHNSON JON W. BLADESTechnical Director COL, IN

Commanding

Technical review by

Peter J. Legree -

William Sprenger %-xi>

D--

NOTICES

STRIBUTI Pui distribution of th' been made by addresscospondenc concerning *bution of rts to: U. y Re Institute the

B 'oral an Social Sciences, P I-POX, 5001 Ei wer ye., Alexand Vir ia22333-

FINAL DISPOSITION: This report may be destroyed when it is no longer needed. Please do notreturn it to the U.S. Army Research Institute for the Behavioral and Social Sciences.

NOTE: The findings in this report are not to be construed as an official Department of the Armyposition, unless so designated by other authorized documents.

UNCLASSIFIED

* SECURITY CLASSIFICATION OF THIS PAGE

Form ApprovedREPORT DOCUMENTATION PAGE OMB No. 0704-0188

la. REPORT SECURITY CLASSIFICATION lb. RESTRICTIVE MARKINGS

Unclassified --

2a. SECURITY CLASSIFICATION AUTHORITY 3. DISTRIBUTION/AVAILABILITY OF REPORT--. _ Approved for public release;

2b. DECLASSIFICATION/DOWNGRADING SCHEDULE distribution is unlimited.

4. PERFORMING ORGANIZATION REPOPT NUMBER(S) 5. MONITORING ORGANIZATION REPORT NUMBER(S)

ARI Technical Report 847

6a. NAME OF PERFORMING ORGANIZATION 6b. OFFICE SYMBOL 7a. NAME OF MONITORING ORGANIZATIONCognitive Engineering Design (If applicable) U.S. Army Research Instituteand Research Team (CEDAR) -- Fort Leavenworth Field Unit

6c. ADDRESS (City, State, and ZIP Code) 7b. ADDRESS (City, State, and ZIP Code)Military Systems Group, A-6, P.O. Box 1633, MS P.O. Box 3407

M997, Los Alamos National Laboratory Fort Leavenworth, KS 66027-0347

Los Alamos, NM 87545

Ea. NAME O! FUNDING/SPONSORING 8b. OFFICE SYMBOL 9. PROCUREMENT INSTRUMENT IDENTIFICATION NUMBERORGANIZATION U. . rmy 'esearch (If applicable)

Institute ror e ieavora1and Social Sciences PERI-S

Bc. ADDRESS (City, State, and ZIP Code) 10. SOURCE OF FUNDING NUMBERSPROGRAM PROJECT TASK WORK UNIT

5001 Eisenhower Avenue ELEMENT NO. NO. NO ACCESSION NO.Alexandria, VA 22333-5600 63739A 793 144 C7

11. TITLE (Include Security Classification)The Application of Computers to Learning in the Command and General Staff College:

Identification of Computer Opportunities

12. PERSONAL AUTHOR(S)Sandoval, Richard R.; Thorn, Charlie T.; and Trainor, Mary S. (CEDAR Team)

13a. TYPE OF REPORT 1!3b. TIME COVERED 14. DATE OF REPORT (Year, Month, Day) 15. PAGE COUNT

Final FROM 87/01 TO 87/12 1989, June 52

16. SUPPLEMENTARY NOTATION .

Dr. Stanley M. Halpin was the ARI monitor of this effort.

17. COSATI CODES , 18. SUBJECT TERMS (Continue on reverse if necessary and identify by blcck num ner)

FIELD GROUP SUB-GROUP Computer based training,Computer Assisted Instruction (CAI)Intelligent tutoring systems, - ._ (Continued)

19, ABSTRACT (Continue on reverse if necessary and identify by block number)

% A front-end analysis was conducted regarding the expansion of the use of computers in

instruction at the U.S. Army Command and General Staff College (CGSC). Instruction at the

CGSC is aimed principally at small groups of students. The underlying teaching philosophy

emphasizes the students' responsibility to apply in classroom interactions, concepts, and

procedures learned earlier in individual study. This instructional strategy readily lends

itself to applications of computers to learning (ACL) from each of the categories used by

the project team. However, introduction of many computer applications into college class-

rooms would need to take place over a period of years, depending upon how quickly the

needed computers can be acquired and the appropriate software written. This report suggests

appropriate ACL for each subcourse of selected courses taught at the CGSC and how much of

each subcourse migpt eventually be taught by using those applications.

(Continued)

20. DISTRIBUTION/AVAILABILITY OF ABSTRACT 21. ABSTRACT SECURITY CLASSIFICATION

rUNCLASSiFiED/UNLIMITED 0 SAME AS RPT. C DTIC USERS Unclassified

22a. NAME OF RESPONSIBLE INDIVIDUAL '22b. TLEPHONE (Include Area Code) 22c. OFFICE SYMBOLJon -Faiiesen (913) 684-4933 PERI-SL

DO Form 1473, JUN 86 Previous editions are obsolete. SECURITY CLASSIFICATION OF THIS PAGE

UNCLASS IFIED

• I | li

UNCLASSIFIED

SECURITY CLASSIFICATION OF THIS PAGE(When Data Entered)

ARI Technical Report 847

18. SUBJECT TERMS (Continued)

SimulationsFunctional requirements definitionFront-end analysis

Systems approach to trainingInstructionDoctrine development

Automated classroom

Command and controlStaff officerKnowledge, skills, abilities

19. ABSTRACT (Continued)

This report presents the findings and recommendations of Task G of theFront-end Analysis. Other reports are separately bound. The reports all have

the beginning title, The Application of Computers to Learning in the Commandand General Staff College. -The follow-on headings for the reports are as

follows:

A Front-End Analysis Study

CGSC AnalysisAnalysis of Staff Officer Knowledge, Skills, and AbilitiesAssessment of Computers in Education at Various Institutions

Technology AssessmentAssessment of Computer Literacy in CGSC

Analysis of Institutional and Financial ConstraintsArmy Command and Control Concepts Study

Comparison of Knowledge, Skills, and Abilities to CGSC Learning Objectives

UNCLASSIFIED

SECURITY CLASSIFICATION OF THIS PAGE(When Data Entered)

ii

Technical Report 847

The Application of Computers to Learning in theCommand and General Staff College: Identification of

Computer Opportunities

Richard R. Sandoval, Charles T. Thorn, and Mary S. TrainorCognitive Engineering and Design Research Team

Field Unit at Fort Leavenworth, KansasStanley M. Halpin, Chief

Systems Research LaboratoryRobin L. Keesee, Director

U.S. Army Research Institute for the Behavioral and Social Sciences5001 Eisenhower Avenue, Alexandria, Virginia 22333-5600

Office, Deputy Chief of Staff for PersonnelDepartment of the Army

June 1989

Army Project Number Human Factors In Training2Q263739A723 and Operational Effectiveness

Approved for public release; distribution is unlimited.

iii

FOREWORD

The Command and General Staff College (CGSC) trains officers to act as

coordinating staff members and to assume command positions at Brigade andhigher echelons. The Deputy Commandant of CGSC requested a frLUt-end analysisto determine how emerging technology could be used at CGSC to close the gapbetween the classroom and "real" experience. The talents of the Cognitive

Engineering Design and Research Team (CEDAR) of the Los Alamos NationalLaboratory were elicited for performing the analysis, in cooperation with theFort Leavenworth Field Unit of the Army Research Institute for the Behavioraland Social Sciences (ARI)." The results of the study marked a major milestonein defining requirements for the implementation of instructional technology inthe Army's key training arena for tactical command and control operations.

The portion of the overall study documented in this report involved thematching of computer technological opportunities to training objectives.

This effort was supported under ARI research task 144, Advanced Tech-nology for Command and Staff Operations. The work was performed under thelong-standing memorandum of understanding between ART and CGSC entitled,Research and Evaluation program for Present and Future Command and ControlRequirements and Operations, 31 May 1983. On 24 November 1986 MG Frederick M.Franks, then Deputy Commandant, requested assistance for this project byletter and on 15 December 1986, Dr. Stanley M. Halpin, Chief, Ft. LeavenworthField Unit, responded affirmatively by return letter. Status briefings were

provided to the CGSC throughout the project with final briefings presented tothe Assistant Deputy Commandant and CGSC Directors on 14 August 1987 and to MGGordon R. Sullivan, Deputy Commandant, on 13 November 1987. The effort hasbeen accepted and endorsed enthusiastically by CGSC and, as a result, imple-mentation planning is taking place.

EDGAR M. JOHN ONTechnical Director

v

PREFACE

This report concerns one of the Army's most important institutions, the

United States Army Command and General Staff College (CGSC), which is the fontof tactical and operational knowledge for Army forces. This knowledge is amajor force multiplier that holds potential enemies at bay, enhances deter-rence, and thus moves us closer to a lasting peace.

Thp CGSC is a complex organization that is undergoing a major changebrought about by computer technology. Further, the pace and scope of the

change is faster and broader than in the past. The Army, educational tech-nology, and tactical doctrine are changing concurrently. CGSC must not onlykeep up, but must also assist in the process because the College is an instru-ment of change for the Army. CGSC is the leader of the other Training andDoctrine Command (TRADOC) schools ar centers, which directly affect almost

every field grade officer in the Army. Finally, the College prescribes howthe Army will fight and how its staffs will function.

This view of the CGSC was held by the Los Alamos project team and sug-gests that the actions to be initiated, based upon this report, are farreaching because they will influence the quality of our Army in the years tocome. The study was conducted in this spirit.

The following people, listed alphabetically, contributed to this front-

end analysis study:

Andrew E. Andrews

David L. HudsonDavid R. LittlefieldDesiree Marr

Carol Ann MartzMichelle M. Osborn

Richard R. SandovalPatricia A. Schultejann

Lois M. SpangenbergCharles T. Thorn

Mary S. Trainor

ANDREW E. ANDREWS

CEDAR Team Leader

vi

THE APPLICATION OF COMPUTERS TO LEARNING IN THE COMMAND AND GENERAL STAFF

COLLEGE: IDENTIFICATION OF COMPUTER OPPORTUNITIES

EXECUTIVE SUMMARY

Requirement:

This study was conducted to determine how best to introduce computersinto the Command and General Staff College (CGSC) curricula. This reportdocuments Task G of the study, which identifies the expanded use of computers

to achieve instructional objectives at CGSC.

Procedure:

Information was combined from the study's Task A (CGSC Curricula Analy-sis), Task B (Analysis of Staff Officer Knowledge, Skills, and Abilities), and

Task F (Comparison of Knowledge, Skills, and Abilities to CGSC Learning Objec-tives). Courses of the resident phase of the Combined Arms and Services Staff

School (CAS3), of the curriculum of the Command and General Staff OfficersCourse (CGSOC), and of the School of Advanced Military Studies (SAMS) wereexamined to determine how computer technologies could be applied. Conceptualcategories were developed to classify applications of computers to learning(ACL). For each subcourse, estimates were made of the proportion of classroomhours that might be taught by each ACL category. A summary analysis of poten-tial, risks, and costs was made for each of the ACL categories and an actionwas recommended.

Findings:

The analysis suggested that all CGSC instruction would benefit from usingcomputers for administrative tasks associated with instruction because of in-

creases in the productivity of the staff and faculty. It was estimated that55% of resident classroom instruction would be appropriate for ACL, with anexpected improvement due to ACL. The realization of the ideal use of ACL will

require many prerequisite actions in terms of acquisition planning and course-

ware and software development.

Utilization of Findings:

This front-end analysis will be used to guide implementation of a programto expand the use of computers in CGSC instruction. The findings identify the

initial steps to be taken in the implementation and describe the actions thatshould be taken for a long term payoff in ACL. This phase of the analysisidentifies appropriate opportunities for the use of computers in the CGSC

curricula.

vii

THE APPLICATION OF COMPUTERS TO LEARNING IN THE COMMAND AND GENERAL STAFFCOLLEGE: IDENTIFICATION OF COMPUTER OPPORTUNITIES

CONTENTS

Page

INTRODUCTION ..................................

TASK DESCRIPTION................................3

Goal......................................3Relationship of Task G to the Total Project ................. 3Assumption.................................3

METHODOLOGY.................................5

Subcourse Examination............................5

Products of the Examination. ....................... 5

ANALYSIS OF ACL ELEMENTS.............................

Description of ACL Elcments. ....................... 8Networking................................8The Administration Element.........................0Testing..................................13Computer-Assisted Instruction ....................... 14Simulations for Individual Training (SIT). ............... 20Intelligent Tutoring Systems (ITS)....................22Gaming..................................24Simulations for Collective Training (SCT). ............... 27High-Tech Classroom ............................ 28Data...................................29

RESULTS...................................39

Presentation of Project Team Suggestions ................ 39Summary..................................39

DISCUSSION/INTERPRETATION..........................41

The Project Team Ideal Situation .................... 41Prerequisites .............................. 41Selecting Computer Applications. .................... 41

ix

CONTENTS (Continued)

Page

CONCLUSIONS.................................43

Instructional Strategy..........................43Computer Opportunities..........................43Hardware Acquisition...........................43Courseware Development..........................43

REFERENCES..................................45

LIST OF TABLES

Table I. ACL usage for GAS3 Phase II course. ............... 31

II. ACL usage for CCSOC core curriculum ............... 32

III. ACL usage for CGSOC electives ................. 33

IV. ACL usage for SAMS course....................36

V. Percentage totals of ACL usage for courses of CCSC. .......37

Vi. Number of classroom hours using the indicated categoryof ACL and percentage of total course hours ........ 4

LIST OF FIGURES

Figure 1. Notional network for CGSC. .................... 9

2. Effects of experience and pedagogy on CAI. ............ 15

x

THE APPLICATION OF COMPUTERS TO LEARNING IN THE COMMAND AND GENERALSTAFF COLLEGE: IDENTIFICATION OF COMPUTER OPPORTUNITIES

INTRODUCTION

In completing this task, the project team used the information gathered in doing Task A(CGSC Analysis), Task B (Analysis of Staff Officer Knowledge, Skills, and Abilities), and TaskF (Comparison of Knowledge, Skills, and Abilities to CGSC Learning Objectives) to providesuggested methods of achieving training objectives through the expanded use of computers ininstruction at the Command and General Staff College (CGSC). The results constitute theconsidered judgment of the project team based on its own experience; however, applying com-puters to learning, which is both an art and a science, is generally in its infancy. In particular,conceptual categories to classify applications of computers to learning (ACL) were devised bythe project team in the absence of existing experience with some of the applications in con-texts like that at CGSC.

With regard to the specific material taught ir, the various schools of CGSC, the use of com-puter simulations both for individual and collective training, for example, will have to awaitthe development of suitable models and their conversion to programs in forms that are ap-propriate for the kinds of computers that will be available to the instructors and students.

The history of combat simulation modeling for use with computers principally reflectspreoccupation with the weapons acquisition process or other concerns of the analytic com-munity that has developed or used the models rather than with teaching combined armsdoctrine and staff operations either in the classroom or in the Army in the field. Clearly, modelsdesigned for one function will not necessarily adequately serve the purposes of the others.

Similarly, intelligent tutoring systems (ITS) for use at CGSC are today purely notional andwill remain so until they are actually designed, which, as with simulations, will be a difficult,time consuming, and possibly expensive task with benefits that will have to be weighed againstthese factors. The same will be true to a lesser degree for the other categories of ACL. Thedevelopment of appropriate courseware may be the greatest obstacle to be overcome in ex-panding the use of computers in CGSC classrooms.

In the meantime, instruction given to small groups of students is the preferred instruction-al strategy at CGSC and appears to maximize opportunities to apply computers in CGSC cur-ricula and will presumably be more widely implemented as new facilities are made availableto the schools of the College. ACL will not supplant the role of the instructor in staff group in-struction, whose efficacy depends in large part on success in stimulating and guiding mutual-ly enlightening interactions among the students toward the end of honing the problem solvingskills they will need throughout their careers. The role of the computer in and out of the class-room is to enhance the ability of the instructor to play his role effectively by easing his ad-ministrative tasks, by affording an efficient means of conveying information, and by offering

1!

7n

the possibility of easily portraying a wide variety of situations with which to challenge the

students' mastery of curricu',m material.

2

TASK DESCRIPTION

Goal

The goal of Task G was to identify specific opportunities to apply computers to the learn-ing process at CGSC and to suggest specific methods of exploiting those opportunities in apreferred instructional strategy for imparting the knowledge, skills, and abilities (KSA) stu-dents are expected to acquire at the College.

Relationship of Task G to the Total Project

In completing Task G, the project team drew from the results of Tasks A, B, and F to servethe goals of the project. Task G contributes to reaching those goals by describing a level of useof computers at CGSC that represents the final stages of the acquisition and deployment ofhardware and the development of suitable software. This level of use could be attained onlyafter a number of years determined by the rate that would be achieved in the deployment anddevelopment, which will be subject to the constraints identified in the Task D report entitledAnalysis of Institutional and Financial Constraints.

Assumption

ACL at CGSC from each category defined by the project team in this report will bedeveloped in the next few years and will represent significant improvements in teachingmethods applied in accordance with a preferred instructional strategy.

3

METHODOLOGY

Subcourse Examination

The project team examined each subcourse of the resident phase of the course taught bythe Combined Arms and Services Staff School (CAS3), of the core curriculum and electivesof the Command and General Staff Officers Course (CGSOC), and of the course taught bythe School of Advanced Military Studies (SAMS) for the purposes of determining whichcategories of ACL would be suitable for achieving at least some of the subcourse learning ob-jectives. The examination also resulted in estimates of the proportion of the number of sub-course classroom hours that might be devoted to ACL of each category after the appropriatecourseware had been designed and implemented. Part of the examination dealt with assess-ing alternative instructional strategies for their applicability to computer-based training atCGSC.

Products of the Examination

The judgments resulting from the examination of the subcourses are the suggestions of theproject team for aiming effort at CGSC toward the eventual goal of achieving the best use ofcomputers in its instruction.

5

ANALYSIS OF ACL ELEMENTS

This section provides the project team's judgments concerning how computers could beapplied to learning in the CGSC. These judgments describe how the ACL taxonomy that wasdeveloped by the project team can be applied to the College. Because the discipline of com-puters in education is still in its infancy, the ACL taxonomy is a viewpoint from a 1987 perspec-tive and will necessarily evolve over time. This information represents an ideal world of thefuture that will expand as the ideal develops.

The project team examined the CAS3 Phase II, CGSOC core curriculum, CGSOC elec-tives, and SAMS for opportunities for computer usage. These courses make up the bulk of therelevant opportunities for computer usage in the College.

Upon examining the curricula listed above, the project team found that the CGSOC elec-tives were different from the others in a fundamental way. In academic year 1987-1988, theelectives comprise some 3,120 hours of subcourses ranging in length from 15 to 90 hours. Ifincluded, these 3,120 hours would heavily influence the results. Each student normally takes210 hours of electives. Some electives are not taught because sufficient numbers of studentsdo not enroll. For these reasons, the project team felt that the electives analysis was distinctand separable from the other analyses in the final tabulations.

The original data for this analysis were those in the POI (program of instruction) andcatalog provided by the College. CAS3 and SAMS data were based on POI current foracademic year 1986-1987. Little, if any, change was anticipated in the curricula of these cour-ses for academic year 1987-1988. Late in the study, however, the project team obtained theCGSC Catalog for Academic Year 1987-1988 (CGSC Circular 35 1-1, May 1987). Substantialchange was noted in the curricula for the CGSC core and elective subcourses. The projectteam chose to use these current curricula for opportunities for the ACL because they wouldbe more beneficial to the College.

To analyze the potential ACL, the project team used the taxonomy for the ACL developedearlier in the study. As developed, the taxonomy consists of the following elements:

Administration Intelligent Tutoring Systems (ITS)Testing GamingComputer-Assisted Instruction (CAI) Simulation for Collective Training (SCT)Simulation for Individual Training (SIT)

The taxonomy was modified by adding another entry "NA" for no application other thanadministration. The project team emphasizes that a taxonomy attempting to describe all pos-sible categories of ACL has not been published before and that this taxonomy is conceptualin nature. By this expression, the project team means that many of the application ideasdescribed in the taxonomy do not exist, and some have not been attempted. For example, nogame exists that could be applied to the educational environment at the College, and it is prob-able that none will exist in the next few years.

The analysis was conducted by estimating the percentage of the hours of each subcoursefor each computer application in the modified taxonomy. The NA entry was created to be able

7

to account for 100% of each subcourse. The tables that follow show the project team's es-timates for the number of hours in which ACL could be used for all subcourses for CAS3

Course Phase II, CGSOC Core Courses, CGSOC Electives, and SAMS. The comment columnshows, in short form, ideas for how ACL could apply in each subcourse.

Description of ACL Elements

The results of this analysis consist of a detailed description of each of the elements of ACL.Included in these detailed descriptions are Tables I through V, which show the project team'sassessment of where each ACL could be applied in subcourses throughout the College. TheACL taxonomy that was developed by the project team is applied to the curricula of the CGSCin these tables. A table is presented for the CAS3 Phase II resident course, for the core cur-riculum of CGSOC, for the electives of CGSOC, and for the SAMS course. Each curriculumis broken down into its subcourses as organized by the latest POI given to the Los Alamosproject team.

The information recorded in the columns of the tables includes the number of hours in thesubcourses, the percentages of the subcourses for which the various categories of ACL couldbe applied, and comments. At the bottom of each table a summation line and a percent lineare included. The summation line indicates the project team's assessment of the number ofcourse hours to which the relevant ACL could be applied. Table V shows two summations,one for total hours and percentages for each of the ACL elements for all four courses examinedhere and the other for the totals without the CGSOC electives included.

Networking

Before considering the individual ACL elements, a discussion of the impact of networkingon the College is in order. In many ways, networking is a part of the administration elementof ACL, but it transcends administration because other elements of the taxonomy depend onnetworking.

An analysis of the relationships among the faculty, students, facilities, and organizationswas performed. Figure 1 shows the layout for the following functional requirements for net-working in the College:

* Faculty access to information and databases for both instruction anddoctrine development.

" Faculty communication with other faculty members and individuals at otherservice schools, including electronic mail, asynchronous conferencing, anddata transfer.

" Faculty-student communication for exchange of homework and for discus-sion of instructional material, including asynchronous communication.

" Faculty, staff, and student administrative support.

" Student access to information and databases.

8

!SUDN IN POSTASIMULATION 11

STUDENT IND

IN, ,CGSC RESIDENT STUDENTS I - VAE

CARL OFFICERCAFAT

FUTURE SIMULATION

Fig. 1. Notional network for CGSC.

At present, the College has rudimentary plans for implementing networks in the existingand planned facilities. Therefore, the development of a plan for networking that meets thefunctional requirements of the College must be clearly and carefully considered. Implemen-tation of these plans will occur over a period of years. Initially, the network should supportelectronic mail between faculty inside the College and other service schools outside FortLeavenworth and between faculty inside the College and people in other Fort Leavenworthactivities. Further, the network must support access to CARL (Combined Arms ResearchLibrary), providing remote access by both faculty and students to the electronic card catalogand, eventually, to abstracts and documents.

The network E-mail concept must provide faculty-student interchanges through the"homework" box shown. This box provides for the option of testing of both resident and non-resident students and for transmitting term papers and other assignments as required for anysubcourse at CGSC. The resident student links include access from learning centers andquarters as well as from classrooms. Students should have modems available that allow accessfrom quarters. The classroom links should be implemented in the longer term and must be ex-pandable to meet future needs of the College.

A positive image of the College from the outside must be deliberately nurtured. The net-work elements that concern SOCS (School of Corresponding Studies) students, other serviceschools, and the officer corps at large make this view possible. Using established Army net-works and commercial links, such as Tymnet, can significantly improve the quality of instruc-tion delivered to students in SOCS and the interaction of the officer corps with the College.Doctrine development would certainly benefit.

9

Achievement of such a complete and complex network capability must be time-phased. In-itially, the network system would provide student administrative support, access to the facul-ty by SOCS students, SOCS resident student asynchronous conferencing, and SOCS studentaccess to the resources of CARL and/or other libraries. The presentation of instructionthrough this network would be longer term and requires further study. However, the postingof changes to instructional material could be a near-term reality and would require that thedistribution of study materials to SOCS students be in electronic form, such as compact discread-only-memory technology. This electronic publishing technology is available today.

SOCS has about 33,000 students enrolled; and with the possibility that the reserve com-ponents will require CAS3 instruction, SOCS could expand to 45,000 or more students. Thus,SOCS has the greatest potential influence in the development of the officer corps of the 'TotalArmy." This potential suggests that SOCS automation improvements be given high priority inthe College's overall plan.

The project team believes the only higher ACL priority is to provide computers for thesupport of faculty and staff. This support will affect all resident and nonresident instruction aswell as doctrine development. When the faculty computer network is complete, SOCS wiii bein a position to extend some of the advantages of resident instruction to nonresident studentsthrough the off post network. The electronic linking of SOCS to the rest of the Army willprovide the mechanism to enhance the outside-in view of the College and make the Collegethe electronic university of tactical and operational doctrine.

A description of the individual ACL elements applicable to the College follows. In eachdiscussion, a narrative of the element and its College-specific applications is presented. Wherepossible, a discussion of the learning methodology that the element and its applications sup-port is included. For the purpose of clarity and at the end of each element discussion, theproject team includes a brief summary of the potential for the application, its risk, its estimatedcosts, and the action that the project team recommends to be taken for the element.

The Administration Element

Administration as an element of the ACL is primarily concerned with increasing the ef-ficiency of the way the College's missions are administratively supported. The primary mis-sions are to develop future combat leaders and to develop and promulgate appropriate Armydoctrine. Administering these missions requires performing record keeping, word processing,database management, course registration, and preparation and presentation of instructionaland doctrinal material. As these functions are implemented on computers throughout the Col-lege, an environment that encourages the users to experiment with alternate ways of usingcomputers in the instruction and doctrine development areas should evolve. This environmentwill improve the quality of mission performance in the College.

Many quality hardware configurations and software applications for administration arecommercially available. Therefore, the hardware and software selection, acquisition proces-ses, and the training of users become the most significant expenses in implementing the ad-ministrative use of computers at the College.

10

The potential applications within the College administration are numerous. These applica-tions include the development and production of lesson materials and training aids; doctrinedevelopment to include research, data and information collection, and documentation (suchas writing FMs); expanding access of the officer corps to the College; and generally easing ad-ministrative management burdens. When this administrative workload is eased, faculty willhave more time to concentrate on course content. Higher quality instruction and doctrinedevelopment will probably result. Together, these applications will improve both content andproductivity in the College administration area.

Several issues must be considered in the implementation of computerized administration.First, the hardware configurations and collections of software to be used for each functionalarea or office must be identified and then standardized. This process involves examining thefunctions and organizational structure of the College to define the hardware and software re-quirements. A small number of standard hardware configurations will be needed, and a greaternumber of software tools will be needed. It is important that standard interoperable configura-tions of both hardware and software be identified so that College users will be able to com-municate and collaborate on-line with others both inside and outside of the College. Impliedin this task is the development of a standard interface tool that specifies how the software ispresented to the user and how the user controls the software. This interface must allow for sys-tem growth. The window approach developed by XEROX, marketed by Microsoft, and seenas the strength of Apple's Macintosh meets these needs for the user friendly human-computerinterface.

The second issue is to provide to the faculty and staff the standard personal computers withthe appropriate collection of software programs. Providing these systems also involves thetraining of personnel for their effective use. Implied here also is instilling in the faculty andstaff the confidence that computers can and will improve their personal productivity. The ex-perience of the project team is that if people believe that these tools are going to be effective,then there is a higher probability of success.

The third issue to be handled is the use and availability of computers for students. It is notreasonable to provide each student with a personal computer. However, each student musthave access to a computer and its software tools. Civilian universities provide learning centersthat make computers and their software available. The College must continue to provide forsuch centers in its future building and remodeling plans. While this approach decreases theburden of providing each student with a computer, it creates the additional cost of providingand operating the facilities. Because the facilities must be available during nonduty hours,operating costs can be significant. Because this facility could also be used as classrooms forcomputer training in the College, the expense of learning center implementation can bereduced.

As in the case for the faculty and staff, a standard configuration of hardware and softwaretools for students is required. The equipment and tools should be available for student pur-chase. Further, a variety of mutually compatible systems from a basic model up through thefull size, luxury model must be available to meet the needs of the individual student budget.The College book store could make this hardware and software available to students, faculty,and staff. This approach was successfully implemented at Drexel University.

11

Administration

Potential: The systematic introduction of computers into office support, developing instruc-tional and doctrinal material, publishing training aids, and networking the faculty, staff, andstudents will increase efficiency in all areas of the College. It will also facilitate communica-tion within the College and with the Total Army. The universal availability of computers tofaculty, staff, and students at CGSC will also eventually facilitate the widespread use of com-puters in the field and potentially create a more effective combat force.

Risk: Minimal risk is involved in the administrative use of computers at the College. Hardwareand software for most administrative applications are commercially and competitively avail-able. Only a universal interface linking the different applications has not been developed. Thesole risk seen is in the selection process for the standard configurations of hardware andsoftware described. A problem exists here because so many choices are commercially avail-able yet not necessarily mutually compatible.

Costs: The project team is using the following five categories for cost estimation:

* Faculty and Staff. The project team estimates that the cost for each staff and facul-ty configuration would be approximately $3,500, not including training expenses.Annual maintenance costs cannot be determined until system specifications havebeen defined, but they are estimated at 10 percent of the initial hardware invest-ment cost per year.

" Students. There is no need for students to be issued sets of hardware and software;rather they must have access to these tools. The project team estimates that thenumber of systems available should be about 30 percent of the maximum numberof students resident at any one time. The cost for each setup should be about $3,500,and maintenance costs are estimated at 10 percent per year.

" Networking. The preceding estimates do not include networking costs, which canonly be estimated after the system specifications are completed.

* CARL The cost of converting CARL information (for example, card catalogs andabstracts) into computer-usable form and the cost of associated equipment cannotbe estimated until system specifications are defined.

" SOCS. Estimates for the cost of providing the faculty and staff computers are givenabove. The costs associated with networking SOCS with the Total Army cannot bemade until system specifications for the SOCS system are defined.

Action: The implementation of the administrative use of computers throughout the Collegeshould be started immediately! This implementation will be a prerequisite to the design andimplementation of the other ACL areas.

12

Testing

The testing element of ACL is the use of computers to measure retention of previouslylearned KSA and usually involves objective tests, such as multiple choice, true/false, andmatching. Testing is seen in the literature as part of the larger computer-managed instruction(CMI) process (Kearsley, 1984). The project team sees the following advantages for com-puterizing testing at the College. First, an increase in efficiency in grading is possible. Com-puterized testing of students provides for immediate results for individuals or groups. Theseresults can be readily analyzed on-line by the monitor.

A second and related advantage is increased efficiency in understanding the results of test-ing through the automatic collation and assessment of test results for statistical purposes. Thetest analysis programs can be standardized to examine test results for common errors and touse that information to evaluate the test or the learning strategy applied to that subject mat-ter. Better use of instructors' time would result.

The third advantage is the increase in learning efficiency that the use of situational testingcould bring to the classroom if situational testing were used. Situational testing is the evalua-tion of a person's skills when presented a case to solve in other than written form and is stillan objective test administered on the computer in such a way that the student provides answersto specific situational questions in an interactive manner. Tests could be designed using avideodisc, for example, that would create a problem that the student must resolve. Scoring andfeedback in real time would result. These programs can provide a learning prescription thatwould assist the student in the learning process.

Applications of testing in the College include the following. A few of the CGSOC core cur-riculum subcourses have formal testing identified in their POI, accounting for 2.4 percent ofthe hours of the core curriculum. Testing could easily be implemented in these subcourses.Other CGSOC core curriculum subcourses could incorporate situational testing to improvethe learning of students and provide immediate feedback to students on how they are progress-ing. The P118 and P118L subcourses offer opportunities for situational testing, and the qualityof learning of the more objective material would probably improve. Entrance examinationsfor CGSOC and CAS3 students are examples of opportunities to efficiently and rapidly testlarge numbers of students.

Testing

Potential: The greatest potential uses at the College are for improved testing procedures inthe SOCS and for entrance validation of skills in CAS3 and CGSOC. For example, a personal-ized course of instruction could be prescribed following the testing program. Such a learningprescription could target the specific areas in which a student needs further study. The neces-sary number of workstations to support testing will not be available in the CGSOC until theGeneral Instruction Building (GIB) is built and renovation of Bell Hall is completed. As faras situational testing is concerned, the long term may provide opportunities for integrating SITwith this testing to greatly improve the effectiveness of instruction. Another potential use of

13

the testing ACL is for collection of survey data among nonresident officers. This testing can

be done using the same tools developed for computerized testing.

Risk. Risk is divided into the following topics:

* Entrance testing. There is little risk here. Entrance testing can be done using CMIpackages, which have been proved highly effective (Baker, 1981). Providing suchtesting, however, must wait until the College has the computer workstations andfacilities to effect testing for large numbers of students. Selection of off-the-shelfprograms for testing is low risk.

* SOCS. There is low risk associated with computer testing of nonresident studentsbecause the same software used for resident instruction can be used here. There isno operational network in SOCS or the College to support nonresident testing.Thus, implementation must be deferred until that capability exists.

* Situational testing. Moderate risk in the design and implementation of situationaltesting is probable. Only further study of the specific subcourse subject material andits app!ication in the instructional strategy of the subcourse can determine the mostpromising situational testing applications.

Costs: For conventional computer testing, such as entrance testing for CAS3 and CGSOC,minimal costs are anticipated. These applications can be contracted, or the capability todevelop them in-house could be pursued. The most significant costs incurred are to implementthe administrative use of computers throughout the College and to purchase the networkingcapability for SOCS.

Action: Very little action should be taken before the SOCS networks and student worksta-tions are in place at the College. During implementation of the high-tech classrooms and net-works, computer testing can be developed.

Computer-Assisted Instruction (CAI)

CAI is the use of computer courseware to teach recently introduced KSA needed for basictask elements, where courseware is defined to be a special subset of software through whichinstruction can be accomplished on a standalone basis. When examined in relation to Bloom'sTaxonomy, CAT most often aligns with the lower order cognitive levels. However, some high-ly sophisticated applications have been developed. The Combat Unit Leader Trainer (CULT)developed by Los Alamos for the Armor School at Fort Knox is an example of a sophisticatedcomputer tutor that focuses on the higher order cognitive skills. CULT was developed toprovide realistic tactics problem solving, including feedback to aid the student in arriving atthe "school solution." Among the remaining elements of ACL, CAT is the most mature withnumerous commercial applications available for use in the academic domain. The literaturecontains numerous references comparing the effectiveness of CAI with conventional instruc-tion (Kulik, Kulik, and Cohen, 1980).

14

Cost of Development

In discussing CAI, it is useful to discuss costs in detail. This discussion is presented first be-cause much of the technology to develop CAI courseware already exists in the commercial andacademic realms, and cost is a primary consideration in implementing CAI. Costs can bereasonably well quantified, and discussions of costs provide a basis for evaluating potentialusefulness. The typical basic unit for evaluating costs in CAI is one hour of delivered studentinstruction on the computer.

Figure 2 illustrates the wide variation in the number of hours required to develop one hourof CAI. In Fig. 2, pedagogy refers to the instructional method or format that is to be used tocreate the CAI application, and author experience refers to the level of familiarity that the les-son developer has with the authoring system being used.

Author Experience*

PedagogyLow High

Existing 8-63 6-39

New 165-610 27-180

Fig. 2. Effects of experience and pedagogy on CAI (Avner, 1979).*Production rate (hours of development/hours of courseware) is determined

by author experience and the nature of the CAI.

The costs of labor for the personnel that develop the CAI software varies from about $20per hour up to $90 per hour. The capital costs for specialized equipment needed for CAIsoftware development are not trivial. For example, the use of recent advances in educationaltechnology, such as CD ROM, videodisc, and digital audio are instructively powerful but ex-pensive.

For example, the cost of production of a single videodisc may add from $30,000 to $100,000one time cost to the development effort. Of course, when this equipment is used many times,the cost per videodisc will be much lower. The video technology, however, provides for a muchmore realistic training environment than has ever before been feasible. Once developed, thevideodisc cannot be readily changed, but the accompanying courseware often does change.Considering the possible impact of doctrine changes on course content is an important con-sideration when contemplating development of CAI software. Int.ractive videodisc is an op-tion for CAI that is cost effective for educational applications requiring visual realism.

15

To indicate what CAI costs might be for CAI software development, consider thefollowing example. Suppose that CAI software for I hour of instruction in a 16-person class was being considered. Our example will be for a moderate levelapplication that assumes 200 hours to develop, and the average cost of labor is $50 perhour. Assume that the CAI software is developed for videodisc and that this hour ofCAI software must support 20 percent of a $30,000 videodisc capital investment. Thisdevelopment would cost $16,000.

Now, assume that this CAI software was to be used for the 1,000 students in CGSOC.The cost per student taught would then be $16. For comparison purposes, considerthe costs incurred if the same material were to be presented by an instructor to the16-person group. Assume five hours of preparation time for the class and $53 perhour (based on an officer's annual cost of $100,000). One then calculates a cost of$318 for the instructor's time. If one divides this sum by 16 students, she/he gets acost per student of between $19 and $20.

Suppose that the subcourse does not change content during the following year.The live instructor cost would be constant for the following year at the same $19-$20per student. In the case of the computer CAI software, the cost per student wouldhalve to $8. The quality of the student-computer interaction would be controlled andconstant from year to year but that of the instructor would not be.

As indicated in the sample calculations above, important considerations must betaken into account when deciding whether CAI should be implemented. Discussionsof these considerations follow:

* CAI appears to be more successful in teaching skills at the lower levels ofBloom's Taxonomy. The cost involved in developing CAI courseware forteaching lower cognitive level KSA is usually much lower than that forteaching higher cognitive level KSA. Cost is lower because of the simplerinstructional strategy used for the lower cognitive levels. (See thediscussion of instructional strategies below). The teaching of facts andprocedures and remedial instruction for incoming students are examples ofwhere CAI could be developed profitably.

* The development costs of CAI depend upon the authoring environment,experience of the designer and programmer, the instructional strategychosen, and the structure of the subject area. For example, developmentcosts for drill and practice exercise developed with an authoring system areconsiderably less than those for a game developed with a programminglanguage. Opportunities for CAI development for subcourses where thecontent is stable should be considered before developing CAI for morevolatile subject areas.

* CAI is especially attractive for subcourses that service large numbers ofstudents. The per student cost varies inversely with the number of studentsbeing taught; and yet, the degree of individual learning remains high,making large student enrollment subcourses more lucrative than subcourseswith small enrollments.

16

The hardware configuration that is used to develop the CAI application is oftenmore complex and more expensive than the classroom delivery system. The CAIdelivery system is, in turn, often more complex and requires more capability thanthat needed for administration applications. For example, for CAI courseware,videodisc, CD ROM, digital audio, and graphic overlays on video are often con-sidered. Hardware to develop the CAI application also requires much greatermemory capacity than the classroom delivery system. Both development anddelivery hardware configurations must be considered when planning for CAI. Weestimate that the cost of a top-of-the-line CAI delivery system is about $12K.However, CAI delivery can be and often is done for much less on an ordinary PC.

Often, synergistic relationships develop among the aforementioned factors. For ex-ample, if subcourse content does not change from year to year, obviously, more stu-dents can be taught for a longer period of time with a single CA development.Therefore, per student costs would be driven down because of more than one fac-tor.

Authoring Systems

CA programming can now be accomplished with software tools called authoring systems.These tools prompt the author, who is a designer, teacher, or subject matter expert, regardingwhat he/she wants displayed on the screen--sequence, answer judging, and branching. Morethan 200 such systems are now on the market from which to select. The design of the CAI les-son is still the most complex part of CAI development, but such tools make the programmingfeasible for institutions without a staff of programmers.

Instructional Strategies

CAI is only as successful as its design. Central to the design of a CA software package isits instructional strategy, the pedagogical approach used in helping the student achieve thelearning objectives. The literature contains several taxonomies for instructional strategies. Theproject team recognizes distinct instructional strategies (Alessi and Trollip, 1983):

TestsTutorialDrill and PracticeGamesSimulations

Although the names of several of these strategies are identical to ACL, the taxonomies arecomplementary. Specifically, any of these strategies can be used within the CAI and ITS ACL.

The particular instructional strategy used depends upon the following factors:

o The expected behavioral change as a result of instruction. For example, if the verbis "to recognize," a drill and practice exercise with multiple choice questions would

17

be acceptable. However, if the verb is "to demonstrate," a simulation strategy wouldbe necessary; and if the verb is "to show skill at using," a gaming strategy might beappropriate. One would not use a tutorial strategy where new information ispresented and then tested if a person needed to discover a student's ability to syn-thesize and apply a great deal of information.

" Other types of instruction currently available in the subject area influence thestrategy chosen. For example, classroom instruction of facts may exist, but theremay be no realistic opportunities for application of those facts. CAI could providegame or simulation strategies to complement classroom instruction.

" Cost and time constraints affect the selection of an instructional strategy. If aproduct must be produced quickly and on a very small budget, the drill and prac-tice or testing strategies should be seriously considered. With the modern author-ing systems, tutorial CAI can also be produced quickly. The expense for originalgames and simulations is higher because of the extensive design and programmingwork required.

Therefore, the objective in taking an instructional strategies approach to CAI developmentis to maximize learning for the desired target skills. So often in education, we have taught stu-dents to recall and recognize and subsequently tested their ability to evaluate or synthesize.Then, we cannot understand why scores are so low! The instructional strategies approach addscomplexity to the design phase but has a high payoff for student learning.

Conversion of instructional materials from the classroom directly into the computer is rare-ly successful. If such a conversion were possible, CAI development costs would be consistent-ly lower because of differences in on-line versus classroom instructional strategies. In liveclassroom instruction, the instructor is able to adapt in a way computer technologies current-ly cannot. The potential exists for such adaptation, however, through intelligent tutoring sys-tems research.

CAI, MAPP, and COTES

Many of the subcourses of the College share a need for common computer tools and ap-plications. MAPP (Military Application Program Package), which was developed for CAS,and COTES (Combat Orders Training and Evaluation System), used in CGSOC, are examples.Such tools are incidental to CAI but, when considered wisely in developing specific CAI cour-seware, they will contribute to overall course effectiveness. This consideration reinforces thevalue of such common tools, produces far more effective learning, and provides the officercorps with tools that can be exported to the field. This situation would improve the outside-inview of the College and improve the computer literacy of those officers who would be exposedto the tools before attending the College.

18

The Power of CAI in the Classroom

The CAI design must take into account administrative changes that result from introduc-ing computers into the College. When networking of the students and faculty in a classroomis implemented, the capability would probably exist for an instructor to show any student'scomputer display on a large screen for all to see. CAI development should consider thiscapability and others and capitalize on them. For example, if a CAI application were generatedthat involved student development of a project, that result could easily be displayed for stu-dent discussion and critique. The effectiveness of staff group instruction may, in turn, be in-creased.

CAI Hardware Issues

A number of additional hardware-related issues must be discussed in the CAI context. Anumber of Army activities will influence the availability of development hardware. TheClassroom 2 evolution and related introduction of various Army systems, such as ManeuverControl System (MCS) and ArmyTactical Command and Control System (ATCCS), will openup possibilities for CAI applications (as well as other ACL) in the College. The developmentand introduction of EIDS (Electronic Information Delivery System) in the College may havea positive effect on the development of CAI. Approximately 125 EIDS systems are scheduledfor delivery to the College, and using EIDS hardware to effect near-term CAI should be con-sidered.

Finally, consideration of CAI for SOCS deserves special attention. At present, as has beennoted elsewhere, the SOCS material for CGSOC is largely out of date. This situation can beeliminated for future CAI development by planning for exporting CAI through SOCS for allCAI developed. This planning may place an extra burden on CAI development but is impor-tant for improving the quality of the nonresident material. The hardware to export CAI is notavailable, but this capability must be planned as an integral part of the electronic universityconcept of SOCS.

CAI Development In House

The College must establish the capability to develop its own CAI course material. The Col-lege does not have the necessary skills to develop CAI in house now. If a basic authoring sys-tem were selected and personnel assigned to the task of design, a prototype could be producedwithin a year. The College faculty and staff must be involved in the CAI development process.Only the College knows its subject matter and is aware of the changes that occur in thatmaterial. Only the College would probably be responsive to the needs of its academic depart-ments and Schools. It is also important that at least some of the technical (not subject matter)specialists that develop the CAI material be on site for an extended period of time to ensurea smooth transition when staff turnover occurs.

19

Computer-Assisted Instruction

Potential: For those subject areas that are at the lower cognitive levels, are not volatile in con-tent, and will be taught to large numbers of students, there is great potential for CAI to aug-ment faculty in the classroom. In particular, the COMPS (Combat Skills ComprehensionProgram) and other remedial subject material appear ideal for CAI. The integration of CAIinto the SOCS electronic university also has significant potential for positive influence on theTotal Army.

Risk Low risk is seen for using CAI to teach the subcourses that deal with lower cognitiveKSA. This technology is relatively well proven. For teaching KSA at the higher cognitive level,risk increases slightly because there are fewer proven models to follow.

Costs: Design is the most costly part of CAI. Some indication of costs and associated payoffscan be seen from the example above. The costs per hour for CAI development, however, varywidely among applications and are directly related to the instructional strategy selected.

Action: Use an authoring system to develop an exemplary prototype hour of CAI software ona delivery system that demonstrates CAI capability. The subject of the CGSOC elective A035,Faculty Development, would be appropriate for this CAI section because it would be relative-ly nonvolatile, has large numbers of students each year, and would introduce prospective facul-ty to the techniques of CAI and skills that they will need as faculty in the College. Thisdevelopment should be under the purview of the Department of Automated Command andTraining Systems (DACTS). Furthermore, this project will permit the selection of media forfuture CAI applications in time to influence the GIB and Bell Hall remodeling needs.

Continued development of MAPP and COTES or some equivalent for use as aids in theclassroom should be pursued. As these applications become more sophisticated, ways to in-tegrate them into CAI development courseware should be investigated.

Tables I through V display a detailed listing of the project team's views of potential ACL.These applications should be carefully evaluated by the College faculty for appropriate CAIcandidates. A plan for CAI development should be established under the auspices of DACTS.

Simulations for Individual Training (SIT)

What Are SIT?

Simulations for Individual Training are the uses of computers for the analysis and applica-tion of previously learned material. These instructional simulations model some aspect ofreality with sufficient fidelity to present the student with situations requiring them to applypreviously acquired KSA. They can potentially provide the "artificial experience" the Collegeseeks for its students. Development of these simulations is more challenging than that requiredfor any ACL discussed so far. SIT include part-task trainers that, in the CGSOC context, areused to develop an individual staff member's KSA. SIT are intended to be used following thestudent's acquiring of the facts and procedures in a given knowledge domain. The simulation

20

seeks to require the student to integrate these previously learned KSA into a functional

capability.

The Design of SIT

It is important that the SIT that is developed will emulate the real world, including thecomponent of interactions with subordinates and superiors. The design and development ofSIT must take into account the characteristics of real world communications and informationgeneration systems, such as MCS. Verisimilitude to what the student will see in the field is im-portant. SIT have three major components: 1) the scenarios, 2) the user interface, and 3) theshell.

The shell consists of the rules governing the delivery of the scenario, the sequence of stu-dent actiofi, and the feedback delivered by the interface. This three pronged structure for SITallows several advantages. Maintenance costs are lower because one can readily update con-tent information without having to touch shell code. The user interface can be modified readi-ly, which is important because interface requirements vary with the target population. Finally,the shell can be used with a variety of different scenarios. These scenarios could all be fromone subcourse or be from different subcourses or even different schools of the College. Theuse of a shell structure increases the usefulness of SIT development and decreases its cost perapplication.

The Los Alamos project team knows of no SIT developed that meets the needs of CGSC.At Los Alamos, however, a SIT was developed for Fort Knox student armor officer training.This project, called CULT, is funded by ARI and uses an instructional model called "computer-tutor." (See Andrews and Trainor, 1987.) This computer-tutor assumed student knowledge oftactics and challenged the platoon leader to specify fires, formations, and techniques of move-ment for various offensive and defensive scenarios. This SIT has undergone preliminary test-ing and a multitude of reviews and has received favorable feedback. Its success is not onlyattributable to the realism afforded to scenarios through videodisc and digital audio but alsoto tutorial feedback. When a student's performance is not optimal, the system provides severallevels of feedback to aid him in reasoning to the correct answer. This model for SIT could beadapted for use in subject matters of the College. Research is needed to define the specifica-tions of SIT for the College and to move toward their development. As can be seen in TableI, each School has a need for SIT.

SIT and BCTP

The issue of Battle Command Training Program (BCTP) must be discussed regarding SIT.When fully linked to CGSC, BCTP may be useful as a learning vehicle for at least some of theneeds that SIT should satisfy. The suitability of BCTP for providing the representation of con-sequences of a staff officer's actions, the goal of SIT, is yet to be determined. Other uncertain-ties related to BCTP include the design of the human-computer interface.

Regarding the issue of demand, it seems doubtful that a significant number of students "the CGSOC could simultaneously use the BCTP system without seriously overloading the

21

computing capabilities of the computers planned in the BCTP system. This issue is a seriousone because to accomplish the goals of SIT, many individuals must be using the unique simula-tions that are responding to each student's needs. The BC-TP environment supports manyplayers on a commonly used scenario.

Simulations for Individual Training

Potential: SIT has high potential for the teaching of higher cognitive level skills involving theapplication and analysis of facts and procedures already learned. Unfortunately, no SIT existthat can meet the needs of the College. Los Alamos does, however, have a model that the Col-lege could follow in building SIT. Specifications for SIT applications need to be defined as afirst step. When SIT are fully developed, they will be useful in supporting several teachingmethods but will be particularly adaptable to supporting staff group instruction.

Risk: High risk is seen in starting on an immediate program for the development of theCollege's ideal SIT. As indicated above, the issues and requirements for SIT in the Collegemust be better defined. The use of prototypes to test out various areas of SITwould amelioratethis high risk. In any event, SITwill take some years to be fully realized in the way that the Col-lege would like.

Costs: No estimates can be given at this time because the costs involved in realizing a fullyfunctional SIT are dependent upon the particular application, the model used, and the avail-able resources. Development costs will be greater than for CAI.

Action: The College and its Schools must define their classroom requirements for the use ofSIT. This study forms the base line for defining SIT and other applications for the College.The College should expand this base line into a plan for development and design of SITthroughout the College.

Intelligent Tutoring Systems (ITS)

What Are ITS?

Intelligent tutoring systems (Sleeman and Hendl ey, 1972) are computer-based systems thatemulate the learning environment between a student and a human tutor (Roberts and Park,1983; O'Shea et al., 1984). They employ artificial intelligence (AI) techniques to representdomain knowledge, employ a specified instructional strategy, use inference techniques to as-sess the student's grasp of the material, and build a model of student behavior to individual-ize feedback. The greatest potential of ITS is in their ability to detect student misconceptions(Brown and Burton, 1978; Goldstein, 1982; Burton, 1982) and attempt to correct them. Theycan operate on noisy or incomplete data (Sleeman and Hendley, 1972) and "decide" when tointervene and what advice to give.

22

ITS Components

ITS have four major components: 1) an expertise module, 2) a student knowledge model,3) a tutorial advisor, and 4) a communication system. The expertise module contains thedomain knowledge that the student is learning. The student knowledge model is a database ofstudent performance and a categorization of student errors. The student's knowledge may bemodeled as a subset of an expert's knowledge or alternatively as a set of rules consistently usedby the student, whether or not they lead to the correct results (Brown and Burton, 1978).

The tutorial advisor is a model of the knowledge for selecting and presenting material tothe student. Its actions are dependent upon feedback received from the student knowledgemodel. The tutorial advisor contains knowledge about teaching, about selecting material ap-propriate to the student's perceived learning level, and about when to intervene and what totell the student in a tutorial session. The tutorial advisor implements the instructional strategyimposed by the ITS designer.

The instructional strategy is the teaching approach adopted by an instructor, typicallycoaching, games, simulation, Socratic dialog, a diagnostic approach, or some combination ofthese methods. The communication module is the set of functions that enables the expertisemodule, student knowledge module, and tutorial advisor to interact with the student. It is thepart of the system that drives the actions of other modules in response to student input. Thecommunications system encompasses the human-computer interface of the ITS.

The State of the ITS Field

The first ITS was produced in 1970 (Carbonnel, 1970), and yet the field is still in its infan-cy. The greatest research challenges facing the field include 1) development of a robust,flexible, natural language parser that does not exceed machine memory capacity, 2) perfectionof knowledge engineering technology to facilitate creation of the expertise module, and 3) ac-quisition of more knowledge regarding optimal tutorial approaches for the construction of thetutorial advice module. In spite of these unsolved research areas, a great deal of attention isbeing focused on ITS because of their great potential (Kearsley, 1987; Wenger, 1987). Workis in progress in many institutions to produce prototype ITS, which are not optimal but canfunction prior to the solution of the unsolved research areas. These prototype ITS will not onlyaid in selecting those research areas but will also serve as models for other institutions that areinitiating ITS development efforts.

ITS Examples

The opportunity to use an ITS is rare because of their nonproduction status. It is thereforeuseful to describe briefly the functioning of three ITS systems: PROUST, SOPHIE, andGUIDON.

PROUST is a misconception identifier, which was developed at Yale University (Johnsonand Soloway, 1984). It provides students who are learning the Pascal programming languagewith automatic debugging. The student writes a Pascal program and attempts to compile it.

23

PROUST is then called up to examine the program and provide feedback on errors to the stu-dent. PROUST not only finds all the bugs but also describes why a bug occurred and suggestshow it can be corrected. No dialogue occurs per se, yet the student emerges with personalizedtutoring advice to use in correcting specific misconceptions.

SOPHIE is an electronic troubleshooting tutor that provides a reactive learning environ-ment. The problem solving centers around a laboratory model of a circuit. The student canpropose and test hypotheses relative to the circuit, have the hypotheses configured, and receiveadvice.

GUIDON is a tutor based upon cases (Clancey, 1983). The student is given a medical caseto diagnose, asking questions to gather important data and propose hypotheses. The tutorialmodule intervenes to provide help or advice when the student's actions are not correct.

ITS and the College

As this technology evolves and becomes better understood and more functional, there arenumerous possible applications for the College. The Los Alamos project team feels that themost interesting application is in the teaching of command and control. Defining theknowledge domain in the enormously complex world of command and control will be a seriouschallenge to the College. Additionally, when implemented at the College, ITS may provide animpetus for more robust training of command and control throughout the Army. Because ofthe enormity of developing ITS, these promises must wait for the future.

Intelligent Tutoring Systems

Potential: Very high potential for ITS exists, particularly in complex areas such as commandand control. The technology is in its infancy and exists only in laboratory programs. This poten-tial will be realized only in the long term.

Risk. Very high. The greatest challenge here is defining the selected subject area throughknowledge engineering techniques, which is essential prior to implementation of any AI sys-tem.

Costs: Based on what is known about expert systems and knowledge engineering techniques,the costs are very high. Hopefully, as the technology matures and becomes common in thescience and education communities, costs may decrease significantly.

Action: Do not pursue ITS as a learning technology at this time.

Gaming

What is a Learning Game?

In gaming, concepts are applied and practiced through a computer game. Because gamesare not usually concerned with the detailed results in the situation portrayed, they model the

24

relevant subject at lower fidelity than that of a simulation (Crawford, 1986). However, the en-tire spectrum of pertinent detail is presented to the student, allowing the user to understandand manipulate the big picture without worrying about the precision in the results. The exer-cise, manipulation, and practice of concepts are the goals of gaming. Gaming requires the stu-dent to exercise the relevant concepts against a real or virtual opponent.

Games for the CGSC

Games can be used by an individual or by a small group of students. This aspect makesgaming attractive for SAMS because the essence of SAMS is to develop a personal philosophyof the tactical and operational art in an elite group of officers. Gaming could be used by in-dividual students to develop and test their detailed understanding of the art of war, arriving atsupportable conclusions.

One possible subject area for a game is the operational and tactical level battlefielddoctrine with which SAMS is concerned. The game could provide variable resolution of thebattlefield from an individual fire unit through battalion and division to corps and joint andcombined forces. Individual officers could use it for developing their personal philosophies ofthe art and conduct of war. The same game could be used in SAMS-wide exercises that wouldprovide the "artificial experience" needed for the students to test their views in a dynamic situa-tion with their peers.

Because the fidelity of gaming is not usually high, gaming can often be run on computersthat are modest in capacity compared to those needed to run simulations for collective train-ing. Conceptually, games can be run on a personal computer, and they require small amountsof input from the user. As an example, consider a conceptual G3 game emphasizing tactics.The other staff functions as well as the opposing forces would be simulated by the personalcomputer. If this game were run as a two sided game, then the opposing forces could easily beplayed by another student.

Gaming has great potential for development of higher level cognitive skills in the tacticsand operational art subcourses in the students of CGSOC and SAMS. It could easily be ex-ported through SOCS and used to close the quality gap between resident and nonresident in-struction. Such a game would directly support the staff group instruction teaching strategy ofthe College, allowing the faculty to concentrate on the content and quality of instruction in;esident courses.

SIT Versus Gaming

Gaming and SIT can complement each other. Gaming deals with the broader issues of thesituation, and SIT deal with the detailed account of what happened. Both levels of detail areappropriate. For example, consider the use of both a G3 SIT and a G3 game in the CGSOCcourse. The G3 SIT would deal with the details of procedures and special actions required ofa G3, while the G3 game would deal with the broader concepts of the tactical operation andthe relation of the G3 to the rest of the staff in the tactical environment being played.

25

Gaming in Teaching Control

The opportunity of gaming to contribute to the teaching of the control portion of commandand control is probably significant. As discussed in the Task B report, FM 101-5 does not clear-ly represent the KSA required of the staff officer in the area of control. Yet, these KSA arecritical for officers to be effective in any tactical situation. Conventional instructional methodsare awkward when teaching control because in any realistic control environment, the situationis rapidly changing.

Because gaming is not overly concerned with detailed representation of all aspects of thebattlefield, it has the potential to allow practicing the control portion of command and con-trol in the classroom environment. SIT and SCT are too focused on the details of the tacticalsituation to contribute to quality instruction of control. The project team believes that the in-struction of control shows the greatest potential for the use of gaming at CGSC.

The Current State of Gaming

The most glaring negative aspect of gaming that would be relevant to the College is thatno suitable game has been developed. However, games can be built up over time to display ahigher level of performance so that they can grow to meet the needs of the user. Games canbe developed in a relatively short period of time and at a cost relatively low compared tosimulations, so the financial risks are much lower than with large simulations.

To illustrate this point, consider the Balance of Power (Crawford, 1986) game referencedearlier. Balance of Power is a commercially available game that runs on a PC or equivalent.The player can choose to play either the United States or the Soviet Union side. The goal ofthe game is to accumulate a world prestige score that is higher than that of the opponentwithout causing a world ending nuclear holocaust.The opponent can be either another playeror the software of the game. The players take turns determining the state of the world and es-tablishing changes in the world by actions, such as supporting revolutions, sending military aid,and lodging diplomatic protests. Each turn evokes a response in the other player that is notobviously predictable. The presence of rules having results that are not easily forecast challen-ges the player to become involved in the game and to stretch his/her mind, which is the essen-tial ingredient of a successful game. Chris Crawford took about one year to develop the game.Considering the remarkable ability of this game to develop a view of world politics in theplayer, a relatively small development cost was paid. This example shows that higi-qualitygames can be developed on submillion dollar budgets.

Gaming

Potential: Very high potential exists for gaming to teach control in situations that arereasonably realistic. Gaming can be used to evaluate the consequences of one's actions in tac-tical situations.

26

Risk: Moderate risk exists because tactical games of the type needed have not been developed.However, this risk is ameliorated by the fact that complex games have been developed forrecreational uses.

Costs: Relatively low costs are expected for researching the development of the games neces-sary to significantly improve the learning of CGSC students. If these research approaches provesound, further development costs should be moderate.

Action: Research into the development of games should be pursued. If results of this researchwarrant, pursue vigorous development of games that would be useful for teaching control andfor meeting the needs of SAMS in developing its students' views of the art of war.

Simulations for Collective Training (SCT)

Collective training, in the Army sense, is the training of groups of people together as a teamthat will function jointly in accomplishing its mission. SCT involve the use of computers todevelop the team KSA needed to accomplish a mission through the use of a simulation. Thesimulation is used to present evolving situations that demand that team skills be applied.Fidelity to the real world is as high as can be achieved in the simulation environment, makingSCT complex and influencing their cost.

Historically, command post exercises (CPX) and field training exercises are the vehiclesfor training units in their war fighting missions. These exercises have enormous overhead ex-pense associated with their controllers. As a computerized CPX develops, the overheadremains. For example, this situation can be seen in BCrP where more people are required tomake the simulation run than can be trained at one time. This overhead expense is typical ofSCT.

The College is primarily concerned with individual training. BCTP and other SCT can beused to teach individual KSA needed by staff officers and have great potential. At intervals inthe curricula, there are course wide exercises that are used to pull the concepts learned in thecourse together so that students can practice the individual KSA. These exercises provide thestudents experience in the total human staff context. A single tool can, therefore, be used forindividual and collective training. BCTP may or may not evolve into low-overhead SCT thatmeet these needs.

Simulations for Collective Training

Potential: High potential because SCT can be used as reduced overhead tools to train stu-dents in KSA that relate to functioning as staff officers during combat. If the mission of CGSCbroadens to include collective training, SCT will be required.

Risk: High risk because no SCT to date have been developed that will provide sufficientlyhigh fidelity to the situation being modeled at overhead levels that are comfortable. However,once SCT of sufficient detail and low overhead are developed, modifying them for other ap-plications should be low risk.

27

Costs: High cost is associated with SCT development and in the operation of SCT. The fidelityis high, so large, complex programs demanding high-capacity computers are required to runthe simulations.

Action: Because of the high costs for development of SCT, no recommended developmentprogram is suggested. The BCTP program should be closely monitored. Then, if it evolves intoa low-overhead SCT, integrate it fully into the College. Consideration should be given in thisregard to using BCTP in the facilities being developed at Fort Leavenworth.

High-Tech Classroom

The preceding discussion has centered on the individual ACL. The ACL are onlyworthwhile if there are organizational and physical structures to facilitate their use. The high-tech classroom is one physical structure that can help to achieve this end. The project teamenvisions a prototype classroom, which contains the latest developments in instructional tech-nology. This classroom includes the use of computers and other technical aids to instructionand serves as a laboratory for experimentation with these technologies. The faculty and staff,along with the students, can explore some of the limits of these technologies. Faculty could tryout various educational technologies and methodologies and become familiar with them.

The classroom might have a computer workstation for each two students and a controllingworkstation for the instructor. The student workstation would connect to two separate but in-terconnected computers, allowing for data access and manipulation, emulation of MCS,videographics capability, and networking to the appropriate CGSC and Army activities.

The student-computer interface would be user friendly in the sense that few instructionswould be necessary; it would be easy to use and easy to learn regardless of previous computerexperience. This interface would be accomplished through use of menus, icons, natural lan-guage, graphics, voice, and other techniques. The instructor would have the ability to displayexamples or sample problems for the student to dynamically solve and then discuss. Availableto the students and instructor would be the appropriate administrative, test, CA!, SIT, ITS,gaming, and SC7' systems. As appropriate, these systems could be used and improved upon bythe students and instructors. Additional applications would be developed and documented forfuture College implementation. Army activities and programs, such as ATCCS, BCTP, EIDS,and others, would be tested in the high-tech classroom first and expanded or rejected as neces-sary for meeting the needs of the College and the Army.

Details of a high-tech classroom design remain to be established. However, the projectteam sees this classroom as central to the implementation of many of the applications thatwould be incorporated into the College; and eventually, all classrooms in the College wouldcontain some instructional technologies. The high-tech classroom would be the vehicle forbringing CGSC into the age of computer education.

28

Data

The ACL taxonomy that was developed by the project team is applied to the curricula ofCGSC in Tables I through V. A table is presented for the CAS3 Phase II resident course, forthe core curriculum for CGSOC, for the electives for CGSOC, for the SAMS course, and forpercentage totals of ACL usage for courses of CGSC. Each curriculum is broken down intoits subcourses as organized by the latest POI that the project team had. The following infor-mation is recorded in the columns of the tables.

Coltimn Heading Information in the Column

HOURS # The number of hours in the subcourse.

NA The percentage of the subcourse to which no application of com-puters was assigned.

ADM A "Y" indicates that administrative use of computers in the sub-course is appropriate. All subcourses have a "Y" entry.

TEST The percentage of the subcourse to which computers could beused in the testing function.

CAI The percentage of the subcourse to which computer-assisted in-struction techniques could be applied.

SIT The percentage of the subcourse to which simulations for in-dividual training could be applied.

ITS The percentage of the subcourse to which intelligent tutoring sys-tems could be applied.

GAM The percentage of the subcourse to which gaming could be ap-plied.

SCT The percentage of the subcourse to which simulations for collec-tive training could be applied.

COMMENTS The number of the subcourse and the subcourse name are usual-ly repeated here. In addition, the specific applications that theproject team judged could be used are stated briefly with othercomments as appropriate.

At the bottom of each of the tables, a summation line and a percent line are included. Thesummations line shows the number of hours that we judge the ACL to be in effect for the en-tire course. The percent line shows the percentage of the course to which the ACL could beapplied.

29

The final table shows two summations. One of these summations includes total hours andpercentages for each of the ACL elements for all four courses examined here, and the othersummation shows the totals without the CGSOC electives included.

30

W IA; Li UJ nUr 3c -M "i- 2

- L Lia u -

Li<ca - c--- Iii -4 4c dm a U)a

~~~~~L -O cm ~ 4 -- U O .U.- L. at :x cj c=

mL- a- W L" 2c -4 )c~

~~Li4 "c -U)c M ~LZa x - 42-~a . ~ ~ ~= 4 U =Loa.~ m co CU _j - - "A

La ~ U -m L-- =- cc Z-m- 0. U)-~-

L. W~ CD cm me -- m LA Li n Z~U. )M U) Ia CL. -a a ) )LLnUJar 3 _j CL . -- 0.~U . . a

U)~L cK ! - . K- c

a -~- - - -a cm m. == = U)06 - - "I40LL" cr. c= -a --

-3 - L ~ a W- U)= tn LJ- - - - - - - -LgCD Li x -j -z N.- cla <c ; ;. z ai - Li U) U)U

0.~ ac Ln U) CM=~ )= -~ = a =.iaa

Ln C3- a. cc)U Lc4 L" -t w) w0. -- i U U= . a L Lo4U L-- U) ~- - -~ ) 4 -4 ac.. '= =' L" =)- _0-4--~~~ Ln cc 0U-*~. ) ~ -_ )-" c3W L Cc LU.U 1= J ) U - . w U) U m U) U

-CL. -4 za_ ;- U) U U -= 4 M cm

3w 0. =- L" cm -= 'A U= - c 4 - U) LrJJ LiL.

2 =- = wi - an w- - abLt

- --.=i'.- .a

a

2c

a=tLn -M QL i aknw.L4 Li Jn~

c)0 - i . L i U U L 2U. L SU CO wi L fUL i L~~~~- +-H'K wu+ i c me L U 0ia.

- a Z L -aj, a- a~ aj a a) a -t L, U' n a- ZE aj Za a" a a a a az4 rUC =U n c z cU

a u a : a 1A .0 aj " ~ M a L n a a aa L&. a a - I a a a a a 'e a a .'C" C" C L .

-a~~ ~ aoZwc uCAqja

C3. 4DU CC 4D, 40. C-5 CD CU C3. c. C- 4: U) <=-

<2 m clU clcl) = m 11 I

9=1~~~~~~' 1=CU4 > hr ' >C

4=1 C. .' D C C1I

V, U)c " l I u u CICjc p- - a

cp c: cm- -. c,.p C -= c3W- C), CD C- c~l a3 cp c:. C3 CP CD al -N -C c3' 4= - . P.

0.aaaaaa C- an W') U) Q-i CDa . 4= UD a a a3 aD a a! a= cp a a aa rEn an WI ,U )C- 5a S)uP) .u ' ) W- a l an) U1I a- a na n n W-) W-) U. Xn r-0 CU.

CN=)

w c cc0- a

"j -j w ' r' o2a ~ am . t

Lo .-4 w4 3c L " wU2ch !5 t n ca U) L0:L

I ~ . U) c 2t 2c 2c -U) Ic c2 B - =! Li ISa-f" 3z L CL. 21- 2c

-~~~~~ co )- Ilz Cia . ) a~- U La cc wU L" G3 Li L" 0. dc LiUa = -j lz _j -K dn U)m U) LAO L- ccC Q .L ) U U 4 -. wi

* ~ ~ ~ ~ Ze La a lan -- l c- C- aC* )~ ) U a L wmi - )-z .i 4. )-- m W2 ___ W U)~Cmf M5 U'"=C = ch -

-4 0, a g11 1 b'a a , . w a, g I 41 _j a , ,

cnu' ) C'W'a *5-rru San U, 0 . 0 0 . 0.0-,a0.0 .0 .00 CC- -4cc W

31

= -C-s.a -lota

t2 -Ca.m t - Z - e

Sig- a, 2 x

22 z

a~~ -x Ms. ~ -

-- - - - - -- - -3- -Km-

- ~a '- ,,,m n - a ma -.. a -m -s

a~ ~~~~ EU ~~.

- a. ~a m 0 ma- . -.. ca. aa.o 32

LUU) -a -C 40 W0-LUSM ~ ~ I - ..L C.J 2-

CLUr a! L"L W U C~- U LU )

- w wj Ch_ L ~ L . U2 M C* -. A J4aftj - ' K = wJ MU 0 c at = - .j - LU LU dU C=U* U

CJ 4-! I= CC LUd~U)C - = LU z U)i M Up =C LU L L ..S

- L %- U) an - m M- MU 4-C - .~ - e Ij~ i ,wl ZPIWU - = L, I ~ L LC C L C C - C

LU me 0-; CCC C2 W-U 4z C D U - -L0- ~ ~ z~ mj w-IUU0 4.j MU LI LUw 0 )u ~ 0L LS

CLU.C- -LC0 - j CU- "cc m.A C I L L 2-wU.J LI W LC - LLM -c ~ 0 M a wUL - a:)CLU 0 U

-~~~s LiL LOL CLU- M;6 0- C, L3 -w U LLL 0 U)U U *U U C *U

=QCUI) -a w~) U)CL 4c 0 *UO -C Lr mI U - -- iU)C UL C W& toL LIC E =U( =w ,9 WL,. LUl Uj.J..ww M LU ULUS Ww 4

U)C~ ~~ ~ LU M0- atL .M * U L C-I O. UU C C U L C C

WO0 - - ) S U-L - CULU00 -C IU -CL U-L U. C Cc 0-LU U) 0 0 )

*LW rC I L LU L 0 I. L L L UU 0CM L IU at *,m 0-W C J U L

CZ m WI C.JU)U)L -L CL U00-,- M ML C0- L C-j LUUzL 3CU

MrU L )L )L C C C L - - C U ) ) U L 0 L UL = LU LU LI CC 0- W .- L0 2-L U L C2 9K - CC ~ ~ ~ w j Ujw 0-x WU CL00--U U La5 x WL LU - 0 ) £0LUU

C~~~~ C m -CL I L L UL IU - m ~~ WU C UU ) CU U) LU -= U) -t:; .LU LU C UCU . .. )~0-.J)C) - aU) ww W =W-0 == wmCL ..- U). - w CC = a)U "ICU- L

3 UUL ILLA U)0-Llw =E k- = co; CMULU0- U .C 0-CL WWM .LCC- WM-- um tz.to -L Lo In U) *ZM O :c"M n I L Wa ,- LU= U) LIr - L LI C C 0 0-21 LU- 22 -=W C1 0- 0wd- c C U= U)-=CM or 2 W -A m =-

=5t .LI lLzMLozz.j 0 SC- -U 0 UC C C 0CM= = m - cw-L IU U CL U~- U = )- U U L )L CCU .- L I U 00

C C - C C - C - L LUU U)0- C C C 0- - U U LU- C -CU U) - - J = C 0-U -- - - - - LU - LUm,U) U)U)U CLI-U)LUU C0--- L ) 0 .. U - - - - - - U LU - J LI - U) = 2=2=222-0- .JU) L

-~ LU US L CLLL I LI-,. J0- .- MW~s U 0~ L tl U C ) - CmU - C C C0 . C L Ui - LIr -1 _-CL -j-- ) C.J I 'UL U. M)=0

TI =T I RI Cs 0, ) = : - 4 8-CL ICU CC C0 -C- c0-CL C-CCCCC0- U

0-~wg a -) *.) C UC - e-

U) CC2 C3 C C C C =C C- Wv C " ) C C

* cm C3 C CC C C C ,C C U CC ) C C 2C C C C C C C U) C C C

4= C* C0 C0 C, U) W C CC) C CC U C C C C C C C* C C WS CM C C C* WI CC U U C j C)08 CU - U U)j c- U) U) U ) . .0 . Uu)

CCC l C C C= C C C C. C01C C W1 Cn C C C C C C C C= C' C C C C C C

CU)n Co C b) U)s U) U"4 C C2 C- U) C1 U) WS 1= C = C I U) C. C C 43, C U ) ) C ) U

B C"CU)C CC C C U) M en C C C C C C C" C C" C% C c" C C c" C C C CD

.0.4

-A* NIC-C -1. 4*U 2 C, U 9D'-~ ft, U ) C - -~ C" f".. J cu Cu Ck C1 Cu C C C C" C" C C" C, (U(9U"UU (U U U U U UU L U LLU C -a M C9 -I -a C Z C .- C M Am 49- -0 0 C

33

3c ID Li IDUn miU %n yD wD mW cm *L I-~4 ID Li xD m ch.. i U D I Li iIL l D.J n

- i* -I DLi I Li nI -I D .. J DI . I IL i

U L - D I * Li - I ID ID i L U D L Li- t- W -A L - :.;;g..J i ID ImII - -CL. ILn a c~ -j an =i IA - - ULID I U II

MD Li 4 . ZD wz. ; -' tD 1v W. cr. UJ %D Un Cr. ID MD I D L nI D I - I nI i Idc z Li 4DL D l~ i - W WD l.a acI I DID U L WID W L-) Wi - Dio -

.wDI i L L mD 4t -= -m ID L" I L- = M .L Li -- J =. 49 W LiC3 DLiL nLi~t -Q Li L DU -- ID I D I . D .. D I D I2L UU I iI D I

LU D i L L U IID U-a I -m ccI -c '90 cm -w lac -U L -= =ID ~ 4 uwD IDI D I DI DI I .I D L D n L- DI D I

cc WD mLi a- ID U n at I ~.JIDi C3 A; zU : .w I a I.. aD w of.A al 3c

In ~ ~ W C4z, Li at ca W- 4.J W 9i Ul !d WID LiL n I D I iI DI D L D I D- -I iL - ID LiLi IAIDI=IccULi ID IDi n coLIaLIa~ D aLi ni~~I UD L

ULI 1L *LIIDCC~ Lim~ n Uc sm =n.wDD~ toi~ an %U~ IDLAX UIIDLi~ ~ ~ ~ ~~~~~~~~K =1 2 Ue WD U DJI i-I UI DI DI I nUL iIDUI . iLiL iI iID=I iI .

U..aI~iU .I W IDI IDIL IDA iD !nDD U I.-i I i L I'~ D Ua i L

U~ID LID Li1-Ial.aU~IDiiU - I Ia ==a. ==D -- W In. .- a~ Mi Wl~~~~~a~ La U., ID ILn a Li cn IL a DI iL -U I DI -I D UU -I nU aI . I . Li

Li ~ ~ =m w Li z--DII~III . aig I I m D C. WD Mn-I WD~ =* -C~ID I XI W I

WmciUc FI ) -C CID It La t1 4- a G - -I toI MDD-to 2c IAN, -Wk !:!-LiDII Ik v-W~I ~ ~ U U D J .aD.JIc cc cc-. in. L W *D1 IDr =iD Li.'I UIDaL In' l 3 c = - = .

in= D D D L -W. WD IIm DL L DL L la I Li L ID Li~~9* m~ n= .. = DI 1 Ig DI D iI D .JI DIID I "." -iii I DOIU - D )- - -- -J==A U ! z Est"LD *~-U nIL

LiILID ~ ~ ~ ~ ~ ~ ~ ~ ~ I 'Lo C3DID I .J - U aI.,~ -a ID D D D inI-II LI ta.I JDD laI~ n D. Li'... ~a aID iI ID I J;g Iwoafl - -j aU U i w~~'I~ anDi . I i I n ID I U D

InC ; LI2 - iIhV iI 411 Wl 5 I = a...a ID, C& 51n-a .- JiI E~i 0 Z I DC k% !!!- IS W n-J-I . I Ua D- =DaI I-AL DI I DL en Cie .- m... LiI JL L D UIIc iI II"iD--JD iII~I

U4 U IDja Cie-I WD~I I ~ I I~Li~ ~~~~~~ =IID..IIO alaI~~ II D I I D I I I D D II D WI D I lI;I DI D IIII I

U ~ C 4 I LI L W @ WILzD iI 1IW L ULim I i =.%i.A! mi I D i1 aW u D -- ID-C Il-- iWWI~~4 - CD-2- cxc L a DC IUC 11.WI-W iW L .W ia.1*C C I I -I 1n cni C iW L IW IWIL I-W

... J (II n Z In I 11la 1 DIl U ~ D I i, L D ~ I D 4 I D W I Wm ID ~ flI II II 11 IW DI) . W D W IWI W D W D WI.J ~ ~ ~ ~ wt- IDI DI D I D DI DI DU- D. DI DUI UI LI LL D LL DI iI UI I DI DIDI DI DI D DI DI DU

U2 ID W D mD %C 4 Ij Ch. -23 ac a= IDG IDn a- IDJW IDm D I D I DI DI D I DI DI

U WI WI ID WI WI ID WI ID ID ID ID WI I WI ID ID IID ID cc ID i ID DI D I DI DI D IIDC we" CU CU CU ( CU U CUWI

ID t&IJ "u e' m M V m WC Ww , W

w I 19 -O WI (IA @ 0 ws 4 42 W U ~ NI4 NIa C2 lo Cm CU AM WI WI40 C C2. W

t D I I D C2 ID ID ID ID ID ID ID ID ID I ID D ID1 ID ID ID ID. ID ID ID ID ID 40. ID .ID ID 2I

21 p - .- 2. p .. 3-- 2. 2.. 2.p p . p . . .p -

L.i

4= 4U 0 CD I CD ID C . 4 C4 Cl 19 ID1 WI1 .D C C' A.* I.Ml C9 WI WI I ID 1. WI C 4en C 3 " r " C- C " m mmn e C - nm9 " C - C- "m " e

sm

as

'a

i

I34

a- _jz -a .

_j LiU A- 0-

cm, - U L 5 C Li

CA LLUL Go w U LI ~-~~q 4K -a LIt) .. Uj~

CL. 212 =2= W U -*UW) z. U) ac 5-.,.m -W) I)'- z - -~ L

L 3 "i L= Li~-=U ) )U

I-.a I C2. U) a) zU U UL I*LIU )

-3 W U. LI -- a QU LU aUL a-0 L. ) U

2t do en W) LI U) U U *S. - ) ' Li - P- U LI ULU W) LI a m) C= LU-JW W U)_ V= LU C' -CL U LU )

ac -Z LI, U) W=3 U) V4 " )L UU )-- U)U ) S ILc =)~ LU3 LI ccUU -~ WI L U I Lnw) L" LU U)J LU0 LI Um) U ) U U L U )U UU . -

Li LI U) 9. LUm m) - ULUL U m UL ) 0- - L-~2 m) w) LI cc U) m) zm -c U)) UU)r)) - )

-n CD :X -Z _. ej)U I WS .LUqc LU .L U ULI LULI LU; LU 4U5CC -U ~ U . *U U-- = ) *LL

-~ ~ U -L=_ C U) -. LIda WJU)L LIMLU ) UUJ-* ) - P ) L

en W. = Cie I. I= U); - LU _0)1 . UL I US U C3 _j))UU ==.n In)UU 11 W' M)UU) U *2.L

S.= L U) at -aL ) UU )) U) U U U ) ) )*) U mL.3U m) zU m)UU) UU LI m 1-4 aU 01... _))U CLUULU %)U n!UUz)mLU L" -C =n &M4 LIC U) .))UUU U P )* )) LUU) GLC)) - UccU ) - LU))))). U)* U ))) U)1 1-..Cc P- ) nU

U) ~ ~ ~ ' cc m U)S )L UU ) ) U.UU )U U .- U )U U ) U ) U U UC2 MR LU - U.C~ I) m)U)U.U LI) - UU U *UL UU)U) U .L U LU ZLn W) U)iN ~ u , , Z *U ) !e w e = ) * U * ) ~ ) U ) * ) . - . J ~ - ) ) . = W W %n) )

U)~~~~~~~~~~~=Z~_WGA M * 3. UUUUU- JJ). .a~)*n-))~U.UUU U) UiU) ~ ~ ~ ~ ~ ~ ~ ~ ~ - x.U )P )P )U )U )~..JU )U )U .U .UU )US U . IL )U ) ~ -UUXt!.n LUmw Wm))L W) LU J.U )J ) UUU))- LLU) s - U)G Lnm PU) ) )U U O)LIZJU.LI ) UP.5jJP w

to U )~t.U U L L L.II)I) L U )U )U P. U ) )U U)))) -U)~~~~~~~~~~~~~~~~~~~~ LU0 )=--- ).U )U ) U -U IU )L )U -- U - )UU a-- - U) U)=) - U) ) U m)U L Ln =W-U.IUL W)- II L

U) *) *~)U)U U)U) )LUUa-LI UU)U U).JL U 5- I U.- U LI LUL _ ) UU) U).. U . .. J LU LU LU U)- ) ) ) ) U)LU -~ U)U) U)U) U)U)U)U)S.U)~ . L .. J.. U 4. WM.U..4- ) L 2.- ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~~~~~~~4 LUU 1) 1 .4)..U J).ULUUGLU)U.J)U 4)UUi))..U))) -UUUUU )U.U

U)~ ~ ~ ~~~~~~~~)C U)UUU))))UUU UUU -) -)))))) -))UU LULUU4Ui..4~ ~ ~~~~~~~~~~~~~~r ,Ui),S I a1a aU.aaaaf a Uaaaa

C2 U). LU 3 c~ LU om W- x -n - SS W- hn V ) .1 (U -Z -c -4 5o -0 - -0 U o AC 4'- IU )1 UI UU S I1 U a -=LU 49' -mat LU) -f ) IU I0.L=.S0. .L .0 =L I-I UL *7L ' 'S' .US IU )U)U )a,0.. .6. )CSI U , )m~ISU

U) CD 4=0 4m) =) a3, U) ad' C30 =) U) 10 43. 4S CDO adS U) CU) U) 40 4m 40. C2. C) CD U) 4% 4U) UD 4) 4x 4W U) C2. U)L )U )U ) )U )U U0 ) .0

to m)U )U .U .U .U . )U )=U )U )U )U .U U )U )U )U )-U )U U )aU -U )U )U )U U

C-

Cpm o 4 3 OC D 40C *Cp C P <1 4 pCC : CC. C c 0C 3 0 PC. = C.4 aC C 31 4

U) ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ . 21 a a. 2- 3. a 31- aaaa aa aaaaa

a m C, aC C3 a. 4m 4c 'm C, m C, C' C, m m c. 4a q* ab C3 cp a~ 00 ap a5 CD C3 a aP 00 CD CP ato 2,24 2 , 2 !!W WW, mwlr 2 b.) , r r V

L" 4W 4m om -a t a a9 a9 a c c a4 9 a C a-a a6 a Z 4a aW a VC a a a9 ac -C ac ac 4a aI ) U ) IS U f f ) IS ISU) aSf SU f , ) V )a, -

LUiP.s . P P . P .P P .P .P P .P . P . PP . .P .P . P PP .P .P

LI L f

a3

CD CDj 4= U; U)LU L Cl -A ;: XULU

=D = = = D 49 -M -=

L ) .. C= l -oCDCL. D C c cc:- Lai WD -c =D -cm C 3CD _a. C" an CDJ - j co M..C:3 M- - C C -~ - U UU )

a- Q. Ia. L 9L C2 C= Ln L"

L, C LA : ca -= 9 - C- -j C= - L Jn 4. -j 4c -Z 0. LU C D D= U U

W= cc -cc Cc LU LU LU C" W -- - acc &- cc C2 -N -. S muc = =

L. Lo M.. 'm Li -= -= LU MELD UL ~ C CD 2=w C-2 .1 ==2 C) CD

-3 -j Li C3 dmL i L D W - ~ U0 ~~~- =1 i LU- CD- U UU)L )

- = - j L i L i. 4 = L O A . . C L Da- C2 - CL C= - I- U C= -j

w wC L w Ci -j Li wD IC6.L -M coP.. .. J U - W 2- MD MD =D =. P.- U) = cn a. C

Cm ZLJ = L LU _5CD L" U.i-c

vD g -W zD .. j =1- UU LU CD LUJ U) =U)= C L . 4m

=AJCw=I . = -K .- =CD CM CD u CD C " C-2 f- LLUnLC- o jL Li n C DUC . -n *LCI-i-LUU I. -M -W

a- ) U2 D ) Lb - cc. =-C LU="" L - L. _ )U =W =) w= ) - C LU cm

= LUi.. b. -C C = W c c -MtCD w. w C> - w cc I.- 'K LU 4c L DU L c

CDat -mAmL DC CD W.U.UCC0 C4 U m

.4C D Li _5 4c= .. m "i. DC . )L DU=i =r L nam UU d= =CD CD

CDU2 LU.. L ~ ~ ~ -W.~ w) w)DDLU .0L CLi ~ ~ = = ;.4 -- CD ) U ) CU C D U- C3 -

-M L c c 4 4 U).U0LU.L.0 U) 2U).M.=n4 CtlAD CN. wL"U cmC L cDU -- I CD C C U C CD,

Ln I=CD U3 C wD CD CD CDDC U CD.- *

-0 = In L U) =L -C * 4= S -C. C C CD L" w CD i IS CD C LA.

LU LU U ~ en Lft L" -cc 2. wD = = D ca 8=-C U DL DL 0C-L

C-3 LU cm =U w) 2. MD U3 Li j Li "~ wi 0 = wDLi -n -= LUn U La

d= c -j XM Lim d= I-cc ~ a) r ) M w W.= co to CD I- C c to I. Li fU L L

$- -CU C C C= L L UL ,C - UiLi = -CDK -- LU DL C

C- CD CD CD 4L CD U) gn CD CD CD CDn =D CD =D tn =D C" C." CD WD CD3 WD en CD CD Ln CD CD

C3D CDl c:. c:, cD 40 C3D C:. 4= 4=0 CD C C D CD C=O C.4= C cD C2. CD CD, CO -1 c!CDj to C=

C"D

I.- C C %n =) %" CD CD CD CD, c. CD. CD <D1 CD, CD CD C> cD CD CD C2D CD CD I= CD C

C" Co C C ') C2. CZ,) CU N CN, ') - p CDC3 zt'=) ~ lCD4 SC') 0£') c0 cz

- CDDC C D 1 CD ' CDD CD CD CD3 D CC D CCDDD CCCCCCD CC D CI C

1. D D C D C D C3D CD. -M CD 4= C2 C2D ICD Cp CD CD, CD C3D CD CD CD CD CD CD CD 4=4 C:,

Cb aLi

4 cD CD. CD D C C D CDD CD. 4=- CD CD C> CD C2D CD CD It, CD. CD CDl CD CD. c C CDU) C> CD en r' b" 612U CU CU cu C.) -W CUj CU C-2 c'" C'n 8w C C2, CD ru s. CU 4r C,

UU)

c:- . Ci CU CD CD cc - CU .8P C -0 -D 0 C C C CD CD-CM r8 CD MD -0 .0 CU 4p .o .CDC') . CU M Cr) Co U)CD .0 wU)-C C 0.4 c" C'r-

C, C2

rU CDbs ~C3 iD .- C0U4 * tL 3 20 C.C 4C- CD -CD CD. 42 CD, Ol C=. C) C I) CD C C CD. C

CCD C3D CD C3D CD CD C 5t CD CD CD CD, CD C CD CD 0 g CD2CD4C CD CD, CD CD CD C

.4 4cf Li

LU Q

36

TABLE V. Percentage Totals of ACL Usage for Courses of CGSC

Couse Hourx s TsI CAI SM ES GAM S=CAS 296.5 157.7 0.0 63.5 60.8 2.4 2.4 9.8

53.2% 0.0% 21.4% 20.5% 0.8% 0.8% 3.3%

CGSOC 674 241.3 16.2 !92.1 53.2 42.5 63.4 65.4(Core) 35.8% 2.4% 28.5% 7.9% 6.3 9.4% 9.7%

CGSOC 3120 1834.6 0.0 970.3 74.9 12.5 227.8 0.0(Elec) 58.8% 0.0% 31.1% 2.4% 0.4% 7.3% 0.0%

SAMS 2116 971.2 0.0 0.0 399.9 0.0 596.7 146.045.9% 0.0% 0.0% 18.9% 0.0% 28.2% 6.9%

Total 62065 3204.8 6. 1225.9 588. 51a B2g2 22i2106.8% 0.5% 40.9% 19.6% 1.9% 29.7% 7.4%

CAS3 Hours NA I= CAI SIT ES GAM SC296.5 157.7 0.0 63.5 60.8 2.4 2.4 9.8

53.2% 0.0% 21.4% 20.5% 0.8% 0.8% 3.3%CGSOC(Core) 674 241.3 16.2 192.1 53.2 42.5 63.4 65.4

35.8% 2.4% 28.5% 7.9% 6.3% 9.4% 9.7%

SAMS 2116 971.2 0.0 0.0 399.9 0.0 596.7 146.045.9% 0.0% 0.0% 18.9% 0.0% 28.3% 6.9%

Total 3086.5 17. 16 2511 5M. 4A. ffi2A 221244.4% 0.5% 8.3% 19.6% 1.5% 21.5% 7.2%

37

RESULTS

Presentation of Project Team Suggestions

The judgments made by the project team regarding the appropriate categories of ACL andthe proportion of classroom hours to which each is appropriate are listed by subcourse in thepreceding tables. The project team is unable to estimate how long it would take for its sugges-tions to be fully implemented, primarily because of current uncertainties concerning howsuitable courseware will be developed and how long that process will take.

Summary

The project team believes that all CGSC instruction would benefit from applying com-puters to administrative tasks associated with instruction, primarily by raising the productivityof the staff and faculty. Furthermore, the team endorses the choice of small group instructionas the preferred training strategy at CGSC. It also believes that 55 percent of resident CGSCinstruction would be improved by suitable ACL in the classroom. Estimates by the projectteam of the proportion of course classroom hours that could appropriately use those applica-tions are shown by course and category of application in Table VI. As already noted here, ap-plications from the category of administration, as defined in the taxonomy displayed in themain project report, are universally appropriate in virtually every activity of the College andconsequently are not listed. The other categories are symbolized as follows:

0 = No applications other than administrative are suitable.2 = Testing.3 = Computer-assisted instruction.4 = Simulation for individual training.6 = Gaming.7 = Simulation for collective training.

39

TABLE VI. Number of Classroom Hours Using the Indicated Category of ACLand Percentage of Total Course Hours

Total CourseClf S Hours fi 2 34 5 2GAS 296.5 157.7 0.0 63.5 60.8 2.4 2.4 9.8

53.2% 0.0% 21.4% 20.5% 0.8% 0.8% 3.3%

CGSOC 674 263.5 16.2 214.3 70.8 42.5 1.3 65.4(Core) 39.1% 2.4% 31.8% 10.5% 6.3% 0.2% 9.7%

SAMS 2116 971.2 0.0 0.0 399.9 0.0 596.7 146.045.9% 0.0% 0.0% 18.9% 0.0% 28.2% 6.9%

The data in Task F established a correlation between the cognitive levels of CGSC learn-ing objectives and those of the KSA needed by graduates of the various schools of the College.This correlation can be seen in the table above in the increasing use of gaming and SCT as thecourse subject matter concerns itself with higher order command and control problems in theprogression from CAS3 through CGSOC to SAMS. The table also shows the high degree ofapplicability of CAI for instruction in CAS3 and CGSOC, of SIT for CAS3 , and gaming forSAMS.

40

DISCUSSIONAINTERPRETATION

The Project Team Ideal Situation

The preceding judgments assume that the appropriate courseware, models, games, and in-telligent tutors had all been designed and that their use in the classroom would demonstratea significant improvement in the effectiveness of instruction compared with previous methods.Thus, they envision a situation that can only be regarded as an ideal toward which CGSC mightdirect its efforts in coming years.

Prerequisites

Realization of any version of an ideal situation for presenting instruction at CGSC with theaid of computers will have to follow the formulation and implementation of an approved planfor acquisition of a network of suitable types of computers at CGSC, the design and prepara-tion of appropriate courseware for each subcourse with a role planned for computers, and thewriting of accompanying programs for the computers.

The establishment of functional requirements demonstrating a need for the enhancedcapability that computers could bring to CGSC is the obvious first step in preparing a suitableacquisition plan, which must be made compatible with other plans of the installation and ofhigher headquarters. The ideal presented in Fig. 1 could serve to identify part of the function-al requirements.

The development of software for classroom instruction using computers would be a con-tinuing task whose definition should logically serve to describe the characteristics of the com-puters sought through the acquisition plan. Clearly, the computers acquired should be capableof accommodating the most complex of the required computer programs, that is, those creat-ing the greatest demand on the capacity of the computers using them. An unavoidable com-plication of the logic is that until at least a preliminary design of suitable software has beenprepared the capacity required of each computer can only be estimated. For flexibility, thecomputer network should be designed for maximum feasible growth potential.

Selecting Computer Applications

Small group instruction appears to be the most appropriate strategy for imparting the KSAwith which CGSC curricula are concerned. This situation may be especially true becausefunctioning as part of an efficient team is a major element in the qualifications of a competentstaff officer and teamwork is difficult to teach without the opportunities for practice affordedin small groups. There are obvious exceptions, as when a guest speaker can spend only a limitedtime at the College, forcing a single presentation or appearances before a few large groups.By and large, however, teaching small groups of students will be more effective than the alter-natives and should present few problems when computers are used in the instruction.

Implicit in estimating the capacity required of the computers is some conception of themost demanding kinds of programs they may be required to handle. One aspect of that kind

41

of conception has to do with the kinds of combat simulation models CGSC will adopt for usein the classroom and in the other activities in which it will be engaged, for example, doctrinedevelopment and the BCTP. The project team believes that large, exquisitely detailed, deter-ministic models, such as some that have been developed for use in operations research/sys-tems analysis activities, may be inappropriate for use in CGSC classrooms.

A number of reasons can be advanced for taking this view. First, effective use of this kindof model could only be made by instructors who had been extensively prepared beforehandby becoming intimately familiar both with the model and with the programs and their com-ponents representing the model in the computers using them.

Additionally, specific learning objectives might more easily be achieved without elaborateattempts to recreate actual combat situations in a war whose terms might well turn out to beunprecedented. Effort to capture the complex interactions characterizing force on forceengagements of whatever nature in a model must often resort to aggregating some predictablekinds of interactions in an approximation reflecting an expectation of the net result of thoseparticular interactions. Inevitably, an element of unreality is thereby introduced that, in fact,may detract from achieving learning objectives, as when the student implicitly accepts the ex-pectation in place of the considerably more complicated and unpredictable outcomes he willface in combat. This situation seems particularly likely in the case of deterministic models thatprocess inputs to produce outcomes that are predetermined by the algorithms built into theprocess.

In particular, it is the conviction of the project team that many CGSC learning objectivescould effectively be reached through the use of games, which can be designed perhaps moreeasily than more faithful simulations to present the student with the need to appreciate andapply concepts rather than procedures. A place will remain in the classroom and in the fieldfor simulations, both for individual and for collective training. It is not clear, however, that thesame simulations would serve in every case for each of these different purposes. Specifically,while each of those teaching contexts needs to portray particular hypothetical situations withwhich to confront students, it may not be necessary in the classroom that each successive situa-tion be seen to follow from the action chosen in the preceding situation.

Finally, it is evident that realistic force on force combat computer simulations, apart fromthe difficulty of deciding the criteria on which to base an assessment of their realism, are someyears away from development to the point of obviating the current need for large and expen-sive overhead in the form of human controllers and other support personnel. In the meantime,it may be that less ambitious simulations and games could be developed for achieving specifi-cally identified CGSC learning objectives at much less expense and in a much shorter periodof time, permitting their early introduction in CGSC instruction.

In any case, CGSC will have to decide whether tools for developing doctrine and for train-ing staffs from Army organizations in the field will be the same in every case as those used forits part in teaching students the KSA in its schools that they will need to perform as principalstaff officers of tactical and operational commands.

42

CONCLUSIONS

Instructional Strategy

Instructing small groups of students in each classroom appears to be the most effective ap-proach to teaching the material CGSC needs to convey to its graduates. This strategy facilitatesthe use of networked computers in the classroom to simulate the functioning of a staff with itsteamwork and can be adapted to other computer applications.

Computer Opportunities

Opportunities abound for applying appropriate further uses of computers in CGSC class-rooms. Those uses could come from each of the categories of ACL defined in this report andlisted in the section entitled Analysis of ACL Elements.

Hardware Acquisition

The computers that will be required for ACL at CGSC should constitute a networkdesigned to accommodate the following: the courseware that will have to be developed, theother missions of CGSC in addition to individual training, and the acquisition plans of the in-stallation and higher headquarters.

Courseware Development

The development of courseware for ACL at CGSC should begin as expeditiously as possible after a careful assessment of each subcourse learning objective for the suitability ofachieving that objective through ACL from one or more of the categories defined in this report.

43

,a

REFERENCES

Alessi, S. N., and S. R. Trollip, Computer-Based Instruction- Methods and Development (Pren-tice-Hall, New Jersey, 1983).

Andrews, A. E., and M. S. Trainor, "Constructing a Computer-Tutor," Daita Trainng, February1987, pp. 28-34.

Avner, R. A., "Production of Computer-Based Instructional Materials," Issues in InstructionalSystems Development, H. F. O'Neil, Jr., Ed. (Academic Press, New York, 1979).

Baker, F., "Computer-Managed Instruction: A Context for Computer-Based Instruction," inComputer-Based Instructior H. F. O'Neil, Jr., Ed. (Academic Press, Orlando, FL, 1981) p.23.

Brown, J. S., and R. R. Burton, "Diagnostic Models for Procedural Bugs in Basic MathematicalSkills," Cognitive Science, 2, 155-192 (1978).

Burton, R. R., "Diagnostic Bugs in Simple Procedural Skills," in Intelligent Tutoring Systems, D.H.Sleeman and J.S. Brown, Eds. (Academic Press, Orlando, FL, 1982), pp. 157-183.

Carbonnel, J. R., "Al in CAI: An Artificial Intelligence Approach to Computer-Assisted Instruc-tion," in IEEE Transactions on Man-Machine Systems 11, No. 4, 190-202 (1970).

Clancey, W. J., "GUIDON," Journal of Computer-Based Instruction, 10, No.1, 8-14, (1983).

Crawford, C., Balance of Power, (Microsoft Press, Redmond, Washington, 1986).

Goldstein, I. P., 'The Genetic Graph: A Representation for the Evolution of ProceduralKnowledge," International Journal of Man-Machine Studies 11, 51-77 (1982).

Johnson, W. L., and E. M. Soloway, "PROUST: Knowledge-Based Program Debugging," inProceedings of the Seventh International Software Engineering Conference, (Orlando, FL,1984) pp. 369-380.

Kearsley, G., Artificial Intelligence and Instruction (Addison-Wesley, Redding, MA, 1987).

Kearsley, G., Computer-Based Training, (Addison-Wesley, Reading MA, 1984) p. 20.

Kulik, J. A., C. C. Kulik, and P. A. Cohen, "Effectiveness of Computer-Based College Teaching:A Meta-Analysis of Findings," in Review of Educational Research 50, No. 4, 525-544 (Winter,1980).

O'Shea, T., R. Bornat, B. DuBoulay, M. Eisenstadt, and I. Page, 'Tools for Creating IntelligentComputer Tutors," in Intelligent and Human Intelligence, Elithorn and Baneriji, Eds. (El-sevier Science Publishers, 1984), pp. 181-199.

45

4

Roberts, F. C., and 0. Park, "Intelligent Computer-Assisted Instruction: An Explanation andOverview," Educational Technology., 23, 7-12 (1983).

Sleeman, D. H., and R. J. Hendley, "ACE: A System Which Analyses Complex Explanations," inIntelligent Tutoring Systems, D.H. Sleeman and J.S. Brown, Eds. (Academic Press, New York,1972), pp. 99-118.

Wenger, E., Artificial Intelligence and Tutoring Systems (Morgan Kaufman, Los Altos, CA, 1987).

46