Embed Size (px)
Citation preview
AN INTEGRATED APPROACH TO COMBINE
COMPUTER-BASED TRAINING (CBT) AND
IMMERSIVE TRAINING (ImT) FOR MECHANICAL ASSEMBLY
By
YOUNG JUN KIM
A dissertation submitted in partial fulfillment of the requirements for the degree of
DOCTOR OF PHILOSOPHY
WASHINGTON STATE UNIVERSITY School of Mechanical and Material Engineering
May 2007
© Copyright by YOUNG JUN KIM, 2007
All Rights Reserved l
© Copyright by YOUNGJUN KIM, 2007 All Rights Reserved
ii
To the Faculty of Washington State University:
The members of the Committee appointed to examine the dissertation of YOUNG JUN KIM find it satisfactory and recommend that it be accepted.
___________________________________ Chair ___________________________________ ___________________________________
iii
ACKNOWLEDGMENT
I wish to express my thanks first to God and my mother. Without the support and
confidence of my mother this would not have been possible. I would also like to thank
my advisors, Dr. Uma Jayaram and Dr. Sankar Jayaram for being patient, supporting, and
guiding all my research on my dissertation. Special thanks to Dr. Jandhyala for being on
my committee.
iv
AN INTEGRATED APPROACH TO COMBINE
COMPUTER BASED TRAINING (CBT) AND
IMMERSIVE TRAINING (ImT) FOR MECHANICAL ASSEMBLY
Abstract
by Young Jun Kim, Ph.D.
Washington State University May 2007
Chair: Uma Jayaram
There is currently a vital need to improve effectiveness and efficiency of training
technologies used in industry – especially in the areas of assembly, disassembly,
maintenance, installation, and troubleshooting. Current training techniques vary from
paper-based (books, manuals) to computer-based (presentations, web pages, on-line help,
etc.) to immersive (VR-based) systems. This dissertation presents an integrated approach
to create a Computer-Based Training (CBT) and Immersive Training (ImT) system in a
unified manner to exploit commonalities and that is integrated with a Learning
Management System (LMS).
We coin this approach iTrain, integrated Training. iTrain consists of CBT, ImT, and
a learning management tool. The goal is to enable transformational change of mechanical
assembly training through the creation of a new generation training ensemble. In this
dissertation, the methodology, approaches, and algorithms for such integrated training
systems are presented. Based on this methodology and these approaches, a CBT
application that allows interactive training using traditional computer tools (keyboard and
mouse) and an immersive application that allows interactive learning using VR devices
v
have been both developed and integrated. These applications are integrated with the
Sharable Content Object Reference Model (SCORM) so that the training can be managed,
tracked, and evaluated in an effective and efficient manner. A unified approach has been
designed to manage common data such as model name, and model ID. A database has
been designed and implemented for access from CBT, ImT, and LMS. This prototype
integrated training system is presented and discussed in this dissertation.
vi
TABLE OF CONTENTS
ACKNOWLEDGMENT................................................................................................... III
ABSTRACT...................................................................................................................... IV
1. INTRODUCTION .......................................................................................................... 1
2. LITERATURE REVIEW ............................................................................................... 4
3. PROBLEM DEFINITION AND PROPOSED SOLUTION.......................................... 9
3.1 PROBLEM DEFINITION ................................................................................................ 9
3.2 OVERALL SOLUTION ................................................................................................ 10
3.3 RESEARCH OBJECTIVES............................................................................................ 13
3.3.1 Computer-Based Training (CBT) Application............................................................................13
3.3.2 Immersive Training (ImT) Application.......................................................................................13
3.3.3 Learning Management System (LMS)........................................................................................14
3.3.4 Unified Approach........................................................................................................................14
4. OVERALL APPROACHES......................................................................................... 16
4.1 OVERVIEW ............................................................................................................... 16
4.2 COMPUTER-BASED TRAINING .................................................................................. 16
4.3 IMMERSIVE TRAINING .............................................................................................. 17
4.4 LEARNING MANAGEMENT SYSTEM.......................................................................... 18
4.5 UNIFIED APPROACH ................................................................................................. 20
5. COMPUTER-BASED TRAINING METHODOLOGY AND DESIGN..................... 22
5.1 OVERVIEW ............................................................................................................... 22
5.2 COMPUTER-BASED TRAINING METHODOLOGY........................................................ 23
5.3 ALGORITHMS IN THE CBT APPLICATION .................................................................. 25
5.3.1Creation of a Scene Graph ...........................................................................................................25
5.3.2 Support for Assembly/Disassembly Functionality......................................................................28
5.4 DESIGN, DESCRIPTIONS, AND IMPLEMENTATION...................................................... 29
5.5 MODEL PROCESSING APPROACH.............................................................................. 33
5.5.1 3D Geometric Models .................................................................................................................34
5.5.2 Data Extraction ...........................................................................................................................35
6. IMMERSIVE TRAINING METHODOLOGY AND DESIGN .................................. 37
6.1 OVERVIEW ............................................................................................................... 37
6.2 IMMERSIVE APPLICATION - I .................................................................................... 38
vii
6.2.1 Methodology for Training Sessions in ImT ................................................................................40
6.2.2 Training on Assembly of a Large Press Machine .......................................................................41
6.2.3 Training on Assembly of the Fifth Wheel to a Truck Chassis ....................................................48
6.3 IMMERSIVE TRAINING APPLICATION – II.................................................................. 54
6.3.1 Enhancement of the ImT Application .........................................................................................69
7. LEARNING MANAGEMENT SYSTEM METHODOLOGY AND DESIGN .......... 71
7.1 OVERVIEW ............................................................................................................... 71
7.2 METHODOLOGY ....................................................................................................... 72
7.3 DESIGN OF THE SCORM WEB PAGES ...................................................................... 73
7.4 IMPLEMENTATION OF THE SCORM WEB PAGES...................................................... 73
7.5 INTEGRATION WITH APPLICATIONS .......................................................................... 80
8. UNIFIED APPROACH ................................................................................................ 84
8.1 OVERVIEW ............................................................................................................... 84
8.2 METHODOLOGY OF UNIFIED APPROACH .................................................................. 84
8.3 DESIGN OF UNIFIED APPROACH ............................................................................... 85
8.4 IMPLEMENTATION OF DATABASE ............................................................................. 87
8.5 ACCESSING DATABASE ............................................................................................ 89
8.6 TRAINER PERSPECTIVE............................................................................................. 90
8.7 TRAINEE PERSPECTIVE............................................................................................. 91
9. TEST CASE.................................................................................................................. 95
10. SUMMARY AND CONCLUSION ......................................................................... 102
REFERENCES ............................................................................................................... 106
APPENDIX A................................................................................................................. 110
APPENDIX B ................................................................................................................. 122
viii
LIST OF FIGURES Figure 1: Ensemble training.......................................................................................... 11
Figure 2: Overall system............................................................................................... 12
Figure 3: SCORM web page......................................................................................... 19
Figure 4: Overall scenario of iTrain.............................................................................. 20
Figure 5: Overview of CBT .......................................................................................... 23
Figure 6: The format of the data file ............................................................................. 27
Figure 7: A scene graph ................................................................................................ 27
Figure 8: Apply a transformation matrix to a selected part .......................................... 29
Figure 9: Classes of the CBT application ..................................................................... 29
Figure 10: A snapshot of the CBT application ............................................................. 31
Figure 11: A basic node of scene graph........................................................................ 32
Figure 12: Highlighting a component in the CBT application ..................................... 33
Figure 13: CATIA V5 pseudo code .............................................................................. 36
Figure 14: Overall system of ImT................................................................................. 38
Figure 15: Relative size of operator large press machine ............................................. 43
Figure 16: Exploded Pro/E assembly model X31-00-21440 of large press machine ... 44
Figure 17: Comparison between Pro/E assembly hierarchy and VADE assembly hierarchy ............................................................................................................... 45
Figure 18: Environment modeling ................................................................................ 46
Figure 19: Custom environment for the fifth wheel assembly ..................................... 50
Figure 20a: Lowering the fifth wheel with a crane in the real factory.......................... 51 Figure 20b: Lowering the fifth wheel with a crane in VADE ...................................... 51
Figure 21: Labeled bolt positions ................................................................................. 52
Figure 22: Worker inserting bolt 1 in VADE ............................................................... 53
Figure 23a: Worker inserting bolt 2 in real life b: 1st person and 3rd person views in VADE ................................................................................................................... 53
Figure 24a: Positioning bolt 5 in a real factory b: Positioning bolt 5 in VADE........... 54
Figure 25: Overall architecture of ImT [47] ................................................................. 56
Figure 26: Reorganizing assembly hierarchy [47]........................................................ 60
Figure 27: Constraints for engaging tool and fastener [47] .......................................... 65
Figure 28: Overall flowchart of the event handling [47] .............................................. 68
Figure 29: Flowchart of event handling for ToolInHand stage [47]............................. 68
Figure 30: Flowchart of event handling for rotating bolt stage [47]............................. 69
Figure 31: A wrench is gripped in ImT ........................................................................ 70
Figure 32: Design of the training course....................................................................... 73
ix
Figure 33: A relationship between the web page and LMS.......................................... 74
Figure 34: SCORM Conceptual Run-Time Environment [31]..................................... 75
Figure 35: SCORM API, API Instance, API Implementation [31] .............................. 76
Figure 36: imsmanifest.xml .......................................................................................... 78
Figure 37: RELOAD GUI............................................................................................. 79
Figure 38: Remediation of the practicing mode ........................................................... 80
Figure 39: Integration between SCORM web pages and digital application................ 81
Figure 40: FirstLaunch function ................................................................................... 82
Figure 41: Data structure of the shared memory .......................................................... 83
Figure 42: Overall architecture of unified approach..................................................... 86
Figure 43: Database design........................................................................................... 87
Figure 44: The design of DLL ...................................................................................... 88
Figure 45: Database access in JavaScript of the web page........................................... 90
Figure 46: LMS user interface ...................................................................................... 92
Figure 47: Course registration and available courses ................................................... 93
Figure 48: SCORM web page with CBT and ImT ....................................................... 94
Figure 49: The learning mode (Step 1-3)...................................................................... 97
Figure 50: The learning mode (Step 4–9) ..................................................................... 98
Figure 51: The practicing mode.................................................................................. 100
Figure 52: Test mode .................................................................................................. 101
Figure 53: Results from SCORM ............................................................................... 101
x
LIST OF TABLES Table 1: Assembly steps in real and virtual environments ........................................... 47
Table 2: A list of tasks for flat tire replacement ........................................................... 95
Table 3: A list of tasks for a flat tire replacement......................................................... 99
xi
Dedication
This dissertation is dedicated to my mother,
who provided both emotional and financial support.
1
CHAPTER ONE INTRODUCTION
State-of-the-art training techniques in mechanical assembly have evolved quickly
with the introduction and rapid diffusion of digital training technologies. The growing
availability of training technologies has challenged training personnel to assess and refine
the way they structure courses, deliver content, and measure achievement. Some of the
technologies provide Computer-Based Training (CBT) while others may provide
Immersive Training (ImT).
Computer-Based Training is a process of training using CBT applications installed in
computers. Recent developments in CBT have made it easier to access training courses
by using personal computers. The computer is used to show some of the processes being
modeled. For example, a trainee can watch an assembly process of a new product using
video files. Technologies such as Macromedia Breeze and EON Reality in CBT have
grown fast and wide.
Nevertheless, most applications in CBT are based on pre-defined scenarios, and
trainees do not fully interact with the applications. In addition, some applications do not
have functionalities to show simulations in the applications. These applications are able
to show some sequences, but the lack of interaction may affect trainees’ learning time or
perception. The keyboard/mouse interactions are key functionalities to be considered in a
computer-based training application.
Virtual Reality (VR) technologies have been used in a variety of training applications.
Immersive Training (ImT) is based on these technologies and has expanded and
developed rapidly over the past few years. It has great promise as a training tool. Some of
2
these applications have been used for training astronauts, dentists, medical doctors,
construction workers, and engineers [1]. Even though costs to set up devices such as
immersive screens and tracking devices may be expensive, immersive training is one
solution to address training needs in difficult and dangerous situations. For example, a
high-risk training session, such as a medical surgery or safety training has the potential to
benefit from ImT. This is also the case where the cost of hands-on training in certain
fields such as construction of a special structure would be very expensive or situations
where practice using realistic interaction is important.
A learning management system (LMS) is a software package that manages and
delivers the training contents through a world wide web (WWW). The LMS usually plays
the role of providing tools for registering, delivering, and evaluating the course. Web-
based training has been growing fast, enabling trainees to gain information more easily
and quickly than in the past. Integration between the digital training applications and web
pages makes it easy to provide instructions and manage courses. Creating web-based
courses without a learning management system needs careful and involved management.
Many training technologies are not designed to be easily integrated with other
technologies. For example, immersive simulation training environments are relatively
new, and integration with commercially available learning management systems has
rarely been attempted, despite the anticipated benefits such an integration would produce.
Additionally, not all computer-based instruction technologies are designed to promote the
best practices in training and learning. An ensemble is a set of components that work
together. The primary objective of this dissertation is to conceptualize, design, develop,
3
and demonstrate an ensemble for training, coined iTrain. iTrain consists of a set of key
components and a unified approach working with a learning management system.
The goal is to enable transformational change of training through the creation of this
new generation training ensemble. In this dissertation, the methodology, approaches, and
algorithms for a system will be designed and implemented. A prototype that demonstrates
the feasibility of this solution will be created.
4
CHAPTER TWO LITERATURE REVIEW
This chapter describes a review of some of the previous work in fields that are
relevant to our work, such as Computer-Based Training (CBT), Virtual Reality (VR)
applications, learning management systems (LMS), and software engineering. The
research for this dissertation will build upon several research concepts and efforts. There
are many illustrative efforts that have used educational technologies to promote learning
and training [2-4]. These studies discovered that a well designed technology-supported
curriculum can significantly improve learning and training environments. Lessons
learned from examples such as these will be incorporated into the design.
CBT can maximize the one-on-one interaction between an instructor and trainee,
including the interchange of information and feedback. The benefits of CBT interactive
learning are as follows: 1) The trainee receives immediate feedback on performance in a
structured and non-threatening environment; 2) Trainees can focus on the information
they need, at the point at which they need it; 3) The curriculum and pace can be tailored
to the experience level of each trainee; and 4) Active learning occurs as trainees interact
one-on-one with their computer [5].
Some noteworthy examples of organizations that have CBT programs are Armstrong
Lab, the Army Research Institute, the Office of Naval Research, and the Naval Air
Warfare Center [6]. A 1992 study by Interactive Communications found that students
completed their material 60 percent faster on computer based interactive video disc than
they did in a classroom. The study credited improvements to the ability of interactive
CBT to allow students to put their knowledge into action quickly via simulations. A load
5
dispatcher training simulator at the Power System Training Institute conducts two week
courses that cover various aspects such as Black Start and Synchronization of Systems,
Basic System Characteristics, Economics, and Load Schedules [7].
Considerable literature exists that describes how immersive virtual reality (VR) has
opened new realms in the area of training related to the military, manufacturing, and
medicine [1,8-15]. The literature states that VR-based learning can, in some cases, offer
profound benefits over conventional learning methods [16]. For example, a virtual reality
training application being developed for the U.S. Army, The Dismounted Infantry Virtual
Environment [17], will allow soldiers to be totally immersed in a simulated battlefield
environment and participate in simulated exercises. A virtual learning system [18] was
modeled for manufacturing industries as an integrated system consisting of a training
visualization suite, an interface module, and an instruction module. Motorola conducted a
study of robotic manufacturing plant operations by comparing three groups: (1) trainees
in an immersive VR environment, (2) trainees who only observed the environment being
simulated, and (3) a control group that worked directly in the manufacturing laboratory.
The fewest errors were made by the students using VR, particularly during complex tasks.
Boeing uses the virtual environment in training. This method has increased training
efficiency and improved student output [19].
A 2004 report based on data from seventeen countries states that the market for
visual simulation/virtual reality systems is financially strong, valued in 2003 at nearly
$43B, and that this trend represents a healthy annual growth rate of 9.8%. Strong growth
is predicted in most areas of the industry, continuing for the next several years, with a
projected market value of almost $78B by 2008 [20].
6
While real-life experience is in some cases the best way to learn, at times it is too
expensive, dangerous, or not feasible [5].
The term “blended training” is now becoming increasingly used by companies to
describe the way they combine traditional and electronic learning to provide the best
overall results [21-23]. “Blended does not mean chopping off the first day of a classroom-
based course and adding a bit of CBT (computer based training)… What the 'blended
movement' gives us is the chance to get our breath, re-assess, re-think and re-engineer
training processes so that the benefits of each method can be realized and maximized for
tomorrow… While healthy skepticism is good, it would be wrong to go into denial over it
because, in the long run, blended is a compelling proposition [24] .”
Companies considering training as a solution for their business need to consider a
variety of delivery options. Each of these options has specific strengths and weaknesses
that must be considered before a viable solution can be constructed [25]. Examples of
products that have used the blended strategy successfully include Cybesphere [18], which
employs a fully immersive VR visualization suite along with a collaborative product suite
to achieve an ideal training environment for manufacturing industries; Thomson Learning,
which combined three of its units, NETg, Wave Technology and Course Technology, to
form a single entity [24], and Right Hemisphere [26]. The key to a properly developed
blended learning program is to know what, when and how to combine these strategies
[27]. It has been predicted that we can expect to see more and more companies offering
this type of “bundled” blended learning solutions [28-30].
A learning management system is able to manage, track, and evaluate a training
course. Training applications need to be integrated with a learning management system.
7
Shareable Content Object Reference Model (SCORM) is a learning content standard
developed by ADL [31]. SCORM is used for communicating data between web-based
instructions and the learning management system. Once the learning content is created, it
can be used with little adaptation in any LMS. SCORM also allows re-use of content.
SCORM is used widely for training courses, and it is a standard in developing
interoperable learning content [32]. In other words, any conforming LMS can allow
eLearning developers to post their web courses developed in SCORM.
Bringing CBT and ImT together with a learning management system requires
knowledge of software architecture and software engineering. Many philosophies and
theories relate to software architectures [33,34]. For example, the concept of structural
modeling resulted from the efforts of the Air Force Aeronautical Systems Command
(ASC/YW) and has been used by Air Force contractors since the late 1980s to design
large-scale, high fidelity trainer simulation software [35]. Object oriented design
techniques apply the concept of abstraction, encapsulation, hierarchy, inheritance, and
modularity [36]. Booch states: “We have observed two traits common to virtually all of
the successful object-oriented systems we have encountered, and noticeably absent from
the ones we count as failures: the existence of a strong architectural vision and
application of a well managed iterative and incremental development cycle [36]..”
Architecture-based development is a process that utilizes software architectures as
the primary tools for the design, evolution, implementation, migration, and understanding
of a software system [35,37,38]. Architecture-based development involves domain
analysis, iterative development, effective representation and communication, and
enforced system conformance to the software architecture. Structural modeling [39] has
8
provided an effective solution to the problem of software complexity [39,40]. Structural
modeling views software architecture as the backbone of the entire software cycle, not
just the software development effort. The use of model problems as a component-based
software engineering technique is gaining importance [41]. An approach based on risk
analysis, realization of a model problem, and repair of residual risk is developed. This
process specifies the necessity of using an iterative or spiral process. Modeling simulation
architectures for distributed simulation are described in the literature [42-44].
This dissertation both CBT and ImT focuses on methods and algorithms to integrate
digital training applications with an LMS. The system developed in this dissertation has
been informed by concepts presented in this chapter and seeks to extend the knowledge
of creating digital training applications.
9
CHAPTER THREE PROBLEM DEFINITION AND PROPOSED SOLUTION
3.1 Problem Definition
Traditional training programs in CBT often employ instructional manuals using web
pages, animations, and movie files. Traditional training methods in CBT offer some
positive aspects such as availability, efficiency, speed of delivery, instant feedback, and
so on. There are well-known commercial CBT tools. For example, Adobe eLearning,
which offers solutions for rapid training, includes capabilities such as “create,”
“manage,” “deploy,” and “track the training courses” [45]. Adobe eLearning can show
animations and movie files.
Even though these CBT applications can show simulations, it is not possible for the
user to manipulate models. Most traditional training softwares in CBT do not allow the
trainee to interact with the models. Trainees can not directly interact with the models
using a mouse and a keyboard. This disadvantage becomes a roadblock in demonstrating
an assembly/disassembly process. In the assembly process, the sequence of the assembly
process and realistic interactions are important. Being able to use the correct tool to
assemble a component is also important. Traditional training methods might not be very
efficient in assembly process training because of this lack of interaction with the models.
A CBT application needs to be created that allows the user to perform
assembly/disassembly operations.
Immersive training applications have been widely applied to train certain sectors of
industry. ImT is still growing as a viable technology. There is considerable research
needed in creating and ImT environment for assembly/disassembly.
10
Another issue is that neither ImT nor CBT are well integrated with a management
tool. Some CBT applications have attempted to integrate with a management tool, but
these applications usually do not provide an interaction method with the models. Most
ImT applications have not attempted to add a management tool, so these applications are
mostly stand-alone training applications.
Also, the two have not been considered together to identify commonalities and
methods to exploit these commonalities. There are commonalities in terms of the models,
the overall interaction schemes, and the consequences of the interactions. Thus, a well-
designed and well-coordinated approach to providing training in both a computer-based
environment and an immersive environment is desirable.
3.2 Overall Solution - iTrain
We have coined our solution iTrain for Integrated Training. iTrain is composed of
three key components (Figure 1). A Computer-Based Training (CBT) application is one
of the key components developed to instruct trainees using a common computer. An
Immersive Training (ImT) component is a core component created to support an
immersive environment that allows a simulated procedure to be performed in a natural
manner. One more component can be Live Training (LT), but LT is not considered in this
dissertation. Both ImT and CBT have been created to support the modeling and
simulation needs of iTrain. Differences between ImT and CBT, although both are
computer-based, are the level of interaction and sense of presence in the environments.
Both ImT and CBT communicate with standard web-based e-learning. The web-based
contents interact with a learning management system.
11
It is not the intent that all three components of the ensemble be used for every
exercise. Certain exercises lend themselves well to a planned sequence, starting with a
CBT exercise, transitioning to an ImT exercise, and culminating in a LT exercise, while
other sequences of training components may be appropriate for another training module.
Thus, the intent of the ensemble is to allow both individual and sequenced exercises
through the use of different configurations of technologies in the ensemble. In this
dissertation, digital training methods (CBT and ImT) are considered as being used
together as a training method. Live Training coordination will need to be blended in
creatively later – that is not our current focus. It is included here only to demonstrate the
complete model.
Figure 1: Ensemble training
Instead of a farrago of technologies crafted independently and pieced together, a
foundation and architecture is laid for a unified approach during the design of the
components to determine how they will work to provide a coherent set of task
simulations. This objective will be achieved by focusing on development of a common
12
repository in a learning management system (LMS) for the model that can be shared by
both the CBT and the ImT environments so that the administrative and technological
burden of maintaining both systems separately is alleviated. The computational model
interoperability between the two computer-generated environments will present new
methodology to allow domain specific scenarios to be generated in both environments
quickly and methodically. In addition, this commonality will provide human-computer
interface and transfer of training merits. Thus, a unified approach in training exercises
will contribute to a more sustainable solution for the organization and the individual.
Figure 2 shows the overall system of iTrain. The web pages are connected with a
digital application, and these pages are also in communication with a learning
management system. Students will interact with the web pages, which show instructions
of lessons, and they will also interact with digital applications, which will be either CBT
or ImT based.
Figure 2: Overall system
13
3.3 Research Objectives and Functional Specifications
The focus of the research in this dissertation is to design and create the CBT
application, the ImT application, and the Learning Management System (LMS) in an
integrated manner. The commonality between CBT and ImT will be identified to create a
common repository. The assembly/disassembly operations with tool interaction will be
created in both CBT and ImT. The following sub sections identify specific research
objectives for CBT, ImT, LMS, and the unified approach of the system as developed in
this dissertation.
3.3.1 Computer-Based Training (CBT) Application
Requirements for the CBT application are to design a new architecture in order to
interact with the models and to communicate with the LMS. A mouse and a keyboard
will be used as interaction methods so that the trainee is able to manipulate the models.
The CBT application will have the functionality to assemble/disassemble the models for
the assembly process training. The CBT application will be able to display the 3D
geometry models. The CBT application will provide a scene graph based on the assembly
hierarchy from the CAD system. As a supporting component, 3D geometry models and
properties of the models will be exported from the CAD system.
3.3.2 Immersive Training (ImT) Application
The ImT application needs to support training related to assembly/disassembly,
including complex cases that require a crane or a tool. The research goals of ImT are as
follows:
14
Create functionality to allow training using a crane for the assembly process
Create a functionality to allow the use of a mechanical tool in an immersive
environment
Create functionality to allow a virtual hand to interact with the tool and the
part during the assembly/disassembly process
Allow realistic assembly simulations with these mechanical tools in the
native CAD system
Use kinematics when a component is being assembled/disassembled
3.3.3 Learning Management System (LMS)
The overall architecture needs to be designed so as to provide an LMS and the
relevant connections to the digital applications (CBT or ImT). The LMS will be used for
the following objectives:
Manage trainees’ accounts
Manage a training course
Deliver a training course
Provide feedback to the trainee
Give a final score to the trainee
3.3.4 Unified Approach
The research objectives of the unified approach are to investigate and collect the
commonalities of the digital training applications and exploit them so that the trainee is
effectively instructed through iTrain. In addition, the repositories for common data
15
need to be defined and implemented for both CBT and ImT. The common repositories
need to be built into a database so that digital applications are able to update common
data such as geometry models, paths of models, and properties of models. In addition,
the web pages linked to the LMS should be able to access common data in order to
provide feedback to the trainee.
16
CHAPTER FOUR OVERALL APPROACHES
4.1 Overview
The overall approaches that will be pursued will create capabilities to support the
training mission in various digital environments utilizing a learning management system.
The subsequent sub-sections will describe the process in detail.
4.2 Computer-Based Training
The CBT application for assembly/disassembly training will have two main modes.
One is a “Learning Mode”. This mode allows students to view a certain predefined
assembly process as a part of the lesson. For example, if there is an assembly model, the
CBT is able to show the disassembly process to students. The other mode is a “Practicing
Mode”: This mode lets students interact with 3D models inside the CBT application.
Using a mouse and a keyboard, they will be able to assemble/disassemble a 3D model
inside the CBT application. When students interact with the application, the result gets
recorded in a learning management system. The trainer can evaluate the trainee’s
performance based on the result.
There are many graphics libraries and open sources, such as OpenPerformer,
OpenGL, and OpenSceneGraph. A CBT application will be developed for iTrain based
on OpenSceneGraph (OSG) [46]. OpenSceneGraph was chosen because of the following
featured capabilities:
It can read various 3D geometry data files, such as VRML, Inventor, 3DS,
and native OSG files (.osg, .IVE).
17
It is easy to navigate in the application.
It uses a scene graph to create a model hierarchy.
It can visualize a 3D stereo image.
It is scalable and an open source library.
4.3 Immersive Training
Two ImT applications have been developed and modified for assembly/disassembly
training. One is for assembly simulation and training with a crane in an immersive
environment and the other one is for using a tool such as a wrench for assembly training
in a VR system embedded in a CAD system.
The first immersive application was created to provide functionality so that the
immersive virtual assembly application could simulate a real assembly process. This
application will be used as a stand-alone application and it will show the capabilities of
the immersive virtual assembly application for training. This was an existing application
that has evolved over the years in the lab and was modified and fine-tuned with additional
functionality as a part of this dissertation for the training scenarios.
The second ImT application provides haptics-enabled virtual tools embedded in a
native CAD environment, such as CATIA V5TM [47]. The key functionality of the ImT
application is a unique capability to perform interactive assembly and disassembly
simulations in a native Computer-Aided System (CAD) environment. Since the
embedded system in a native CAD system is used as the ImT application, native CAD
models can be directly used for training purposes. The native CAD assembly data needs
to be integrated with constraints data that will be used in the virtual environment. In order
18
to accomplish this, some of the key issues are the following: the native CAD kinematics
capability needs to be integrated with the immersive environment, algorithms need to be
developed to dynamically organize the assembly constraints for use in manipulation with
a virtual hand for assembly/disassembly simulations and an event-callback mechanism
needs to be created in which different events and callback functions can be designed and
implemented to simulate different training scenarios in the virtual environment.
This second application has the capability to provide a simulation environment using
tools such as allen and box-end wrenches in the CAD system with the interaction of VR
hardware (tracking device, glove, and haptic device). As a part of the work in this
dissertation, the understanding of the theory and practice of tool/hand/part interaction
was expanded by using a combination of kinematic motion, collision detection, and
physical-based modeling in the ImT application.
4.4 Learning Management System
Standard HTML web pages will be used for students to interact with each course.
The web pages will be available through a learning management system (LMS). SCORM
will be chosen as the development standard to create common interfaces and data
exchange formats for the CBT and ImT lessons to conform with the LMS. In each web
page, SCORM will deliver content, track students, and get feedback from ImT or CBT.
SCORM will manage a list of students, check the completion of courses, etc. Figure 3
shows the data exchange processes for the web page with the API. The student will first
register for the course in SCORM, and SCORM will show the student how to do the
lesson. Shared memory will be used for sending and receiving the data with CBT or ImT
19
using ActiveX. Between SCORM API and the web page, JavaScript will be used, and
ActiveX is used for interaction.
The recording of data during training sessions is essential for providing feedback to
the trainees, scoring the trainees, and for certification. Methods for creating, recording,
and storing “script” data will be created through SCORM. These scripts will record the
motions, interactions, and actions/events in a condensed manner. Only the data that
results in events and consequences will be recorded in order to minimize data storage
while ensuring that valuable process steps are recorded. To record the events of training
courses, the SCORM application will be used for tracking students [31].
Figure 3: SCORM web page
A protocol is a communication method between two different systems. The
simulation of protocols presents a different challenge. Methods will be created to allow a
trainee to walk through a protocol, will provide guidance where necessary, and will
capture mistakes during a protocol. Mistakes captured during the simulation will be
communicated directly to students. Figure 4 shows the overall scenario of iTrain. Both
20
CBT and ImT are connected with SCORM API, so that applications send and receive
feedback while students perform a training course.
Figure 4: Overall scenario of iTrain
4.5 Unified Approach
Both CBT and ImT have some commonalities. A database will be used as a common
repository for models and properties of those models. The details of a database that will
support the approach will be designed. For example, if a model has been changed in ImT,
the model should be updated in CBT via a database. Therefore, both CBT and ImT have
the same models, which the trainee will use to learn and practice in iTrain.
21
iTrain will use the methodology discussed in this chapter in order to integrate digital
applications with an LMS. A unified approach will be used to combine CBT, ImT, and
LMS. A common repository will be created to bring all applications together.
22
CHAPTER FIVE COMPUTER-BASED TRAINING METHODOLOGY AND
DESIGN
5.1 Overview This chapter describes the methodology, design, and implementation of a
computer-based training (CBT) application. The CBT application is one of the key
systems in iTrain. CBT uses a common personal computer to instruct trainees. CBT
does not require having any other hardware such as 3D mouse, glasses, glove,
trackers, and so on. There are many options for instructing trainees using a personal
computer.
Figure 5 shows an overview of the proposed system in CBT. Detailed
descriptions about LMS will be provided in Chapter 7.
The featured requirements for the CBT application are as follows:
Ability to load 3D geometric models
Ability to easily navigate the models
Ability to interact with a mouse and a keyboard
Ability to select/deselect 3D models
Ability to practice a certain sequence
Ability to assemble/disassemble a component
Ability to communicate with a learning management system
23
Figure 5: Overview of CBT
The purpose of this chapter is to describe the following:
methodology of the CBT application
design of the application
algorithms in the CBT application
implementation of the CBT application
supporting components for the CBT application
5.2 Computer-Based Training Methodology
A new application was developed for Computer-Based Training since the
requirement was that a user should be able to not only watch a simulation of a training
course, but also to interact with the new application. As mentioned in the previous
chapter, the CBT application focuses on performing two different modes: a learning
mode and a practicing mode. The learning mode demonstrates a simulation to the trainee,
24
and the trainee will be able to navigate the models in a given lesson through the CBT
application. In addition to the CBT application, various digital materials such as mpeg
files and figures on web pages can be also used in the learning mode. However, the
trainee will not be able to select or assemble a component in this mode. The practicing
mode allows the trainee to interact with the models using a keyboard/mouse. Moreover,
the trainee will get information, such as a name of a part, from the application.
The purpose of the learning mode is for the trainee to be able to look at a pre-
defined scenario that shows a proper simulation. The trainee may watch digital materials,
such as digital videos, digital photos, and simulations in the CBT application. For
example, if a certain disassembly process is defined in a course, some data such as 3D
models, their transformation matrices, and the paths of moving components are illustrated
to show the disassembly process. The application allows the trainee to navigate the
models in this mode, so the trainee is able to experience the process.
In the practicing mode, the trainee will be able to actually interact with the
models. In the CBT application, the trainee will use a keyboard/mouse to control the
action/event. For instance, if the trainee is asked to turn off a switch, the trainee should
click the switch with a mouse in the CBT application. The CBT application allows the
trainee to do the following in this mode:
navigate 3D models
select/deselect by clicking a component
assemble/disassemble a component with a mouse/keyboard by checking
transformation matrices
25
The above capabilities in the CBT application enable the trainee to practice with
digital models before doing a real job. Moreover, the models should be easily prepared
for the CBT application, and make it more convenient for the trainer to create a new
course.
The CBT application needs to communicate with a learning management system
for sharing data such as model names. This functionality allows the trainee to see model
information through a learning management system. The integration between CBT and
LMS will be described in chapter 7.
5.3 Algorithms in the CBT application
Several algorithms in the CBT application were built using OpenSceneGraph
because of powerful functionality it provides. For example, OpenSceneGraph has a
library to get a mouse position in an application. Based on the current mouse position, the
CBT application calculates a viewing transformation matrix. The left mouse button is
used for rotating a view while the middle button is used for the pan. The right mouse
button can be used for zoom in/out. In addition, OpenSceneGraph has a library to select a
component in the CBT application. The functions are inherited from
Producer::KeyboardMouseCallback so that a model can be selected using a mouse.
Specific algorithms were developed to address the following required functionalities:
create a scene graph from the data file
assemble/disassemble a component
5.3.1Creation of a Scene Graph
26
The scene graph that is created in the CBT application needs to be based on the
hierarchy from the CAD system so that assembly/disassembly simulations can be
performed. The format of the data file is shown in Figure 6. A detailed description for
generating the data file will be introduced later in this chapter. Each part or assembly
includes information such as name, assembly/part flag, parent name, and transformation
matrix. The transformation in the data file is a global transformation matrix. Therefore,
when the scene graph is created in the application, a relative transformation needs to be
calculated if the part is in a sub-assembly. Figure 7 shows an example of a scene graph.
The known transformation matrices are [P1] and [G1], as obtained from the CAD system.
To create a scene graph, a transformation matrix, [P2], needs to be calculated. Equation 1
illustrates the equation to calculate a relative transformation matrix. If there are several
levels of sub-assemblies, a local transformation matrix can be recursively calculated by
the same method.
]1[]1[]2[]1[]2[]1[
1 GPPGPP×=
=×−
Eq. 1
27
Figure 6: The format of the data file
Figure 7: A scene graph
28
5.3.2 Support for Assembly/Disassembly Functionality
A functionality that allows assembly/disassembly in the CBT application allows the
trainee to interact and assemble the model. The trainee is able to select a model with a
mouse in CBT. However, the mouse is also used to change the viewpoint. This needs to
be resolved using some algorithms. In order to assemble the model, the trainee is
expecting the model with respect to the trainee’s view point. The CBT application
enables the trainee to change a viewport with the mouse interaction and it has only one
mouse to interact with the model. The model cannot be moving while the viewport is
being changed. Therefore, the current viewport should be disabled by the trainee when
the the part needs to be moved.
When the a toggle key (F9) is pressed with the part selected, the trainee is able to
move the selected part. Instead of calculating a new transformation matrix for moving the
selected part, a previous viewing matrix, (stored when the toggle key is pressed) and a
current viewing matrix (obtained by a current mouse movement) are used to move the
selected part. The equation is shown in Figure 8. In order to assemble/disassemble the
part, positions of x, y, and z are compared between the moving part’s location and its
final location. The part is snapped when it is within a tolerance. These capabilities allow
the trainee to practice the assembly process in CBT by using a keyboard and mouse.
29
Figure 8: Apply a transformation matrix to a selected part
5.4 Design, Description, and Implementation of Classes
The complexity of software requires a method to create an application that is
efficient and flexible. The object-oriented programming (OOP) method is one concept for
developing software that is manageable, expandable, and flexible. Figure 9 shows the
overall OOP design of the CBT application. The application is designed in an object-
oriented method so that it can easily be expanded in the future.
Figure 9: Classes of the CBT application
30
A data file created from the CAD system is read in the DataManager class, and this
information is used to create a scene graph in the NodeManager class. The scene graph
has a hierarchy which is the same assembly hierarchy as in the CAD system. The
InputManager class handles the inputs, such as mouse clicks and keyboard inputs. The
input data is used to ascertain the current position of the mouse and to send information
to the NodeManager class. In the NodeManager class, the current mouse position is being
checked to determine whether or not it collides with a component. A user interacts with
the InteractionManager class with a keyboard/mouse. Inputs from the user enable
selection/deselection of a component, and these choices also guide the user to drag and
drop a selected component.
The CBT application consists of several classes. Some of the key classes are
described here.
NodeManager Class
The NodeManager class is a class for reading geometry files and creating a scene
graph. Models are loaded in the CBT application using the osgDB::readNodeFile
function from OpenSceneGraph. When the geometry models are loaded, the data file, as
shown in Figure 6, is also read in the CBT application. This data file contains the
assembly hierarchy information, such as parent names. Based on the assembly hierarchy,
the components give the parent/child relationships. Figure 10 shows a snapshot of the
CBT application, which consists of 12 parts loaded.
A basic node of the scene graph is shown in Figure 11. Each node in the scene
graph requires having osg::Node, osg::Transform, and osg::MatrixTransform so
each node has its own transformation matrix attached. A scene graph enables
31
developers to manage an assembly hierarchy because each component has a
parent/child relationship with other components so the CBT application can create
the same assembly hierarchy as the CAD system with the scene graph.
Figure 10: A snapshot of the CBT application
32
Figure 11: A basic node of scene graph
InputManager Class
This class handles mouse and keyboard interaction. The mouse interaction in
OpenSceneGraph enables developers to navigate models and select a component. When
the mouse button is clicked, the intersection between a current mouse position and a
component can be obtained by using osgUtil::IntersectVisitor::HitList in
OpenSceneGraph. When the component is selected, it will be highlighted. The name of
the selected component is retrieved to give information to the learning management
system.
33
AssemblyManager Class
One of the key roles of the AssemblyManager class is to check transformation
matrices between a current selected part and its designated final location. A component is
highlighted when the component is selected in the CBT application (Figure 12). When
the component is selected, the trainee has an option to move it. As described in 5.4.2, the
trainee is able to move the component in the CBT application. The trainee can
disassemble the part and re-assemble the part. When the trainee tries to assemble the part,
a distance between the current selected part and its final position is calculated. If the
distance is close enough, the current selected part will be snapped into the final position.
Figure 12: Highlighting a component in the CBT application
5.5 Model Processing Approach The CBT application needs to have a pre-process step for preparing models
and data file. This section shows how to create 3D geometry models and data file.
A CAD system can create 3D models, and it also allows us to export a neutral
file format such as VRML, STL, and IGES. More realistic models, with proper colors
34
can be obtained through commercial software, such as 3D Max Studio. A pre-
processing system is implemented to support the CBT application. The following sub
sections show how to prepare 3D geometry models and extract relevant information
from the CAD system.
5.5.1 3D Geometric Models
Geometric models are created in order to allow a clear view of a new product.
CAD systems such as CATIA and Pro/ENGINEER are some of the most common
software programs that can be used to create digital models in manufacturing industries.
However, native CAD models cannot be directly loaded into other stand-alone
applications; the CAD models can only be read in the CAD systems in which they were
created. In order to load 3D geometric models into the CBT application, the CAD models
need to be exported into a different format that can be loaded in the CBT application. The
CAD system, CATIA, allows export of several formats, such as IV, VRML, IGES, and
STL. An STL model format has been chosen for this study, because it is easy to take the
STL files and convert into OpenSceneGraph formats in 3D MAX Studio.
3D Max Studio is a powerful software program for rendering 3D models. It has
the following capabilities:
It can import several file formats, such as 3D Studio Mesh (.3DS, .PRJ),
VRML(.wrl), AutoCAD Drawing (.DWG, .DXF), StereoLitho (.STL).
It can also export other 3D file formats such as 3DS, STL, VRML, and IGES.
It has an API through 3D Max script.
OpenSceneGraph can load various 3D geometry models. In addition,
OpenSceneGraph can load native OpenSceneGraph formats (.osg, .IVE) and it provides
35
an extension of 3D Max [46,48]. Using this extension, 3D MAX Studio can export
models as the OSG format. 3D MAX Studio then can change a color of models and
export geometry models with colors in the OSG format. The OpenSceneGraph
application is designed to read both 3DS and OSG formats. It should be noted that the
color of 3DS files cannot easily be changed in the OpenSceneGraph application; all
models have the same color in the application. However, the color of OSG files can be
modified in 3D Max Studio, which makes it possible to create a realistic model in the
OpenSceneGraph application.
5.5.2 Data Extraction
OpenSceneGraph enables the developer to load the geometry models in the
application, but all components must be placed into proper position. A specific
transformation matrix should be applied to each component in order to achieve the
same model location as in the CAD system. To obtain transformation matrices from
the CAD system, a CAD API application has been developed. The API application
consists of various functions, which are shown in Figure 13. The
“ExtractAssemblyTree” function is a recursive function, which traverses an
assembly hierarchy to retrieve a name and a transformation matrix of each
component. In addition, a name of a parent and an assembly flag are stored in the
data file.
36
Figure 13: CATIA V5 pseudo code
The format of the data file is shown in Figure 6. The output from the CAD API
application has been used in the CBT application. Appendix A includes the code for this
API.
37
CHAPTER SIX IMMERSIVE TRAINING METHODOLOGY AND DESIGN
6.1 Overview
The ImT requires state-of-the-art technologies such as a 3D stereo system, helmet,
glove, and a tracking device. In this dissertation, two ImT applications were enhanced
from their original functionality so that it would support the training requirements..
Depending on the training purpose, the appropriate ImT application should be chosen.
The first ImT application will allow sophisticated assembly simulation and training
in an immersive environment. Two industry pilot studies were used to refine the training
process and demonstrate the applicability of this environment for training.
The second ImT application allows tool interactions within a CAD system for
assembly training. This ImT application will be integrated with a learning management
system and this ImT application can be used mostly in a practice mode, allowing trainees
to practice a lesson. For instance, an assembly that consists of several bolts and nuts
needs to be unscrewed in a specific order. With ImT, trainees can go through the process
of screwing on the bolts in a realistic manner.
The immersive training application should communicate with a learning
management system similar to how the CBT application communicates. Figure 14 shows
the overall architecture. The trainee is able to interact with the models using VR hardware
such as a glove, a tracking device, and a passive stereo screen.
38
Figure 14: Overall system of ImT
The featured requirements for the ImT applications are as follows:
Ability to load 3D geometric models
Ability to easily navigate the models
Ability to interact with VR hardware such as glove and a tracking device
Ability to use a tool such as a wrench to unscrew a bolt
Ability to communicate with a learning management system
Ability to practice a certain sequence
Ability to assemble/disassemble a component with the glove
6.2 Immersive Application - I
The first immersive application developed and utilized for training is a stand-alone
immersive application, VADE. VADE (Virtual Assembly Design Environment) has been
developed at Washington State University. and is one of the first of its kind, with the first
functional prototype being available in 1995 [49], and is now a fairly mature and well-
39
known tool. There are several papers with details of VADE [50,51]. However, a brief
overview will be provided here to support the relevance to our training work.
VADE includes several key capabilities of an immersive virtual reality system. It
supports stereoscopic viewing, tracking, and user interaction. VADE supports two-
handed assembly. When the user selects a part, the constraints belonging to that part are
loaded. These are visually represented by planes and axes on the component that was
picked and the corresponding component to which it needs to be assembled. When the
user matches a plane or axis on each of the components, the part becomes constrained,
and will no longer move in the direction associated with that plane or axis. When a part is
partially assembled on the base part and then released, a combination of collision
detection, gravity-based motion, and constraint-based motion is used to simulate realistic
motion of the part [51]. This allows the simulation of sliding, swinging, and also sliding
and swinging simultaneously. Parts follow free projectile motion when dropped by hand,
thrown by hand, or when they are free to fall without other parts or environment objects
obstructing their path. If the user should let go of the part, gravity will take over and
move the part along any unconstrained planes or axes in accordance with the laws of
physics. Although the software does not use collision detection in performing assemblies,
collision detection is available to aid in identifying interference problems or incorrect
assembly paths.
The VADE software has the ability of extracting the assembly information from the
CAD model. This information is then used to build a constraint database, which is then
utilized in real-time while running the VADE program. The surrounding environment can
be created using any user-preferred method. For example, any CAD system can be used
40
to model the geometry of the environment and export the triangulated model. Texture
maps can be defined for these environment objects. VADE allows limited parametric
modification of parts while in the virtual environment. These parametric changes are
communicated back to the CAD model, updated in the CAD model and the geometry sent
back to VADE in near real time. For larger parts that cannot be assembled with just two
hands, a crane allows users to simulate the manipulation [52]. Parts can be attached to the
crane by specifying one or more attachment points interactively. Pendulum equations
simulate the motions of parts being manipulated by a crane. Algorithms have been
implemented to allow a person to push a part to turn it while it is attached to the crane.
Some of this this functionality was developed a part of this dissertation.
6.2.1 Methodology for Training Sessions in ImT
In order to use an immersive application in training, the methodology used was as
follows:
Information gathering and analysis of the real assembly process - Study the provided
video of the real life assembly, visit the sites to observe the real-life assembly situation,
and analyze assembly process documentation.
Scoping and identification of the specific assemblies to train and the process of
assembly - Discuss the desired simulation with the engineers and managers. Select the
components to be trained from the overall scenario. Identify the process for the assembly.
Model and environment preparation - Obtain CAD models of components to be
simulated, if needed re-organize the as-designed CAD assembly tree to reflect the as-
41
manufactured assembly on the factory floor, and then export the data from the CAD
system to the simulation application. Prepare the environment.
Replication of assembly process in the simulation environment - Set up the initial
location of the components in the environment. Perform the assembly using the
immersive virtual assembly environment and go through the scenario.
Practice mode of training lesson – The trainee interacts with the models in VADE
and practices the sequence. Gather problems and issues and discuss lessons learnt during
practicing the lesson.
6.2.2 Training on Assembly of a Large Press Machine
This training session involved assembling the components of a large press machine
for a company that manufactures construction, mining, and heavy industrial machinery.
In many cases, because of size and transportation issues, several stages of final assembly
are performed at the customer site. Thus, in addition to fundamental assembly planning
issues, in order to be adequately prepared, the unique environments and constraints of
each customer site also need to be considered on a case by case basis. It is not enough to
have a generic environment in which the assembly planning is performed and checked.
We have to train and show the customer how to assemble the models at a new
environment.
The usage of an immersive assembly environment for training in this scenario is
expected as follows. Manufacturing engineers would create a tentative assembly
sequence for the training and the customer site environment in VADE. They would do a
preliminary evaluation and identify potential problem areas. During the meeting with the
various personnel, VADE would be used to perform the assembly simulation for training
42
with special emphasis on the specific junctures in the sequence that seem to cause
problems. Using VADE, engineers can be trained to assemble a component before they
assemble the component in the real environment.
The overall methodology described in the previous section was used. The key steps
are described in detail here.
Information gathering and analysis of the real assembly process
The video provided by the company and the assembly process documents were
analyzed to obtain a reasonable assessment of the assembly scenario for training. The
consultations with the engineers provided additional assistance. The immense size and
complexity of the large press machine (Figure 15) made it a challenge to map the
components in the video to the components in the Pro/E models provided. It was also
difficult to assess the human interaction with the objects. Numerous personnel were
working in conjunction to assemble the components together. Their individual
interactions with the objects were sometimes hard to identify, due to the fact that some of
the workers were hidden behind the very large components. A summary of the training
actions that were observed are listed (Appendix B).
43
Figure 15: Relative size of operator large press machine
Scoping and identification of the specific assemblies to simulate
The assembly provided was composed of a large number of sub-assemblies in
Pro/Engineer (Figure 16). Specific sub-assemblies and components were selected for
testing, with eight identified as the key components in the assembly training.
44
Figure 16: Exploded Pro/E assembly model X31-00-21440 of large press machine
Model and environment preparation
The CAD to VADE extraction module (Pro/VADE) was used to extract the assembly
hierarchy, assembly constraints, and polygonal models of all components from Pro/E.
The assembly hierarchy from Pro/E had one main assembly (x31-00-21440), which
contained several sub-assemblies and individual parts at the top level of the assembly.
The left half of Figure 17 shows the assembly tree reported by Pro/E. The assembly
process provided by the engineer indicated that the sequence needed to be re-organized
before being used in VADE. This was a classic case of the as-designed assembly
hierarchy being different from the as-manufactured hierarchy. The sequence was re-
ordered using the tree re-organizer [53]. The resulting sequence is shown in the right half
45
of Figure 17. Note that some sub-assemblies and parts have been removed from the list
shown because of the large tree size. In the VADE assembly hierarchy, any assembly past
level 2 is pre-assembled for use in VADE.
Figure 17: Comparison between Pro/E assembly hierarchy and VADE assembly
hierarchy
A factory environment that replicated the physical factory was created. Architectural
drawings and pictures of the actual factory were used to quickly create the relevant
geometry such as walls, floor, pit, and ceiling of the factory using Pro/Engineer. The
model was then exported to VADE as triangulated data. The environment was texture
mapped with images from the actual factory. A crane was also a part of the environment
since the human/crane interaction is vital in predicting problem areas and object
movement (Figure 18). The components were located correctly to show the initial
position at the start of the simulation.
46
Figure 18: Environment modeling
Replication of assembly process in the simulation environment
After all the components were located properly and the environment was finalized,
the assembly process for training in the video was simulated in VADE. A training
sequence for the assembly process followed by the operators in the factory was as
follows (Table 1):
47
Table 1: Assembly steps in real and virtual environments
Practice mode of training
VADE successfully completed the segment of training for the assembly of the large
press machine that we set out to perform in the immersive virtual assembly environment.
Overall it was concluded that the virtual assembly environment had sufficient modeling
and interaction realism to be of considerable value for evaluation and training.
In Real Factory In VADE
Step I
Part 92a-00-10930 was moved from the cinder blocks to the pit.
Part 92a-00-10930’s initial position was located in the pit (since this was the base part in VADE)
Step II
Crane and human manipulation were used to attach parts x31-00-23130 (4x) and 92a-00-10920 to part 92a-00-10930
Each of the four x31-00-23130 parts were attached to the crane using four hanging points inserted at the end of the column (Fig 3)
Step III
Part x31-00-21450 was picked up with the crane with eight hanging points and moved in an L-shaped path to its final position. The part was then lowered and attached to part x31-00-23130.
Part x31-00-21450 was picked up with the crane with four hanging points and moved in an L-shaped path to its final position. The part was then lowered and attached to part x31-00-23130
48
6.2.3 Training on Assembly of the Fifth Wheel to a Truck Chassis
This training session involved the assembly of a forward cab support of a truck
chassis for a multinational technology company that manufactures premium commercial
vehicles. The focus of the training lesson was on assembling the fifth wheel to a truck
chassis.
Information gathering and analysis of the real assembly
process The first step was to understand the existing assembly process thoroughly from the
information collected (video, snapshots, site visits, and discussions with engineers) and
the informational documents provided by the industry partner. The parts for simulation
were identified. The Pro/E models were studied and compared with the models observed
on the assembly line and in the video and snapshots.
Model and environment preparation
The assembly was modified to include only the relevant models. The original model
provided had over 350 parts. As it was not possible (and not required) to simulate all
these parts, the number of parts were pared down using the reorganization tool described
earlier. All relevant information was exported from the CAD model.
A roadblock initially arose when it was determined that the models for the fifth
wheel assembly provided by the consortium member were in IGES (Initial Graphics
Exchange Specification) format. Pro/E was unable to convert this imported IGES part to
the inventor file format to be used in VADE because of gaps and holes in the model due
49
to the IGES translation. This required that the IGES file be imported and the gaps and
holes fixed before the model could be added to the Pro/E assembly.
In addition, an environment had to be created in order to accommodate this assembly
scenario. The snapshots taken at the industry partner’s site were used to create texture
maps. The walls, floor and ceiling were created as thin protrusions in Pro/E, exported to
VADE, and then texture mapped to convey the look of a real factory environment (Figure
19). Bins, trolleys, and other components of the environment were also created. The time
to create this custom environment was about 20 hours, yet we believe that a user,
knowledgeable and experienced in this field, could create the environment in about 5 to 7
hours. This environment can also be reused for multiple assembly scenarios.
Before simulating the assembly process for training, all components had to be placed
in the appropriate bins or the locations where they were kept in the factory. Originally,
the initial position of the parts had to be determined through trial and error, and entered
manually into the initial location file. This process was very laborious and time
consuming. However, an “immersive positioning” module of VADE was used to
manipulate the initial position coordinates by moving the parts around in the environment.
The time to place the components in the starting location obviously depends to a large
extent on the number of components. Each part takes a few minutes for a first time user
using the “immersive positioning” module.
50
Figure 19: Custom environment for the fifth wheel assembly
Replication of assembly process in the simulation environment
As observed from the video, the 5th wheel was put into place using the following
sequence:
1. Pick up the 5th wheel out of the shipping crate using the crane.
2. Place the 5th wheel on top of the crate and reattach using different
attachment points.
3. Move and lower the 5th wheel over the assembly position
4. Fasten the bolts at the various locations
This training sequence was followed to the best of the user’s abilities. There was one
specific instance where VADE was not able to mimic the real-world process. When the
5th wheel was lowered onto the top of the crate, the crate supports the 5th wheel as it
pivots and comes to rest on top of the crate. In VADE, the collision detection between the
51
crate and the 5th wheel is possible, but the “cause and effect” of that collision had not yet
been implemented (it has subsequently been implemented). Figure 20a and 20b show the
5th wheel attached to the crane right before the part is to be placed on the frame assembly
in both the real environment and the virtual environment.
Figure 20a: Lowering the fifth wheel with a crane in the real factory
Figure 20b: Lowering the fifth wheel with a crane in VADE
52
After the fifth wheel is placed on the chassis, various bolts are used to fasten the 5th
wheel to the chassis (Figure 21).
Figure 21: Labeled bolt positions
During the visit to the industry partner site, it was discussed that the worker’s posture
needed ergonomic improvement for easier access and efficient assembly. In the factory,
bolts 1 through 4 are assembled from the rear of the truck. Figure 22 shows the 1st person
and 3rd person view of assembling Bolt1 in VADE. Figure 23a and 23b show the
assembly of Bolt 2 in real life and in the virtual environment.
53
Figure 22: Worker inserting bolt 1 in VADE
Figure 23a: Worker inserting bolt 2 in real life b: 1st person and 3rd person views in
VADE
Bolts 5 and 6 are assembled by the worker squatting down and reaching out (Figure
24).
54
Figure 24a: Positioning bolt 5 in a real factory b: Positioning bolt 5 in VADE
Practice mode of training
The immersive application, VADE, was successful in replicating the assembly
scenario for training. As before, a less cumbersome virtual reality setup, better user
interfaces in the environment, and force feedback would have improved the training
simulation. Setting up the models and the environment for the simulation are involved
tasks that need considerable attention. This case study showed that the immersive
application could be used as a training tool in the assembly process.
6.3 Immersive Training Application – II
The focus of the second ImT application is on performing a practice training
situation with a tool interaction. In the ImT application, the trainee uses VR hardware to
perform an action/event. An existing ImT application is used for practicing the training
lessons.
A substantial base of knowledge, technology, and software related to immersive
environments has been developed at WSU. The ImT application developed at WSU uses
55
haptics-enabled virtual tools in a native CAD environment, such as CATIA V5TM [47].
This application has the capability to simulate a native CAD model employing tools such
as allen and box-end wrenches in the CAD system with the interaction of VR hardware
(tracking device, glove, and haptic device). This ImT application has been modified for
incorporation into iTrain. The ImT application is able to manipulate native CAD models,
use a tool to simulate models, and use VR hardware such as a tracking device and glove.
In order to use iTrain, the ImT application connects with SCORM web pages. For
example, when the trainee grips a wrench in an assembly/disassembly process in ImT,
he/she should be able to know whether or not the proper tool is gripped.
Overall Architecture
The CAD system chosen was CATIA V5. The haptic device chosen was
Immersion’s CyberGlove® and CyberGraspTM. Immersion’s Virtual Hand for V5
(VH4V5) was chosen as the software tool to drive the haptics devices and for the primary
integration with the CAD system. The overall architecture of the work described in this
dissertation is shown in Figure 25. Some of this work is combined work with another
project [54].
56
Figure 25: Overall architecture of ImT [47]
The VH4V5 is an add-in that was developed to interface the CyberGraspTM and
CyberforceTM haptics devices with the CyberGlove® for use in the CATIA V5
environment. It enables the user to control a virtual hand inside the CAD system, thereby
facilitating assembly manipulations. The VH4V5 Suite comprises two products.
"VirtualHand for V5-Design" enables engineers throughout the enterprise to manipulate
their 3D graphical designs while wearing an Immersion CyberGlove® hand-sensing
glove. Users wearing a CyberTouch™ hand-sensing glove additionally feel tactile
feedback sensations. VirtualHand for V5-DMU, together with Immersion's CyberForce®
system, enhances the V5 immersive digital mockup environment by providing advanced
hand and arm force feedback for more realistic design reviews.
57
The entire system was designed as an integration of the CATIA, VH4V5, and the
WSU Tools modules. The CATIA-VH4V5 integration was already in place through the
commercial product from Immersion. However, because there was no API available for
VH4V5, several modifications were made to the underlying VH4V5 software to facilitate
this project. VH4V5 uses the API of CATIA called CAA to extract the tessellated
geometry from the CAD model. This is then used to provide the graphics display to the
user directly through CATIA. VH4V5 also uses the assembly and kinematic constraints
in the CATIA model to constrain the relative motions of parts and provide haptic
feedback.
The WSU Virtual Tools plug-in was designed to integrate with both CATIA and
VH4V5. This plug-in obtains kinematic information and assembly from the CATIA
model to extract tool-part interaction details and links with VH4V5 through a callback
mechanism. The plug-in also modifies/updates the assembly model and the kinematics
and interaction models as needed during the process of assembly/disassembly using tools.
Thus, the goal of the plug-in was to maximize the use of existing capabilities to create a
new capability through the use of APIs (CAA) and direct integration.
Using Kinematics and Assembly Constraints
The CATIA kinematics workbench provided important functionality. The automatic
conversion of assembly constraints to kinematic joints as well as the ability to create
screw joints proved to be very useful. VH4V5 also uses the kinematic joints available in
CATIA in its interaction model.
58
We decided to use axial, planar, and screw constraints to achieve the required tools
capability. A screw joint between the fastener and the part allows the specification of
pitch and axis of motion for the threaded fastener. Planar constraints between the bottom
face of the head of the fastener and the top face of the part allow the system to define
when a fastener is completely fastened.
Tools such as a wrench and an allen wrench also need to be added to the assembly in
the CAD system for inclusion in the graphics and the manipulation processes. However,
the tool needs to be initially free to move in the assembly as manipulated by the virtual
hand and later be engaged on a fastener. Thus, the tool should not have any initial
constraints, but should be constrained completely to the fastener when engaged. This
will allow the screw joint of the fastener to also be applied to the tool, thus simulating a
tightening or loosening activity.
These assembly constraints and kinematic constraints can be defined for the parts,
tools, and fasteners prior to simulation. Details of the modeling of the parts and
assemblies and the simulation approaches are provided below.
Several methods were devised and tested for managing the constraints and assembly
models for this simulation activity.
Method 1: Dynamically re-organize the assembly tree
The first possible approach was to reorganize the assembly hierarchy dynamically
during the simulation process depending on whether the tool was to move independently
or to move with the fastener. Since VH4V5 treats a subassembly as a rigid component,
if the wrench (tool) and the fastener are in the same subassembly, the two will be
constrained to move together. Thus, if the screw joint were applied to the fastener, the
59
tool and fastener would automatically be manipulated simultaneously. Figure 26 shows
the initial and reorganized assembly in this approach.
We decided to create the Virtual Tools plug-in as a separate CAA program to
reorganize the assembly hierarchy tree inside CATIA. This reorganization method
allowed us to use the VH4V5 software to manipulate a tool inside CATIA (Stage I in
Figure 26). When the tool comes within a certain distance of the fastener, it becomes a
child of the bolt assembly, allowing concurrent rotation of the wrench and bolt (Stage II
in Figure 26). To achieve the above-mentioned goal, it was necessary to be able to
establish a rigid joint between the wrench and the bolt when they were engaged.
Since VH4V5 treated a sub-assembly as a single, rigid body and did not allow any
relative motion between its components, it was required that the wrench be in the top
level assembly when not engaged with a bolt so that the user could move it freely and
align it with the bolt.
Stage I
60
Stage II
Figure 26: Reorganizing assembly hierarchy [47]
Implementation of the above discussed solution involved development of a CATIA
Workbench “add-in” for manipulating the assembly hierarchy. This add-in was activated
using a simple toolbar button created inside the CATIA assembly workbench. The basic
function of the toolbar button was to toggle the assembly structure between the two
hierarchies depicted in Figure 26. The user could pick up a tool and bring it close to any
bolt for simulation. At this point the user would have to interrupt the VirtualHand for V5
simulation loop by stopping the manipulation. The user would then initiate the assembly
restructuring code by pressing the add-in toolbar button.
The assembly-restructuring algorithm finds the closest edge on the bolt and adds the
tool to the bolt sub-assembly. Assembly constraints are created so that the tool is
correctly located with respect to the bolt. The user can then re-start the VH4V5
manipulation process, which reloads the assembly and its scene graph with the updated
assembly hierarchy and constraints, enabling the user to rotate the tool to tighten or
loosen the bolt. Pressing the add-in button after stopping the manipulation loop can then
61
disengage the tool. The user can follow this sequence as many times as necessary to fully
tighten or loosen all of the active bolts in the assembly.
Though the approach discussed above provided a workable solution, it failed to
provide a smooth, realistic simulation in a manner envisioned by the researchers. The
approach involved manual steps (button clicks) for triggering the assembly hierarchy
restructuring. To overcome this shortcoming, a Spy++ program was created to
automatically switch workbenches and press the required buttons when initiated. The
process then improved to where the distance between the tool and the fastener would
automatically trigger the Spy++ program to press all the necessary buttons. However, the
process was still unacceptable because of the time lag involved in reloading the re-
organized assembly into the VH4V5 simulation process.
Due to these limitations, this approach was found unsuitable, and was eventually
abandoned.
Method 2: Dynamically create constraints
We realized that assembly constraints could be used to constrain the tool and fastener
in such a manner that they would behave as a sub-assembly without necessarily changing
the assembly hierarchy. In this second method, the plug-in would assign new constraints
to the hierarchy tree, thus negating the need for reloading the polygonal models of the
model. This would significantly reduce the time required to attach the tool to the
fastener.
In an attempt to remove the need to click on any button to trigger the constraining
activity, a callback mechanism was implemented in VH4V5. This callback function was
called from VH4V5 during every frame of the simulation. The callback function provided
62
a quaternion vector representing the transformation and the name of the grasped object as
arguments. Knowing the name of the grasped part and its position allowed the Virtual
Tools plug-in to determine when the tool was being manipulated, and also to check if the
tool was close to a fastener for engagement.
The Virtual Tools plug-in used CAA function calls to determine the location of the
fasteners and to compare that with the location of the tool from the callback and
determine when to constrain the two together. Three constraints needed to be added to
fully constrain the tool to the fastener. For the bolts implemented in this phase of this
research, there were six possible orientations of the tool engagement on the fastener. Thus,
the proximity of the engaging face on the tool was compared with all possible faces, and
the closest one was used for the constraint.
Ultimately, this method was also abandoned for a number of reasons. VH4V5 was
unable to add new constraints once the model was loaded. Furthermore, the CAA
function calls for applying constraints that were proved to be unstable when invoked
during the simulation as part of the callback process. Thus, a third method was devised
to overcome these limitations.
Method 3: Dynamically activate constraints
The third method was a variation of the second method described above. The
callback from VH4V5 was modified to accept a list of constraints to activate and
deactivate. The assembly of the part, fastener, and tool was modified to include all
possible constraints (in a deactivated state). Thus, for a wrench and one bolt, there were
two constraints that placed the wrench at the right distance from the center of the bolt and
with the top of the wrench aligned with the top face of the bolt head. The third constraint
63
was to be chosen from six possible face constraints (for the six possible faces of the hex
bolt). Thus, there were a total of 8 constraints needed for one wrench and bolt.
During the simulation process, instead of creating new constraints to engage the tool,
the Virtual Tools plug-in “activated” the appropriate set of constraints and deactivated
them as needed. This method proved to be the right choice for this prototype since the
user could then use the tool on any bolt at any time and move from one bolt to another as
needed. The constraints between the fastener and the tool were automatically deactivated
once the fastener was outside the part.
Managing parts held by fastener
One of the key functionalities required was the ability to remove parts once the
fasteners holding the parts were removed. This was managed by maintaining a list of
parts held by any fastener and a list of fasteners holding any part. As each fastener was
removed, the state of the parts held by the recently removed fastener was checked to
evaluate if the part could be freed up. Once all the fasteners associated with a part were
removed, all assembly constraints of that part were deactivated to allow the user to freely
disassemble the part.
Guidelines and Requirements for modeling the tools and assemblies
The first step in the modeling process was to create the tools in CATIA. It was
decided to start with a regular wrench as the first tool. A basic wrench was created which
was then scaled to fit a variety of bolt sizes based on the application. A family of bolt
models was also created in CATIA using standard sizes for the bolt head size, length,
pitch, and major diameter.
64
The original model included three hex bolts and three smaller allen bolts. Due to the
similarities between a hex bolt and an allen bolt, it was possible to use the same algorithm
for an allen wrench. It simply involved creating similar constraints on the existing allen
bolt model. Next, an allen wrench was modeled in CATIA based on the allen bolt
dimensions. Datum planes were created and named according to the convention used for
the hex wrench. This configuration expanded the capabilities of the software to allow the
use of two separate tools, with the limitation that only one tool can be used at a time.
To achieve the creation of dynamic constraints a set of datum planes and datum
points was needed for reference. A hex bolt has six faces, so opposite faces were named
FACE A1 and A2, B1 and B2, and C1 and C2. Planes and a datum point were also
created tangent to each bolt edge. Finally, a datum plane was created on the top surface of
the bolt for height alignment with the wrench (Figure 27). These planes were used as
references for the constraints, enabling a rigid connection between the bolt and wrench.
The screw joint was applied to all the bolts.
65
Figure 27: Constraints for engaging tool and fastener [47]
Managing events during simulation
The simulation of assembly manipulation is rather complex. There are many
different states and corresponding event handlers. This arose primarily from the fact that
in one callback only one CAA task could be performed to avoid instabilities in the
integrated system. To solve this problem, we designed and implemented a finite state
logic.
First, we assigned different stages to indicate different states of the virtual hand:
Initial stage: virtual hand is empty.
Tool is grasped.
Rotating the Tool/bolt assembly.
Stage Number
Stage Name Event Handler
0 EmptyHand, WSUCallback() 1 ToolInHand ToolInHandHandler() 2 RotatingBolt RotatingBoltHandler()
66
For each stage, we designed different states, each with different actions to be
executed. These states include:
Stage 0:
Program will check if the tool is grasped; if so, it will set the stage number to 1.
Stage 1:
The states in stage 1 are as follows:
State Nos.
State Name Action
0 StartState If the wrench is close to the bolt, go to State 1. Otherwise keep the current state.
1 StopManipulation
Call function stopManipulation () from Immersion API. Set the next state to be SaveFile1.
2 SaveFile1 Call function SaveProduct () from Immersion API. Set the next state to be ApplyConstraints
3 ApplyConstraints
Call function applyConstraints() to apply all the constraints to the parts in the assembly. Set next state to be StartManipulation
4 StartManipulation
Call function StartManipulation() from Immersion API. Set the stage to RotatingBolt and the state to StartState
Stage 2:
Next, Stage 2 starts with state = 0
State Nos. State Name 0 StartState Check if the distance
between bolt and tool is too large, set next state to be StopManipulationToSaveFile1 and set stage to be EmptyHand if it is, otherwise check if the bolt is out of the hole, set next state to be StopManipulationToSaveFile
67
2 and set stage to be EmptyHand if it is, otherwise keep the current state
1 StopManipulation Call function stopManipulation () from immersion API, and set the next state to be SaveFile1
2 SaveFile1 Call function SaveProduct () to apply all the constraints to the parts in the assembly, and set next state to be RemoveConstraintsForTool
3 RemoveConstraintsForTool Call function removeConstraintsForTool() to remove all the constraints of wrench, and set next state to be StartManipulation
4 SaveFile2 Call function SaveProduct() from immersion API to save the current status of assembly, and set next state to be RemoveConstraintsForBoth
5 RemoveConstraintsForBoth Call function removeConstraintsForTool() and function removeConstraintsForBolt() to remove all the constraints of wrench and bolt, and set next state to be StartManipulation
6 StartManipulation Call function StartManipulation() from immersion API, and set the stage and state to be EmptyHand and StartState
The entire flow of the program was designed to follow the stages as stated above. We
used event handlers to check the current stages and states and to decide the next stages
and states. For example, during stage 3, when the user is rotating the bolt using a wrench
in the virtual hand, if it is found that the bolt is out of the hole, the system will know that
68
this stage is completed and return to stage 1. Figure 28, Figure 29, and Figure 30 show
the detailed flowcharts of the event handling.
Figure 28: Overall flowchart of the event handling [47]
Figure 29: Flowchart of event handling for ToolInHand stage [47]
69
Figure 30: Flowchart of event handling for rotating bolt stage [47]
6.3.1 Enhancement of the ImT Application
The IMT application was further enhanced to support training applications. Figure 31
shows that a wrench is gripped in the ImT application. The environment is the CAD
system but manipulation with the VR devices is possible because of the architecture and
implementation described above.
70
Figure 31: A wrench is gripped in ImT
71
CHAPTER SEVEN LEARNING MANAGEMENT SYSTEM METHODOLOGY
AND DESIGN
7.1 Overview
The purpose of this chapter is to discuss the methodology, design, and
implementation for a learning management system (LMS) of the client site. The LMS is a
server side of the management system, which enables the management and delivery of
online content to trainees. The CBT and ImT applications have been introduced in the
previous chapters. These applications might be used as a stand-alone application.
However, for training purposes, these applications should be tracked and managed by the
LMS. The web-based training courses require communication with the LMS.
Unfortunately, there is currently no standard format for the LMS. One of the most
popular formats, Sharable Content Object Reference Model (SCORM) [31], is widely
used to create web-based training courses. SCORM is used for communicating between
the training web pages and the LMS.
The goal of this chapter is to show the following:
a methodology for the LMS of the client
the design for the LMS of the client
implementation of SCORM
integration of the client for the LMS with digital applications
72
7.2 Methodology
An important key feature in training applications is management of the training
courses. The training applications need to be managed by the trainer. The trainer should
manage, track, and evaluate the trainee in a proper manner through a web-based course.
The trainee should go through a certain course and take a test at the end of the course.
Depending on the depth of the trainee’s knowledge, the trainee might not need to
work on all available courses. From a pre-quiz, the trainee is evaluated and guided into a
proper course, as determined by the trainer. For example, for a mathematics training
course that has 5 levels, the trainee would take a test at the beginning of the course. If the
trainee passes the first two questions related to the first two levels, the trainee would not
need to take first two levels of the course. The trainee would not waste his/her time to
complete sessions which he/she already has knowledge about.
Hardware availability is another factor to determine a session in the training course.
ImT requires hardware such as a helmet, a glove, a projecting screen, and a tracking
device. If the trainee does not have access to such devices, the trainee cannot perform the
ImT. CBT; however, even without access to these specialized devices, most trainees will
be able to participate in a training course because it uses common input devices such as a
keyboard and a mouse.
Tests created by the trainer are provided to the trainee after taking the course. The
trainee will be evaluated based on a variety of questions that are related to the training
course. If the trainee fails the course, the trainee may need to repeat the course. In
addition, the trainee needs to log in to take a course, which allows the trainer to keep
track of the trainee’s progress.
73
7.3 Design of the SCORM Web Pages
SCORM has been chosen to communicate with the LMS, and web pages provide
instructions. An overall design of the training course is shown in Figure 32. Depending
on the course, the contents (lessons) have to be changed, but the overall design of iTrain
should include a pre-test, a learning mode, a practicing mode in CBT, a practicing mode
(ImT), and a test. The lessons are developed using HyperText Markup Language (HTML)
and JavaScript. HTML is mainly used for creating instructional web pages.
Figure 32: Design of the training course
7.4 Implementation of the SCORM Web Pages
A method that bridges the gap between the training web pages and an LMS is
required for managing the training course. SCORM API has been selected for this
purpose. Figure 33 shows the relationship between the web page (client of LMS) and
LMS. The web-based training pages are connected with the LMS using SCORM API.
74
Since SCORM provides an API via JavaScript, the web pages have been implemented in
JavaScript.
Figure 33: A relationship between the web page and LMS
The SCORM Run-Time Environment (RTE) is provided by Advanced Distributed
Learning [31]. Figure 34 shows a SCORM conceptual Run-Time Environment. The LMS
is on the server side, and the SCORM web pages are on the client side.
75
Figure 34: SCORM Conceptual Run-Time Environment [31]
The SCORM web pages can be either an asset type or a SCO type. An asset type of
the web page does not communicate with the LMS during run-time. It does not call any
SCORM APIs. In contrast to the asset web page, Sharable Content Objects (SCOs) are
able to connect to the LMS, and these SCOs can be traced by SCORM API.
The number of lessons in the learning mode or the practicing mode is determined by
the trainer. The trainer can add more lessons if required. The lessons can be also deleted
if they are not needed.
76
Figure 35: SCORM API, API Instance, API Implementation [31]
In order to use a web page with SCORM API, some functions must be called in each
web page. Figure 35 shows some required functions of SCORM API. The descriptions of
some functions include:
Initialize(): initializes the communication session. This function must be
called in the web page.
Terminate(): terminates the communication session. This function must be
called in the web page.
GetValue(): requests a value from the LMS. This function is optional.
SetValue(): sets a value in the LMS. This function is optional.
Once training courses are created with SCORM APIs in web pages, these web pages
need to be able to run in the LMS. In order to run in LMS, a directory has to be created to
contain the web. The directory has many necessary SCORM related standard data. An
77
important file is a main XML file, imsmanifest.xml, which defines the rules for
sequencing and navigation. Figure 36 shows some data of imsmanifest.xml, including
contents, organization, and resources. For example, one of lessons,‘Open the trunk’, has
the identifier id, which is matched from a ‘resource identifier’. The ‘resource identifier’
links to a physical location of the lesson, and it has information of SCO or Asset.
SCORM contents must have the main XML file (imsmanifest.xml). This file is
created by RELOAD [55]. RELOAD is an open source to create and organize SCORM
contents. The featured capabilities of RELOAD are as follows:
Create SCORM contents
Organize courses and lessons
Add/delete SCOs
Package SCORM contents
78
Figure 36: imsmanifest.xml
79
Figure 37 shows a RELOAD GUI. Using this GUI, the training web pages are
organized to connect the LMS. SCOs and Assets are added using the RELOAD GUI.
Objectives can be added in the SCORM contents, and these objectives are used for
remediation. In the practicing mode, each training page presents an objective to check
whether or not the trainee successfully performs a task. After the training course, the
trainee may go back to the practicing mode and re-do the practicing mode.
Figure 37: RELOAD GUI
Figure 38 shows the remediation of iTrain. If the trainee fails one of the lessons, the
trainee should go back to the failed lesson. The objectives of each lesson (obj_lesson1,
obj_lesson2, and so on) are defined in SCORM web pages using RELOAD.
80
Figure 38: Remediation of the practicing mode
7.5 Integration with Applications
The previous section has shown that the web pages are connected with the LMS. In
this section, the connection between the web pages and digital applications is described.
The first page of the learning mode or the practicing mode of ImT should start with an
application when the first page is loaded. The trainee should then follow the lessons in
each mode. Figure 39 shows the overall design of integration between SCORM web
pages and a digital application. ActiveX is used for starting up the digital application in a
web page. The first page of the instructional SCORM web pages should launch a proper
application, so that the trainee goes through a training course with the application.
81
Figure 39: Integration between SCORM web pages and digital application
When the first page is being loaded, the web pages has to load the application using
<body onLoad="FirstLoad(101)" onunload="return Unload()">. A function
FirstLoad(101) is called and the function is in a separate JavaScript file.
Figure 40 shows the FirstLoad function. This function takes one argument, which is
a current step. When the FirstLoad function is called, three different functions are called
from an ActiveX program. Descriptions of these three functions are as follows:
DWXConnector1.Connect(): creates a shared memory and connects to the shared
memory. In addition, it creates a main thread.
DWXConnector1.WriteSendData(lesson): takes a variable, a current lesson, and
writes the current lesson into the shared memory.
82
DWXConnector1.RunProcess(Executable of the application): takes a string
variable, the executable of the application, and this function starts up the
application.
Figure 40: FirstLaunch function
When the trainee has completed a lesson, he/she will go to the next lesson by
clicking a continue button on the web page. When the button is clicked, the web page
calls an unload function. The unload function in the web pages calls Unload() in a
JavaScript file. The Unload() function calls DWXConnector1.Disconnect(), which
terminates the main thread.
In the practicing mode, the web pages are designed to communicate with the digital
application in both directions. In order to receive the data from the application, another
83
function is created using ActiveX. The function, DWXConnector1_DataReceived, is used
for calling a function of ActiveX. The function of ActiveX,
DWXConnector1.GetRecvData(), is called by the JavaScript function
(DWXConnector1_DataReceived). The GetRecvData() function returns a part id, for the
part which is clicked/gripped by the trainee. The unique part id enables the trainee to
know which part is clicked/gripped.
Figure 41: Data structure of the shared memory
Figure 41 shows the data structure of the shared memory. The shared memory is
used for sharing common data between digital applications and the SCORM web pages.
Both CBT and ImT contain similar data to be shared with the SCORM web pages. For
example, the name of the component is written to m_bSendBuff of the shared memory
when the component is clicked/gripped.
84
CHAPTER EIGHT UNIFIED APPROACH
8.1 Overview
iTrain has two main applications: the CBT application and the ImT application. In
the previous chapters, these two applications were integrated with the SCORM web pages
with an LMS. In these applications there are some commonalities and similarities, such as
models, names, and transformation matrices. The unified approach is defined for these
common repositories.
The subsequent sections show the methodology, design, and implementation of the
common repository.
8.2 Methodology of Unified Approach
The two different applications (CBT and ImT) have similar concepts, but different
interaction methods. CBT uses common devices such as a mouse and a keyboard, while
ImT uses VR hardware to interact with the models. However, there are some similarities
in both applications in terms of models, assembly hierarchy and transformation matrices.
For instance, if a model is changed from the source, the model should be updated in CBT
and ImT. These common data should be updated from a single source in order to
synchronize the course in both CBT and ImT. A database is built to obtain information
such as geometry models, paths of models, and transformation matrix of each component.
A database is used as a terminal for data, which can be accessed from web pages,
CBT, and ImT. The database can store information from the CAD system, and distribute
data to digital applications.
85
8.3 Design of Unified Approach
An overall architecture of the unified approach is shown in Figure 42.
1. Data is extracted from the CAD system and inserted into the database so that
both CBT and ImT applications have access.
2. SCORM API is used for communicating between the web page and the LMS.
3. Shared memory is created for sending and receiving the status of the course,
status of assembly, and model information.
4. The web page accesses the database to gain access to model data.
5. CBT and IMT applications retrieve data such as 3D model information,
transformation matrices, and assembly hierarchy.
86
Figure 42: Overall architecture of unified approach
The database design is shown in Figure 43. There are three tables: Course, ModelList,
and ModelProperty. The following is a description of the tables to be supported by the
database.
87
Course: It organizes a series of courses and model lists of each course. Each
course has different model lists.
ModelList: It contains information names and ModelPropertyID of each
model.
ModelProperty: It has information of models such as paths of 3D models,
transformation matrices, and parent names.
Figure 43: Database design
8.4 Implementation of Database
The overall architecture has been shown in Figure 42. First, necessary data should be
extracted from the CAD system. These data should automatically be inserted into the
database. A Dynamic-Linked Library (dll) has been developed for inserting data into the
database. A design of the database is shown in Figure 44. The DBAccess class is a parent
of the course, ModelList, and ModelProperty classes. There are some common functions
of each class, so the inheritance of the object-oriented programming has been used to
88
design the dll. Each table should have its own class so that more tables can be added in
the future.
Figure 44: The design of DLL
The featured functions in the DBAccess function are as follows:
Open(): Open a table from the database
Close(): Close a table from the database after retrieving the data
MoveFirst(): Move to a first record set of a table
MoveLast(): Move to the last record set of a table
MovePrev(): Move to the previous record set
MoveNext(): Move to the next record set
89
The dll has been developed for the database and it can be used in accessing the data
from digital applications and SCORM web pages. The next section will describe how to
access the data from the web page and applications.
8.5 Accessing Database
Once the database is built on the system, either the CBT application or the ImT
application will be able to access the data through the database. The same dll file
developed for inserting data into the database is now used to retrieve the data from the
database. Regardless of the digital application, the same dll file can be used in both
applications.
SCORM web pages also are able to access the database. Since the SCORM web
pages are developed in HTML, the same dll file can not be used for accessing the data.
The single source in this dissertation is the CAD system. The CAD system is widely used
for designing and creating mechanical systems in industry. Once the models are created,
the necessary data are extracted from the CAD system. This method has been described
in Chapter 5.6.2. A data file extracted from the CAD system is shown in Figure 6.
Accesssing the database on the SCORM web page is implemented in ADO and
JavaScript. Figure 45 shows the key functions to access the database. This capability is
important because the trainee is able to know which component is selected. When the
trainee selects or grips a component in either the CBT application or ImT application, a
unique model id is sent to a shared memory. The web page can assess the model id and
retrieve the name from the database. For example, if the trainee selects a wrench (ID: 44)
in the practicing mode of the CBT application, the web page can get the ID from the
shared memory. Then, the web page can access the database. Once a proper name is
90
found in the database, the name is displayed on the web page. Therefore, the trainee
knows whether or not he/she selected a correct component in the practicing mode.
Figure 45: Database access in JavaScript of the web page
8.6 Trainer Perspective
The trainer should prepare a training course with the following processes:
Create training models in the CAD system
Extract information from the CAD models (Chapter 5)
Create instructional web pages
91
Store information extracted above into database (Chapter 8)
Create a scenario for the learning mode in CBT
After the trainer has the material to train the trainee, the trainer should register a
training course in the LMS. In the LMS the trainer should do the following:
Manage (Add, Remove) a user account of the trainee
Import a SCORM content, which is a training course (Figure 46)
Evaluate the trainee after the trainee is done with a training course
8.7 Trainee Perspective
Once the trainer finishes registering the training course, the trainee is able to log into
the LMS. When the trainee logs into the LMS, the trainee will see some courses, which
have been registered by the trainer. The trainee should select one of the courses. The
trainee then follows the training course.
Figure 46 shows the main log in window and administrator options. The trainer
should import courses and add users. Figure 47 shows how to register a course and
available courses. When the trainee selects the course, the trainee is ready to take the
training course. One of the instructional web pages is shown in Figure 48. The first page
starts up the CBT application for the learning mode. This application can be used in the
practicing mode as well if the trainee wants to use the CBT application.
The trainee should read the instructions and click a play button on the web page.
Depending on a current lesson, the CBT application shows a simulation of each lesson.
Once the trainee finishes the learning mode, the trainee will be able to interact with the
models. The trainee can use common input devices such as a mouse and a keyboard in
92
CBT. Based on the lesson, the trainee should select a proper component. If the lesson is
about “Open the trunk”, the trainee is supposed to select the trunk. When the trainee
clicks a component, the trainee can see which component is selected through a text
message on the web page. The text message is coming from the database regarding to the
selection in the CBT application.
In ImT, the trainee will interact with the models using VR devices such as a tracking
device and a glove. The process of ImT is similar to CBT. However, interaction methods
in ImT are different from CBT. Not all lessons provide for the ImT application.
Figure 46: LMS user interface
93
Figure 47: Course registration and available courses
94
Figure 48: SCORM web page with CBT and ImT
95
CHAPTER NINE TEST CASE
As a test case of iTrain, a flat tire replacement was chosen. Manuals included with a
car usually include some steps to replace a flat tire. In this test case, nine steps are
identified in the learning mode and the practicing mode. Table 2 describes the task for
each step.
Step Task
1 Open the trunk
2 Open the cover of tool box
3 Grip a wrench
4 Remove a nut on a spare tire
5 Loosen nuts on a flat tire
6 Place a jack
7 Remove a flat tire
8 Replace a flat tire
9 Tighten nuts on a spare tire
Table 2: A list of tasks for flat tire replacement
The Learning Mode (CBT)
In this mode, the trainee is able to see a pre-defined scenario. The trainee logs in to
LMS and registers for a course. The instructional web pages guide the trainee to learn the
course. A detailed description of each step is as follows:
Step 1: This step shows a simple simulation of an open trunk while a button is
clicked. Figure 49 shows the first step of the learning mode.
Step 2: This step illustrates the location of the tool box and shows that the
cover is open.
96
Step 3: This step brings a wrench over a nut on a spare tire.
Step 4: This step shows an animation of unscrewing the nut on the spare tire. The
simulation shows that the spare tire is being removed.
Step 5: This step shows a simulation of loosening a nut on the flat tire. The
wrench is being rotated only 1/2 turn.
Step 6: This step shows that a jack is placed under the jacking point.
Step 7: This step shows that the flat tire is being removed.
Step 8: This step shows how to change the flat tire. It also shows a simulation of
fastening the nuts in the crisscross pattern.
Step 9: This is the last step of the learning mode. Once the flat tire is replaced,
the jack is removed and the nuts of the spare tire are tightened in the crisscross
pattern.
Figure 49 shows steps 1 through 3 of the learning mode. The web page sets a current
lesson through a shared memory between the CBT application and the web page. In
addition, the web page has buttons to play an animation of the current step. This is one
direction communication, which sets the lesson from the web page to the CBT
application. For example, a wrench rotates to unscrew a nut on a spare tire while the
spare tire moves down.
Figure 50 shows steps 4 through 9. Step 5 shows what will happen when unscrewing
a nut before placing the jack. In step 5, the order of unscrewing nuts is not important.
Step 8 shows fastening a nut after replacing a spare tire. There is a sequence for fastening
nuts and the CBT application simulates the sequence.
97
Figure 49: The learning mode (Step 1-3)
98
Figure 50: The learning mode (Step 4–9)
99
The Practicing mode (CBT and ImT)
In the practicing mode in CBT, the CBT application allows the trainee to select a part
with a mouse interaction. The part will be highlighted when it is selected. Moreover, the
trainee is able to move the part and practice the lesson.
Table 3 shows the list of tasks in the practicing mode. Both CBT and ImT have a
similar task, but interaction methods are different. In this test case, ImT is used for step 5
and 8.
Step CBT ImT 1 Select a trunk - 2 Select a cover of tool box - 3 Select a wrench and move
into a nut on a spare tire -
4 Click a button in the web page to unscrew a nut on a spare tire
-
5 Move the wrench into a nut on a flat tire and click a rotate button on a web page
Grip the wrench into a nut on a flat tire and unscrew it
6 - - 7 - - 8 Move the wrench into a nut
and click a rotate button on a web page
Grip the wrench into a nut with a proper order
9 Tighten nuts Tighten nuts
Table 3: A list of tasks for a flat tire replacement
Figure 51a shows step 5 in CBT. In CBT, the wrench is selected by using a mouse
and it is highlighted. The trainee knows that the wrench is selected from the web page. In
the practicing mode, the application communicates both directions between the
application and the SCORM web page.
100
Figure 51b shows that the wrench is assembled in CBT and the wrench is gripped in
ImT. The trainee can move the component and assemble into a nut in CBT. The CBT
application does not perform to unscrew the nut, so the trainee has to click the rotate
button from the web page. In ImT, the trainee grips the wrench and engages into a nut.
Figure 51: The practicing mode
Testing mode
When trainees complete the course, they will be evaluated by taking a test. Figure 52
shows a question related a flat tire replacement. The testing mode is a text-based question.
A score will be stored in SCORM, so the trainer can review the result. Figure 53 shows
the result after submitting answers. The top picture of Figure 53 shows that the trainee
101
has selected a wrong answer. The score shown is scaled from 0.0 to 1.0. The bottom
picture of Figure 53 shows that the trainee has submitted a correct answer. The result can
be used to evaluate the trainee after taking a training course.
Figure 52: Test mode
Figure 53: Results from SCORM
102
CHAPTER TEN
SUMMARY AND CONCLUSION
This dissertation introduces an integrated approach to combine Computer-Based
Training (CBT) and Immersive Training (ImT), coined iTrain, for assembly/disassembly
processes. Both the CBT application and the ImT application have been developed and
modified for training purposes. These applications have been integrated with the learning
management system (LMS). A common repository has been identified to share common
data, such as model information, so that iTrain can be easily managed if there is
modification of models.
In the CBT application that was created in this dissertation with input devices such as
a mouse and a keyboard for a PC, the trainee is able to interact with 3D models. The CBT
application has been designed to be used in two main modes: the learning mode and the
practicing mode. In the learning mode, the trainee can watch a simulation of the training
course so that he/she understands the training course. In the practicing mode, the CBT
application enables the trainee to interact with the models with the mouse and keyboard.
Two ImT applications were investigated and modified for training. The first ImT
application was VADE and new functionality was introduced to allow industry test cases.
The second ImT application was an embedded VR environment in a CAD system and
was used for fastening/loosening components in the training course. VR hardwares such
as a tracking device, a glove, and a force feedback, have been used for the trainee to
interact with the models. The ImT application has mainly been used for the practicing
mode.
103
The learning management system (LMS), allowed the integration of digital
applications and client (web pages) applications. In addition, methods were created to
communicate between the client and the LMS. The Shared Contents Object Reference
Model (SCORM) API was introduced to communicate with LMS.
A common database was created for iTrain. Some necessary information for the
training course was shared through the common repository. The database has been
designed to be accessible from any applications and the web page. This common
repository is one of the key concepts of this dissertation for integrating between the
instructional web page and digital applications.
The models were created in the CAD system such as CATIA. These models were
processed using a CAD API in order to retrieve model information such as an assembly
hierarchy, transformation matrix, and name. The CBT application was developed on
OpenSceneGraph using C++. The ImT application was embedded in the CAD system. It
is being enhanced with more functionality. The LMS used SCORM to track a training
course from the web page. The unified approach used ODBC to build the database.
There are some limitations. The CBT application developed for this dissertation has
to have a pre-defined scenario such as sequences to simulate the learning mode. Currently,
the CBT application is not designed to support an authoring tool. This capability is
needed to help the trainer make a new training course. Since the application was based on
standard C++, COM based database such as ADO .NET is not recommended to integrate
with the CBT application.
In addition, if the trainer wants to have a new training course, the following materials
are required to use the CBT application with the LMS:
104
Instructional SCORM web pages
CAD models of the new training course
Pre-defined scenario for the learning mode: Identify which component will
be acting depends on a current lesson
Once CAD models are created, the data file can be exported from the CAD API,
which developed in this dissertation. The database can be built based on the data file from
the CAD system. Instructional SCORM web pages and the pre-defined scenario of the
learning mode should be prepared manually by the trainer.
The ImT application used for iTrain was running on Windows system. This
application can be started directly from a SCORM web page.
In the practicing mode was not designed to detect a mistake. Even though the trainee
failed to assemble a component, the trainee could go to the next lesson.
In the LMS, tests in the test mode were text based multiple-choice questions. When
the trainee selects an answer, the result is stored in SCORM database. However, this
mode needs to be enhanced with the following capabilities:
Ability to record the assembly/disassembly features in CBT or ImT.
Ability to measure the time of the practicing mode.
The design of iTrain was designed to adapt other digital applications as long as some
information can be shared through the shared memory. The data structure of the shared
memory was shown in Figure 42.
In conclusion, the work presented in this dissertation has added to cutting-edge
technology in the field of training technologies. The integrated approach, iTrain, has
successfully integrated digital applications with the learning management system. The
105
CBT application has been developed in this dissertation. The ImT application has been
developed, and enhanced functionality continues to be added. The common repository
has been identified and a unified approach successfully implemented.
106
References [1] Chou, C., H.L. Hsu, and Y.S. Yao, 1997, "Construction of a Virtual Reality Learning
Environments for Teaching Structural Analysis," Computer Applications in Engineering Education, Vol.5, No.4, pp.223-230.
[2] Hickey, D., Kindfield, A., Horwitz, P., and Christie, M., 2003, "Integrating curriculum, instruction, assessment, and evaluation in a technology-supported genetics learning environment," American Educational Research Journal, Vol.40, No.2, pp.495-538.
[3] Kenney, P. J., and Saito, T. 1994. Results of a survey on the use of virtual environment technology in training NASA flight controller for the hubble space telescope servicing mission. Technical Report. VETL.
[4] Mioduser, D., Venezky, R., and Gong, B., 1998, "The weather alb: An Instruction-based assessment tool built from a knowledge-based system," The Journal of Computers in Mathematics Science and Teaching, Vol.17, No.2, pp.239-263.
[5] Pliox, S., Désinde, M., and Michau, F., 2004, "Assessment and diagnosis for virtual reality training," Proceeding of the International Conference on Computer Aided Learning in Engineering Education, Grenoble, France.
[6] Document, U. S. D. o. D. T. 1975. Pamphlet 350-30. [7] Training in Transmission & Distribution, http://npti.nic.in/t&d.html, [accessed April,
2007]. [8] Adams, N., 1995, "Lessons from the virtual world," Training, pp.45-48. [9] Advanced integrated training in aeronautics maintenance, http://www.aitram.de,
[accessed March, 2007]. [10] Bluemel, E., A. Hintze, T. Schulz, M. Schumann, and S. Stuering, 2003, "Virtual
environments for the training of maintenance and service tasks," Proceedings of the 2003 Winter Simulation Conference.
[11] Bluemel, E., Scbenk, M., and Schumann, M., 2003, "Distributed virtual worlds with HLA?," Proceedings of the 2002 Fall Simulation Interoperability Workshop, Orlando, FL.
[12] Hintze, A., Schumann, M., and Stuering, S., 2001, "Distributed virtual training applications for education of maintenance and service personnel," Proceedings of the 2001 European Simulation Interoperability Workshop, London, Great Britain.
[13] Johnson, A., Roussos, M., Leigh, M., Vasilakis, C., Barnes, C., and Moher, T., 1998, "The NICE project: learning together in a virtual world," In Proceeding of Virtual Reality Annual International Symposium
107
[14] Jayaram, S., Jayaram, U., Wang, Y., Lyons, K., and Hart, P. F., 1996, "VADE: A Virtual Assembly Design Environment," IEEE Computer Graphics and Applications, Vol.19, No.6.
[15] Wang, Y., Jayaram, U., and Jayaram, S., 2001, "Physically Based Modeling In Virtual Assembly," Proceedings of DETC/CIE 2001, Pittsburgh, Pennsylvania.
[16] Bowman, D. A., Wineman, J., and Hodges, L. F., 1999, "The education value of an information-rich virtual environment," Presence: Teleoperators and Virtual Environments, Vol.8, pp.317-331.
[17] School, U. A. F. A. 1984. A system approach to training. [18] Fernandesa, K. J., Rajaa, V. H., and Eyreb, J., 2003, "Immersive learning system for
manufacturing industries," Computers In Industry, Vol.51, pp.31-40. [19] Dargue, B., 2005, "The Boeing Fighter Training Center I/ITSEC 2004
Demonstration: Integration of SCORM 2004 S&N and a Weapon System Trainer," IEEE Computer Society Technical Committee on Learning Technology (LTTC), Vol.7, No.1.
[20] CyberEdge Information Services, I. 1995. The sixth edition of the market for visual simulation/virtual reality systems. DoD Human Computer Interaction (HCI) Concept Plan.
[21] Blended training combines the new with the old, http://www.engineeringtalk.com/news/eng/eng105.html, [accessed March, 2007].
[22] Bridging Technology and Training, www.teibelinc.com/BlendedTraining.html, [accessed March, 2007].
[23] Clothier, P., 2003, "Strategies and techniques for designers, developers, and managers of e-learning," The e-learning Developer Journal.
[24] All mixed up by Blended Learning?, http://www.personneltoday.com/Article13133.htm, [accessed August, 2006].
[25] eLearning in Practice: Blended Solutions in Action, http://whitepapers.zdnet.co.uk/039025945,60013383p,00.htm, [accessed August, 2006].
[26] http://www.righthemisphere.com/index.html, [accessed 2004]. [27] Bridging Technology and Training – Blended Training,
http://www.teibelinc.com/BlendedTraining.html, [accessed April, 2004]. [28] Assfalg, J., A. D. Bimbo, and E. Vicario, 2002, "Using 3D and ancillary media to
train construction workers," Vol.9, pp.88-92. [29] Bennett, W. S., and Dekker, M. 1992. Visualizing software: A graphical notation for
analysis. New York, NY.
108
[30] Sutcliffe, A., 1999, "User-centered design for multimedia applications," Proceeding of the IEEE Conference Multimedia Computing and Systems.
[31] Advanced Distributed Learning, www.adlnet.org, [accessed April, 2007]. [32] Ashlock, N., Majors, W., Nilsen, D., and Paschetto, G., 2005, "Meeting Armor
Requirements with SCORM Reuse within the Scope of Content Repositories," IEEE Computer Society Technical Committee on Learning Technology (LTTC), Vol.7, No.1.
[33] Bedford, and Bruce, R., 1991, "Software reliability measurement on the B-2 aircrew training device," Interservice/Industry Training Systems Conference, Orlando, FL.
[34] Spitzer, D. R., 1984, "Why training fails," Performance and Instruction Journal, Vol.9, pp.6-11.
[35] Chastek, G., and Brouwnford, L. 1996. A case study in structural modeling. CMU/SEI-96-TR-035 ESC-TR-96-035.
[36] Booch, G. 1991. Object oriented design with applications. The Benjamin/Cummins Publishing Company, Inc., Redwood City, CA.
[37] Clements, P., and Northrop, L. 1996. Software Architecture: An Executive Overview (CMU/SEI-96-TR-003, ADA305470). Software Engineering Institute, Carnegie Mellon University.
[38] Garlan, D., and Perry, D., 1995, "Introduction to the special issue on software architecture," IEEE Transactions on Software Engineering, Vol.21, pp.269-274.
[39] Air Force Aeronautical Systems Command, 1994, Structural Modeling Handbook. [40] Abowd, G. D., Bass, L., Howard, L., and Northrop, L. 1993. Structural Modeling:
An application framework and development process for flight simulators (CMU/SEI-93-TR-14, ADA271348). Software Engineering Institute, Carnegie Mellon University.
[41] Wallnau, K., Hissam, S. A., and Seacord, R. C. 2001. Building systems from commercial components. Addison-Wesley, Boston, MA.
[42] IEEE Standard for Distributed Interactive Simulation. IEEE STD 1278. 1-1995. [43] IEEE Standard for Modeling and Simulation (M&S) High Level Architecture (HLA),
IEEE STD 1516.1-2000. [44] Heutlign, S., Schumann, M., and Steuering, S., 2003, "Interfacing virtual
environment and expert system via HLA," Proc. of the 2003 European Simulation Interoperability Workshop, Stockholm, Sweden.
[45] Adobe eLearning solutions: Rapid training, http://www.adobe.com/products/connect/solutions/rapid_training/, [accessed March, 2007].
109
[46] Open Scene Graph, www.openscenegraph.org, [accessed March, 2007]. [47] Jayaram, S., Joshi, H., Jayaram, U., Kim, Y., Kate, H., and Varoz, L., 2006,
"Embedding Haptics-Enabled Virtual Tools in CAD for Training Applications," Proceedings of IDETC/CIE 2006, Philadelphia, Pennsylvania.
[48] Autodesk 3ds Max, http://usa.autodesk.com/adsk/servlet/index?id=5659302&siteID=123112, [accessed March, 2007].
[49] Jayaram, S., Connacher, H., and Lyons, K., 1995, "Virtual Assembly Design Environment," Proc. of ASME Design Technical Conferences/International Computers in Engineering Conference, Boston, MA.
[50] Jayaram, S., Jayaram, U., Wang, Y., Lyons, K., and Hart, P., 1999, "VADE: A Virtual Assembly Design Environment," IEEE Computer Graphics and Applications, Vol.19, No.6, pp.44-50.
[51] Wang, Y., Jayaram, U., Jayaram, S., and Shaikh, I., 2001, "Methods and Algorithms For Constraint Based Virtual Assembly," Virtual Reality, Vol.6, pp.229-243.
[52] Taylor, F., Jayaram, S., and Jayaram, U., 2006, "Validation of Virtual Crane Behavior through Comparison with a Real Crane," IJAMS Vol.9, No.1, pp.73-84.
[53] Jayaram, U., Kim, Y., Jayaram, S., Jandhyala, K., and Mitsui, T., 2004, "Reorganizing CAD Assembly models (As-Designed) for Manufacturing simulations and Planning (as-built)," Journal of Computing and Information Sciences in Engineering, Vol.4, No.2, pp.98-108.
[54] Joshi, H., 2006, Open Architecture for Embedding VR-Based Mechanical Tools in CAD. Washington State University, Pullman, WA.
[55] RELOAD: Reusable eLearning Object Authoring & Delivery, www.reload.ac.uk, [accessed March, 2007].
110
APPENDIX A
CATIA API CODE
111
V5Mockup.cpp #include <iostream> #include "V5Utility.h" #include <Windows.h> using namespace std; // --------------------------------------------------------------------------- // arg[1] Command file path // --------------------------------------------------------------------------- int main (int argc, char **argv) { cout<<argc<<endl; if(argc != 2) { cout<<"Invalid arguments"<<endl; return 0; } V5Utility v5_utility; v5_utility.StartCatia (); v5_utility.OpenModel (argv[1]); v5_utility.GetProduct(); v5_utility.ExtractAssemblyTree(); v5_utility.CloseModel (); v5_utility.EndCatia (); // done - no error codes return 0; }
112
FileUtility.cpp #include "FileUtility.h" // Standard includes //------------------ #include <fstream> #include <stdio.h> #include <iostream> //---------------------------------------------------------------------------------------------- //PURPOSE: Constructor //---------------------------------------------------------------------------------------------- FileUtility::FileUtility () { } //---------------------------------------------------------------------------------------------- //PURPOSE: Destructor() //---------------------------------------------------------------------------------------------- FileUtility::~FileUtility () { } //---------------------------------------------------------------------------------------------- //PURPOSE: Create a data file() //---------------------------------------------------------------------------------------------- void FileUtility::WriteFileData (std::list<AssemblyComponentData *>
assembly_components) { std::fstream file; char file_name[256]; float matrix[12]; int i;
113
std::list<AssemblyComponentData *>::iterator assembly_components_i; strcpy (file_name, "D:\\temp\\wrenchInFlatTire.dat"); file.open (file_name, std::ios::out); for ( assembly_components_i = assembly_components.begin(); assembly_components_i != assembly_components.end(); assembly_components_i++ ) { file<<"# Name"<<std::endl; file<< (*assembly_components_i)->name<<std::endl; file<<"# Parent name"<<std::endl; file<< (*assembly_components_i)->parentName<<std::endl; file<<"# Is it an Assembly? "<<std::endl; file<<(*assembly_components_i)->isAssembly<<std::endl; file<<"# Transformation Matrix "<<std::endl; file<< (*assembly_components_i)->transformation[0]<<"
"<<(*assembly_components_i)->transformation[3]<<" "<<(*assembly_components_i)->transformation[6]<<" "<<(*assembly_components_i)->transformation[9]*0.1<<std::endl;
file<< (*assembly_components_i)->transformation[1]<<" "<<(*assembly_components_i)->transformation[4]<<" "<<(*assembly_components_i)->transformation[7]<<" "<<(*assembly_components_i)->transformation[10]*0.1<<std::endl;
file<< (*assembly_components_i)->transformation[2]<<" "<<(*assembly_components_i)->transformation[5]<<" "<<(*assembly_components_i)->transformation[8]<<" "<<(*assembly_components_i)->transformation[11]*0.1<<std::endl;
file<<std::endl; } file.close (); }
114
V5Utility.cpp // This files includes //-------------------- #include "V5Utility.h" #include "FileUtility.h" // Standard includes //------------------ #include <list> #include <iostream> // Catia includes //----------------- #include "CATDocumentServices.h" // Manage CAT documents #include "CATSessionServices.h" #include "CATDocument.h" #include "CATIDocRoots.h" #include "CATIProduct.h" #include "CATIMovable.h" #include "CATMathTransformation.h" using std::cout; using std::cin; using std::endl; //============================================================== //-----------------------Initialize Static Variables----------- //============================================================== //============================================== // ------------------------Creators-------------- //============================================== //---------------------------------------------------------------------------------------------- //PURPOSE: Constructor() //---------------------------------------------------------------------------------------------- V5Utility::V5Utility () {
115
d_v5_session = NULL; d_v5_assembly_doc = NULL; d_v5_product_var = NULL_var; } //---------------------------------------------------------------------------------------------- //PURPOSE: Destructor() //---------------------------------------------------------------------------------------------- V5Utility::~V5Utility () { } //--------------------------------------------------------------------- // FUNCTION: StartCatia //--------------------------------------------------------------------- void V5Utility::StartCatia () { HRESULT status = S_OK; cout<<"Starting Catia..."<<endl; //------------------ //A. Open a V5 session //------------------ status = Create_Session("V5Session",d_v5_session); if(status != S_OK) { cout<<"Unable to start Catia"<<endl; } cout<<"Catia Started"<<endl; } //--------------------------------------------------------------------- // FUNCTION: EndCatia //--------------------------------------------------------------------- void V5Utility::EndCatia ()
116
{ HRESULT status = S_OK; //------------------ //A. Open a V5 session //------------------ status = Delete_Session("V5Session"); } //--------------------------------------------------------------------- // FUNCTION: OpenModel //--------------------------------------------------------------------- void V5Utility::OpenModel (char *in_assembly_file_path_name) { HRESULT status = S_OK; cout<<"Opening Model..."<<endl; //Close any open document //------------------------ if(d_v5_assembly_doc != NULL) CATDocumentServices::Remove(*d_v5_assembly_doc); status = CATDocumentServices::OpenDocument(in_assembly_file_path_name, d_v5_assembly_doc, FALSE); //Error Checking. //--------------------------------------------------- if(status != S_OK) { d_v5_assembly_doc = NULL; cout<<"Failed to open model"<<endl; return; } cout<<"Model Opened"<<endl; }
117
//--------------------------------------------------------------------- // FUNCTION: GetProduct //--------------------------------------------------------------------- void V5Utility::GetProduct() { HRESULT status = S_FALSE; cout<<"Getting Product..."<<endl; CATIDocRoots * piDocRootsOnDoc = NULL; CATListValCATBaseUnknown_var *pRootProducts = NULL; status = d_v5_assembly_doc->QueryInterface (IID_CATIDocRoots, (void**)
&piDocRootsOnDoc ); if ( SUCCEEDED(status) && NULL != piDocRootsOnDoc) { pRootProducts = piDocRootsOnDoc->GiveDocRoots(); if (pRootProducts->Size()) { // the root product is first element of // the list of root elements. d_v5_product_var = (*pRootProducts)[1]; // this is how CAA wants us to dispose off pRootProducts delete pRootProducts; pRootProducts = NULL; } if ( piDocRootsOnDoc ) { piDocRootsOnDoc->Release(); piDocRootsOnDoc=NULL; }
118
} cout<<"Got Product..."<<endl; return; } //--------------------------------------------------------------------- // FUNCTION: ExtractAssemblyTree //--------------------------------------------------------------------- void V5Utility::ExtractAssemblyTree() { cout<<"Extracting Assembly Tree..."<<endl; FileUtility *file_utility = new FileUtility(); fillAssemblyTree(d_v5_product_var); file_utility->WriteFileData(d_assembly_components); cout<<"Assembly Tree Extracted"<<endl; } //------------------------------------------------------------------------- // FUNCTION: fillAssemblyTree (Recursively fills the assembly information) //------------------------------------------------------------------------- void V5Utility::fillAssemblyTree(CATIProduct *pComponent) { CATIMovable *cat_movable = NULL; HRESULT status = S_OK; const char *instance_name; //Data Structure //---------------
119
AssemblyComponentData* assembly_component = new AssemblyComponentData;
CATUnicodeString v5_instance_name; pComponent -> GetPrdInstanceName(v5_instance_name); instance_name = v5_instance_name.ConvertToChar() ; strcpy(assembly_component->name, instance_name); //Set the transformation of the component //--------------------------------------- CATMathTransformation v5_transformation; //Get the component transformation //-------------------------------- status = pComponent->QueryInterface(IID_CATIMovable,
(void**)&cat_movable); if(FAILED(status) || cat_movable == NULL) cout<<"FAILED MOVABLE"<<endl; cat_movable->GetAbsPosition(v5_transformation); cat_movable->Release(); cat_movable = NULL; v5_transformation.GetCoef(assembly_component->transformation); d_assembly_components.push_back(assembly_component); //Get the number of children //------------------------------------ int nChildren = pComponent->GetChildrenCount(); CATIProduct_var v5_product_var = pComponent ->GetFatherProduct(); if(v5_product_var != NULL) {
120
CATUnicodeString v5instance_name; v5_product_var -> GetPrdInstanceName(v5instance_name); instance_name = v5instance_name.ConvertToChar() ; strcpy(assembly_component->parentName, instance_name); } else { strcpy(assembly_component->parentName, "None"); } // Check if this component has children //------------------------------------ if (nChildren > 0 ) { assembly_component->isAssembly = true; //Get the list of Direct children //------------------------------- CATListValCATBaseUnknown_var* v5_children_list; v5_children_list = pComponent->GetChildren("CATIProduct"); // Recursively traverse the children //------------------------------------ for (int i=1; i<=nChildren; i++ ) { CATIProduct *v5_child_product = NULL; status = (*v5_children_list)[i]->QueryInterface(IID_CATIProduct,
(void**)&v5_child_product); if(FAILED(status) || v5_child_product == NULL) cout<<"FAILED CHILD PRODUCT"<<endl; fillAssemblyTree(v5_child_product);
121
//Release the Product. //-------------------------- v5_child_product->Release(); } } else { assembly_component->isAssembly = false; } } //--------------------------------------------------------------------- // FUNCTION: CloseModel //--------------------------------------------------------------------- void V5Utility::CloseModel () { cout<<"Closing Model..."<<endl; if(d_v5_assembly_doc != NULL) CATDocumentServices::Remove(*d_v5_assembly_doc); d_v5_assembly_doc = NULL; cout<<"Model Closed"<<endl; }
122
APPENDIX B
SEQEUNCE OF TASKS IN ASSEMBLY PROCESS
1
Level of support by VADE Sample of scenarios and training actions that need to be supported
Fully
supported
Can be
implemented with
minor changes
Can be
implemented with
major changes
Difficult to
implement
COMMENTS
Parts to be assembled are located on top of blocks X Part is attached to a lifting beam X Part is only attached to a large hook in VADE Attach the base part to crane using eight attachment points and a cable X VADE supported only four hanging points
Assemble nuts and bolts with hand (loosely) X X
VADE supported the insertion of a bolt and then the attaching of a nut. Major changes will be required for using two hands simultaneously for two components being attached to the base part and to each other–.
Place a part on the base part and assemble bolts X X
Minor changes will be required to allow a person to use both hands to pick up parts simultaneously– one to hold the part in place while the bolt is put in place
Personnel standing on parts and reaching X
VADE supports a full human model. Collision detection can be used to implement this capability
Medium size parts attached to the X
2
crane hooks swing in front of the worker Lower these parts and place them in the appropriate place X Perform “high wire” act in tight areas X
Move the part to a desired position X
Use a hammer to drive a part in place X X
A visual simulation of swinging of hammer is fully supported. True simulation would be difficult without force feedback and tolerance issues.
Lower big heavy parts and place them appropriately on the base part X Worker waves off the crane operator X
