Embed Size (px)
Citation preview
Combining Virtual and Augmented Reality to Improve theMechanical Assembly Training Process in Manufacturing
AMAURY PENICHEEAFIT University
Virtual Reality LaboratoryCarrera 49 N 7 Sur - 50, Medellin
CHRISTIAN DIAZEAFIT University
Virtual Reality LaboratoryCarrera 49 N 7 Sur - 50, Medellin
HELMUTH TREFFTZEAFIT University
Virtual Reality LaboratoryCarrera 49 N 7 Sur - 50, Medellin
GABRIEL PARAMOEAFIT University
Production Engineering DepartmentCarrera 49 N 7 Sur - 50, Medellin
Abstract: Nowadays, the industrial sector and specially manufacturing companies, have a huge challenge tryingto train the workforce while maintaining up to date with the technology, machinery and manufacturing techniquesinvolved in the production process. For that reason this sector claims for effective, in time and quality, trainingmethodologies that do not interrupt or interfere with the continuous workflow of the company or its technologicalevolution.
Virtual reality offers an alternative that has been successfully implemented in other industries. Virtual re-ality based training systems have numerous advantages over the conventional methodologies, however, for someapplications, virtual reality cannot replace the whole training process because it may be incapable of sufficientlytransferring the skills to real world. On the contrary, augmented reality operates in a real environment, providingcomputer-generated aids to the user in order to “enhance” the real world; therefore, augmented reality basedtraining systems may transfer skills in a greater extent compared to virtual reality. However, augmented realitydoes not provide all the advantages that virtual reality does, as it makes extensive use of resources like traditionalmethodologies do. This paper proposes a new methodology that combines virtual and augmented reality; acomplete and robust process that fits more complex and demanding training needs of manufacturing companies.
Key–Words: Training Process, Training System, Virtual Reality, Augmented Reality, Manufacturing, MechanicalAssembly
1 IntroductionTraining is a vital process in the industry, and es-pecially in the manufacturing industry. Regardlessof previous experience that new employees mayhave in previous jobs, it is almost certain that theyneed some level of training in company-specifictasks or operations. The traditional methodology isthe most widely used in industry nowadays, whichbasically is carried out by an experienced trainerresponsible for transmitting the knowledge and skillsto the trainee, using real machines or components.This methodology has several drawbacks. It makesextensive use of resources, to begin with, requiring
an experienced trainer, usually a technician or evenan engineer, to be available at all times during thetraining, and keeping the machine and other resourcesengaged in a process that does not directly generatevalue to the final products. This represents a highvariable cost, because the total cost incurred for thetraining process is directly proportional to the numberof workouts. In addition, traditional methodologiesof training have a limited number of scenarios thatcan be trained, so tasks that involve risk or danger bytheir nature cannot be trained using this methodology.
To solve this problem, other alternatives may be
Applied Mathematics in Electrical and Computer Engineering
ISBN: 978-1-61804-064-0 292
considered to train the workforce in the industry,and virtual reality technologies is one of the mostsolid and most promising, as it has been successfullyimplemented for training in sectors where human lifeis at stake, as the medical, aviation and aerospace.
This paper is a continuation and improvementof [13], where virtual reality based training systemhas been proposed, giving excellent results andproviding opportunities that are not available withthe use of conventional methodologies. However,virtual reality has a limitation, no matter how realisticthe virtual environment is, there will frequently bea difference with the real environment, depend-ing on the specific training that can be large orsmall, and when the trainee is faced with the realtask after training, this difference is reflected onits performance. This is because the transferencephenomenona, due to the fact that skills sometimescannot be transferred in a sufficient manner fromthe virtual environment to the reality. Because ofthis, not in all cases the virtual reality can fullyreplace the conventional methods for training, but ifit helps to replace an important part of it. In contrast,augmented reality is a technology that is based onthe real reality, where computer generated aids areadded to a real environment, all the knowledgeacquired in training can be used directly executingthe task. The problem with augmented reality is thatit presents many of the drawbacks of conventionalmethodologies, as it makes extensive use of resources.
To improve the training process as a whole, thispaper presents an alternative training process thatcombines these two technologies, initiating theprocess with a virtual reality based training, obtainingall the benefits like low variable cost, and finalizingthe training with augmented reality, where the traineecan finish the learning process.
2 Related WorksSome studies like [9], [7] and [4] have shown thatvirtual and augmented reality are very useful tools forspecial ability training, as well as parts recognitionand assembly sequence remembrance.
Virtual environments have been widely adoptedfor training purposes with excellent results; in addi-tion, virtual reality has taken a bigger role in computergraphics and is the subject of several studies aimingto improve the capabilities of current simulators tohave higher levels of immersion and interactivity.
Virtual environments for mechanical assemblytraining, and generally all simulators must have ahigh level of immersion to achieve greater effective-ness in the process of learning or skills acquisition.Some studies like [16] show the key features forsimulators to achieve high levels of immersion,and [9] mentions some of the generalities of theseapplications in assembly processes.
Several virtual reality systems have been pro-posed for applications such as mechanical designstudies [3], evaluation of assemblies [15] and [11]and assembly path planning [12] among others. Animportant part of the study of these systems has fo-cused on the following aspects: (i) in the developmentof new algorithms for collision detection, seekinggreater accuracy and efficiency, so they can run inreal time, (ii) in the CAD modeling of the parts usedin the environment, to which is added weight andphysical properties such as roughness of the materialto make more realistic simulations and (iii) in thedevelopment of interfaces for interaction with virtualenvironments Some of these studies are [5] and [8].
Similarly, Augmented reality has also been thefocus of study for numerous applications, like medi-cal, manufacturing and visualization, as described in[4], or aircraft maintenance as described in [6], butthe use of augmented reality for mechanical assemblytraining stills somehow unexplored. One of the mainfields of research in AR is markerless tracking [1],since the markers may be inconvenient depending onthe application.
Most of the studies on virtual or augmented re-ality focus on comparing this two technologies (orchoose one over the other) rather than combiningthem into one process to obtain the benefits ofboth, and this is the main focus of this proposal, tocombine these two technologies and propound a morecomplete and flexible training process for assemblyoperation in manufacturing companies.
3 Materials and Methods3.1 System DevelopmentThe proposed training process consists of two stages;the first one, using virtual reality, where the learneracquires some of the skills to develop the task, anddepending on the tasks nature, the portion of theprocess covered by VR may vary. Up to this pointthere would not be use of resources that are part of
Applied Mathematics in Electrical and Computer Engineering
ISBN: 978-1-61804-064-0 293
the production process, so that the expert and themachine would be available for other activities.
The next step is to continue with augmented re-ality training, where the learner comes into directcontact with the machine, all of its components andparts, and complements everything learned with theVR system. The following describes the developmentprocess of both training systems.
3.1.1 Virtual Reality System DevelopmentThe virtual reality training system has a modulararchitecture. It has been designed to be highly im-mersive, so a tracking device and stereoscopic visionwere integrated into the system. The core module ofthe assembly training system was developed usingPanda3D, an open source 3D game and simulationengine developed by Disney and maintained byCarnegie Mellon Universitys Entertainment Tech-nology Center, and Python 2.7 as a programminglanguage. This module is called the AssemblyTrainer, and is the responsible of the entire logic ofthe system. The other modules are responsible ofcontrolling each interaction device integrated with thesystem, those are, optical tracking and stereoscopicvision. A screenshot of the training system in 2D canbe seen in Figure 1.
Figure 1: Screenshot of the virtual reality training sys-tem for mechanical assembly. This is the initial statewith all parts separated.
The module that controls the interaction of thesystem with the optical tracking device is calledInput Manager, and communicates directly with thetracking controller to obtain the data and pass it to theAssembly Trainer. The optical tracking system usedis an OptiTrack FLEX: V100R2. The tracking systemhas 3 different cameras that capture the position ofthe marker from 3 different angles, and using that
information it calculates the markers position in theworkspace.
The Stereoscopic Manager is the module re-sponsible of displaying the 3D image; it creates thedouble window and controls the two cameras thatcapture the images from two different angles. TheFigure 2 shows the architecture of the training system.
Figure 2: Architecture of the virtual reality trainingsystem. The “Input Manager” controls the interactionwith the optical tracker, while the “Stereoscopic Man-ager” is responsible for sending the corresponding im-ages to the projectors.
3.1.2 Augmented Reality System DevelopmentThe Augmented Reality System was built on AR-Toolkit, a software library for building augmentedreality applications, using C++ as a programinglanguage. The 3d models were made with the aidof Blender, an OpenSource software for graphicsmodeling and animation.
The logic of the application is described as fol-lows: At first, the system captures the video inputframe directly from the camera and processes theimage in order to detect the markers (also calledpatterns), then it calculates the transformation matrixfor the camera position and orientation relative to thepattern and finally draws the virtual models in thescene.
The idea of this part of the training process usingAR is to allow the user to perform the task, whichin this case is to assemble a milling machine, usingthe real parts, and not some computer generatedmodels, so skills can be transferred in a much largerproportion. AR patterns were put over some specificpieces in order to display relevant information asthe training session is developing. A picture of the
Applied Mathematics in Electrical and Computer Engineering
ISBN: 978-1-61804-064-0 294
real milling machine with the markers can be seen inFigure 3.
Figure 3: Photograph of the milling machine disas-sembled and ready to perform a training session usingAR. Some parts have patterns attached.
The system recognizes a number of patternsassociated with specific pieces of the machine onwhich the training is being done, and depending onpatterns recognized at a given time and their position,it determines the stage of the training and thus showsthe necessary information at that point, as shown inFigure 4.
Figure 4: Screenshot of augmented reality system dur-ing a training session.
3.2 Experimental DesignTo validate the effectiveness of the training processby combining virtual reality and augmented reality, aseries of tests were carried out of which process willbe described below. From a single population mainly
conformed by EAFIT students with no previousexperience in the assembly of a milling machineor mechanical assembly in general, subjects wererandomly selected to conform the experimental andcontrol groups.
The trained task consists in assembling a millingmachine. At the beginning of the process, eachparticipant, regardless of which group they belong to,had to perform the assembly task using the real com-ponents to determine its initial skills. Likewise, aftercompleting the training, participants had to performthe assembly task again to determine their final skills.After the initial assessment, the experimental groupwent trough 5 training sessions using the virtualreality training system, and then trough 3 trainingsessions using the augmented reality training system.The control group went trough 8 training sessionsusing the conventional methodology, using the realcomponents of the machine, plus the 2 assessmentsessions.
4 ResultsThe average time of each training session was plottedwith the data collected from all the training sessionsof the subjects. Figure 5 shows the average learningcurve for both methodologies.
Figure 5: Learning curves comparison, note that ses-sion 1 and 10 are the initial and final assessments, forthe RV/AR methodology: sessions 2 - 6 (VR), ses-sions 7 - 9 (AR)
The variables compared between the two sampleswere the time reduction. The experimental group hasan average time reduction X = 190.57 seconds withS21 = 11614.24, while the control group has an aver-
age time reduction Y = 186.90 with S22 = 12463.05.
The conclusion of the hypothesis test with α = 0.05is that the difference is not statistically significant,
Applied Mathematics in Electrical and Computer Engineering
ISBN: 978-1-61804-064-0 295
meaning that there is not enough evidence to state thatthe difference of the two means is different from zero,in other words, the hypothesis that the two means areequal cannot be rejected.
Analyzing Figure 5, it can be clearly seen theperformance reduction of the trainees in session num-ber 7; session number 6 was realized using the virtualreality training system, while session number 7 wasrealized using the augmented reality training system,this shows two things, that the skills the traineesobtained with the VR system could not be entirelyreplicated in a real environment, but it also shows thatthe VR training system was able of replacing a bigpart of the training process, and transferred the skillsin a significant manner. Likewise, if session 9 and10 are compared, it can be noticed that the differencebetween both times is practically inexistent, andconsidering that session 9 was performed using theAR system and session 10 was the final assessment, itconfirms that the performance achieved at the end ofthe training with AR can then be replicated in normalconditions.
5 ConclusionsFrom the initially proposed objectives, and consider-ing the results it is possible to conclude:
• Combining virtual and augmented reality into asingle training process is as effective as conven-tional methodologies, moreover, it allows a moreefficient use of resources and greater flexibilityin the type of companies can benefit from thesetechnologies, as it better fit their needs than whenusing only one of them.
• Virtual reality and augmented reality have differ-ent applications under certain conditions, makingone of them better for some cases, and the otherone better for other cases, so they can comple-ment each other.
6 Future WorkDuring development, several aspects that could beincluded in future work to improve the process wereidentified, these include:
• Develop other RV and AR systems not only tohelp companies to improve their training pro-cesses but also systems capable of helping to
improve the productivity of the companys pro-cesses.
• To facilitate collaborative tasks is possible to addnetworking capabilities over the Internet, wheretrainees can practice tasks that require the in-teraction of several people and obtain teamworkskills.
• In order to enhance the user experience in aug-mented reality, markerless techniques may be ex-plored.
References:
[1] Seok-Han Lee Ahyun Lee, Jae-Young Lee andJong-Soo Choi. Markerless augmented real-ity system based on planar object tracking. InFrontiers of Computer Vision (FCV), 2011 17thKorea-Japan Joint Workshop on, 2011.
[2] Metin Akay and Andy Marsh. Information Tech-nologies in Medicine, Volume 1, Medical Simu-lation and Education. Wiley-IEEE Press, 2001.
[3] Rajarathinam Arangarasan and Rajit Gadh. Ge-ometric modeling and collaborative design in amulti-modal multi-sensory virtual environment.In In DETC 2000, 2000.
[4] R. T. Azuma. A survey of augmented real-ity. Presence: Teleoperators and Virtual Envi-ronments, 6(4):355–385, 1997. Cited By (since1996): 787.
[5] Marvin M. Chun and Yuhong Jiang. Implicit,long-term spatial contextual memory. Journal ofExperimental Psychology: Learning, Memory,and Cognition, 29(2):224 – 234, 2003.
[6] F. De Crescenzio, M. Fantini, F. Persiani,L. Di Stefano, P. Azzari, and S. Salti. Aug-mented reality for aircraft maintenance trainingand operations support. IEEE Computer Graph-ics and Applications, 31(1):96–101, 2011.
[7] A. Dnser, K. Steinbgl, H. Kaufmann, andJ. Glck. Virtual and augmented reality as spatialability training tools. In ACM International Con-ference Proceeding Series, volume 158, pages125–132, 2006.
[8] Nobutaka Endo and Yuji Takeda. Selectivelearning of spatial configuration and objectidentity in visual search. Attention, Percep-tion, amp; Psychophysics, 66:293–302, 2004.10.3758/BF03194880.
Applied Mathematics in Electrical and Computer Engineering
ISBN: 978-1-61804-064-0 296
[9] S. K. Gupta, D. K. An, J. E. Brough, R. A.Kavetsky, M. Schwartz, Ra K. Gupta, Davin-der K. An, John E. Brough, Robert A, MaximSchwartz, and Atul Thakur. A survey of the vir-tual environments-based assembly training ap-plications, 2008.
[10] Satyandra K. Gupta, Davinder K. Anand,John E. Brough, Maxim Schwartz, Satyandra K.Gupta, Davinder K. Anand, John E. Brough,Maxim Schwartz, and Robert A. Kavetsky. Asafe, cost-effective, and engaging approach totrainingtraining in virtual environments a safe,cost-effective, and engaging approach to train-ing, 2008.
[11] Uma Jayaram, Sankar Jayaram, CharlesDeChenne, Young Jun Kim, Craig Palmer, andTatsuki Mitsui. Case studies using immersivevirtual assembly in industry. ASME ConferenceProceedings, 2004(46970):627–636, 2004.
[12] A. Mikchevitch, J.-C. Leon, and A. Gouskov.Numerical modeling of flexible components forassembly path planning using a virtual realityenvironment. ASME Conference Proceedings,2003(36991):1115–1124, 2003.
[13] Trefftz H. Peniche A., Diaz C. and Paramo G.An immersive virtual reality training system formechanical assembly. In Recent Advances inManufacturing Engineering, Proceedings of the4th International Conference of ManufacturingEngineering, Quality and Production Systems(MEQAPS 11), pages 109–113, 2011.
[14] Mateusz Skoczewski and Hitoshi Maekawa.Markerless pose tracking based on local imagefeatures and accelerometer data. In Hamid R.Arabnia and Leonidas Deligiannidis, editors,CGVR, pages 194–199. CSREA Press, 2009.
[15] Huagen Wan, Shuming Gao, Qunsheng Peng,Guozhong Dai, and Fengjun Zhang. Mi-vas: A multi-modal immersive virtual assem-bly system. ASME Conference Proceedings,2004(46970):113–122, 2004.
[16] D. Weidlich, L. Cser, T. Polzin, D. Cristiano, andH. Zickner. Virtual reality approaches for im-mersive design. CIRP Annals - ManufacturingTechnology, 56(1):139 – 142, 2007.
Applied Mathematics in Electrical and Computer Engineering
ISBN: 978-1-61804-064-0 297
