Embed Size (px)
Citation preview
AR4CAD
A FRAMEWORK FOR THE CREATION OF
AUGMENTED REALITY BASED
ASSEMBLY ASSISTANTS
Luca Piccinini, Claudio Salati
T3LAB, Bologna, Italy
www.caeconference.com
What is AR4CAD?
AR4CAD makes use of the CAD description of a
complex object and of CAD tools to support the
rapid creation of the storyboard of the virtual
assembly of that object
Based on this storyboard, AR4CAD automatically
generates a specific, AR (augmented reality) based,
assembly assistant
The AR assistant supports the operator during the
real assembly in several ways
2017, 6 - 7 November 33rd INTERNATIONAL CAE CONFERENCE AND EXHIBITION 2
www.caeconference.com
AR assistant support during
the real assembly Guides the operator through the appropriate sequence of steps of
the process
Recognizes the individual objects being handled (without the need
of markers) and eventually checks their dimensional characteristics
Provides the information required to mount component parts in the
right way
mainly verbally, and
by properly overlaying virtual companion parts aside the image of an
anchor physical object captured by the camera or by overlaying a
completely virtual assembly animation over this same anchor
verifies (to some extent) that assembly steps have been performed
correctly
2017, 6 - 7 November 33rd INTERNATIONAL CAE CONFERENCE AND EXHIBITION 3
www.caeconference.com
AR assistant:
visual assembly instructions
2017, 6 - 7 November 33rd INTERNATIONAL CAE CONFERENCE AND EXHIBITION 4
www.caeconference.com
AR assistant: automatically
generated visual inspections
2017, 6 - 7 November 33rd INTERNATIONAL CAE CONFERENCE AND EXHIBITION 5
Completeness of an assembly Dimensional checks of parts
www.caeconference.com
Scientific challenges
Not a specific AR application but a CAD based
framework that supports the rapid development of AR
based assembly assistants
Development of an AR assistant should require, at
most, up to 10 times the time required to do the job
Markerless computer vision
Not even for localization
But DPM should be supported
Integration of vision based quality inspections
Also to check the work of operators
2017, 6 - 7 November 33rd INTERNATIONAL CAE CONFERENCE AND EXHIBITION 6
www.caeconference.com
System setup
2017, 6 - 7 November 33rd INTERNATIONAL CAE CONFERENCE AND EXHIBITION 7
AR4CAD
run-time &
executable
storyboard
www.caeconference.com
The AR4CAD framework
2017, 6 - 7 November 33rd INTERNATIONAL CAE CONFERENCE AND EXHIBITION 8
Generation of the AR assistant (executable storyboard) • Generation of the executable XML file (from Publisher file)
• Generation of object models for detection by Halcon vision library
• Export of component parts and sub-assemblies as VRML virtual
objects and generation of VRML animations
• Text of audio interactions for on line synthesis, ...
CAD +
Autodesk
Publisher
Creation of the storyboard • Sequence of the assembly steps
• List of component parts and sub-assemblies
• Description of animations and virtual objets: reference system,
involved parts
• AR4CAD pragmas:
• Keywords to identify the instruction associated to a frame
• Audio instructions and multi-media helps for the operator
• Current menu: list of commands available to the operator
• Time info
Execution of the AR assistant
AR4CAD
compiler OFFLINE
AR4CAD
runtime
OFFLINE
ONLINE
www.caeconference.com
Storyboard creation .1
Stepwise exploded view drawing
2017, 6 - 7 November 33rd INTERNATIONAL CAE CONFERENCE AND EXHIBITION 9
Explosion tree
www.caeconference.com
Reversing the tree (any “postorder”) with a simple
paradigm at each step (from child-nodes to their
parent):
fetch - check – mount - check
Storyboard creation .2
2017, 6 - 7 November 33rd INTERNATIONAL CAE CONFERENCE AND EXHIBITION 10
www.caeconference.com
Storyboard editing .1
2017, 6 - 7 November 33rd INTERNATIONAL CAE CONFERENCE AND EXHIBITION 11
First part of the
assembly: Part A
Second part of the
assembly : Part B
AR4CAD instruction:
FETCH
Objects in the frame are parameters of the instruction
Autodesk Publisher
www.caeconference.com
Storyboard editing .2
2017, 6 - 7 November 33rd INTERNATIONAL CAE CONFERENCE AND EXHIBITION 12
Assembly instruction:
A+B
AR4CAD instruction:
MOUNT [before the
assembly]
AR4CAD instruction
POST_MOUNT [after the
assembly] for automatic
generation of animation
and for visual inspection The pose of the objects in the frame is relevant!
www.caeconference.com
Objects’ pose and properties The pose of an object in the frame is relevant!
• It is the pose in which the object will be displayed to the operator
• It represents the start or the end point of an animation
• It is the pose (or one in its nearness) in which the system will try to
detect the object (e.g. as the anchor for AR instructions)
• Other significant poses can be indicated textually around the axes of
the intrinsic reference system of the object (x, y, z)
• Constraining poses, though not mandatory, allows faster and safer
detection
An object may be tagged with parameter CHECK_DETAILS
• At run-time this triggers a visual inspection that performs dimensional
checks on the objects that have been detected
• The inspection is specific for each supported object class: nuts, bolts,
… The class of an object and its sizes are extracted from the CAD
database
2017, 6 - 7 November 33rd INTERNATIONAL CAE CONFERENCE AND EXHIBITION 13
www.caeconference.com
The AR4CAD IDE
Editing an AR4CAD storyboard takes an amount of
time comparable to the execution of the described
physical operation
The AR4CAD IDE (Integrated Development
Environment), though still primitive, takes advantage
of existing CAD tools (Autodesk Publisher)
It proves that an effective, sophisticated IDE could
be implemented as a CAD/CAM plug-in
2017, 6 - 7 November 33rd INTERNATIONAL CAE CONFERENCE AND EXHIBITION 14
www.caeconference.com
SW toolkits used in AR4CAD
Computer vision: commercial machine vision library (Halcon - MvTec)
Computer graphics: open source library (OpenVRML)
IDE: CAD toolkit (Inventor Publisher - Autodesk)
Synthesis of virtual animations: CAD toolkit (Autodesk 3ds Max)
Speech analysis: open source library (CMU Sphinx)
Speech synthesis: open source library (Microsoft Speech API )
2017, 6 - 7 November 33rd INTERNATIONAL CAE CONFERENCE AND EXHIBITION 15
www.caeconference.com
Possible evolution Redesign of the IDE
Redesign of the operator’s interface
Glasses
Support of operator’s feedback
3D computer vision
Moving to the cloud
Voice processing
Information storage
Support of other operations
Replacement
Diagnosis
2017, 6 - 7 November 33rd INTERNATIONAL CAE CONFERENCE AND EXHIBITION 16
www.caeconference.com
With the support of …
2017, 6 - 7 November 33rd INTERNATIONAL CAE CONFERENCE AND EXHIBITION 17
The CAD toolset has been provided by
The machine vision library for 3D object
detection and localization, based on CAD models,
has been provided by
The mechanical parts, their CAD design and the
mounting instructions have been provided by
We are pleased to acknowledge the support of all
these companies, of and of
program
www.caeconference.com
AR4CAD links http://www.t3lab.it
http://www.t3lab.it/progetti/ar4cad/
contacts: [email protected]
videos:
1. http://www.youtube.com/watch?v=LGFBReqK2bs shows the
execution of an assembly with the help of the AR application
generated by our framework
2. http://www.youtube.com/watch?v=y7NJG1_C1M8 shows the
editing of the storyboard from which our framework
generates the AR application that is run during the first video
2017, 6 - 7 November 33rd INTERNATIONAL CAE CONFERENCE AND EXHIBITION 18
