Augmented Reality in
Transportation Construction
FHWA Contract DTFH6117C00027: LEVERAGING AUGMENTED REALITY FOR HIGHWAY CONSTRUCTION
Hoda Azari, Nondestructive Evaluation Research Program Manager
IHEEP 2018 – Lincoln, Nebraska. September 25, 2018
September • 2018
Augmented Reality Research Project
FHWA Contract DTFH6117C00027
Thorough review of state-of-the
art augmented reality (AR)
technologies and applications for
Highway Construction
Draft Report will be completed at
the end of 2018
Augmented Reality (xR) Terminologies
Reality Virtual Reality
Augmented
Reality
Augmented
Virtuality
The Reality – Virtuality Continuum
Mixed Reality (xR)
Augmented Reality (xR) Terminologies
Reality Virtual Reality
Augmented
Reality
Augmented
Virtuality
Mixed Reality (xR)
Daqri Smart Glasses (AR)
Google Daydream (VR)
Augmented Reality (xR) Terminologies
Reality Virtual Reality
Augmented
Reality
Augmented
Virtuality
Mixed Reality (xR)
Typical AR Devices are closer to the
‘Reality’ end of the spectrum:
• Primarily ‘See-through’ of reality
• Some VR overlay
Augmented Reality Technologies
Components of AR System:
Sensory input
Display device
Haptic input (navigation/interaction)
Tracking Systems
Computing Device
Media Representation
Data Input / Storage
Augmented Reality Technologies
Display devices fit into two categories
Hand-held devices – tablets, smart phones
Head-mounted displays (HMD) / Glasses
Augmented Reality Technologies
Tracking Methods – Position and Orientation
GNSS / Wifi networks / V2X/I2X (radio systems for CAV)
Inertial sensors – track local movements of the device
Optical sensors – track 2D video or 3D imagery of the real-world
Sensor fusion – systems that combine all of the tracking methods
Marker-based position initialization
Augmented Reality Technologies
Hand-held devices and commercial AR applications
Google Translate
PokemonGo
Augmented Reality Technologies
Hand-held devices and AR applications
Video see-through: real-world captured by camera on device
GNSS positioning / Inertial tracking for orientation and movement
Some devices use optical or 3D sensor tracking
Initial registration typically achieved with markers / targets in scene
Hand-helds on construction sites not necessarily new – AR could be add-on
to existing uses of mobile devices
Augmented Reality Technologies
Head-mounted displays
Optical see-through: real-world seen directly through lenses
GNSS positioning / Inertial tracking for orientation and movement
Some devices use optical or 3D sensor tracking
Initial registration typically achieved with markers / targets in scene
AR Information Display Options
Categories of information display that AR will support:
Display what is not yet constructed
• Compare design alternatives in context
• Check relationships between existing/future elements
• Site logistics, equipment movements
• Preview complex installation procedures
• Illustrate construction methods and sequencing
• Opportunities for training on-site
• Check traffic management/temporary structures
Display what was intended to be constructed
• Site inspection and validation
• Code/standards/compliance checking
• Checking quantities and work progress
• Inspection training opportunities
• Check traffic management/temporary structures
AR Information Display Options
Categories of information display that AR will support:
Display what is hidden from view
• Buried utilities or structural components
• Elements obstructed from the current view
Display abstract information aligned with real-world context
• Alignment information, easements, site boundaries, ROW boundaries
• Environmental boundaries, such as flood levels or sea-level rise data
• Sensitive areas such as archeological and historic sites
• Meta-data tagged to associated real-world objects
• Potential work-zone hazards
Augmented Reality Challenges
Challenges with current systems
Field of View (FOV)
• Handheld devices limited to FOV displayed on screen held at arms length
• Current HMD’s typically limit virtual display to 60 degrees FOV (see next slide)
Field of View (FOV)
Right eye 135⁰Left eye 135⁰
Binocular Field of View
120⁰
Typical AR Display FOV
60⁰
Augmented Reality Challenges
Challenges with current systems
Occlusion
• Current systems can only display overlaid
virtual information in the foreground
• Virtual elements that are ‘behind’ real-
world elements are not masked out
• Limits the immersive quality of the AR
representation
Calculating occlusion requires that the AR
application have detailed 3D information of both
the real-world scene and the virtual model
Augmented Reality Challenges
Challenges with current systems
Ruggedness / Durability
• Current devices not designed for daily outdoor use, safety issues
Performance / Portability
• Display processing must be performed on-board the device
• Some devices tethered to ‘wearable’ computing device – adds weight
• Most applications require storage for ‘pre-loading’ of 3D assets onto the device
• WiFi connectivity / streaming could allow assets to be loaded on demand – no
examples of this yet (solvable in the near future with 5G?)
Safety
• Hand-held devices – use one or usually both hands and block view
• HMD’s – can limit peripheral vision, and possibly audio
Augmented Reality Challenges
Challenges with current hardware
Tracking technologies
• GNSS requires visibility of satellite network
• Most systems require marker-based or manual registration
• Optical / 3D sensors require ‘visible’ detail in environment
• Bright / outdoor scenes are difficult for optical tracking
These last two requirements could be key challenges with construction,
especially in an outdoor work environment without details to track
Display brightness
• Current display technology in HMD’s is limited in ability to display over optical
see-through devices in a bright outdoor environment
Matching virtual display to real-world brightness problematic, especially in
outdoor environment
Augmented Reality Software
AR Ready ‘Cloud-based’ Platforms
Autodesk
• BIM360 Platform – Not currently supporting Geo-referencing
• Forge– development platform
Bentley
• Bentley Connect
Trimble
• Trimble Connect
AR Ready Developer Platforms
• Apple ARkit, Google ARCore, Windows Mixed-Reality, WebAR
Game Platforms - Unity most common in AR
Augmented Reality Workflows
AR in 3D Model-based Workflows
• AR should become a critical tool within existing BIM model workflows
• Piggyback on BIM data management and 3D modeling efforts
• Some current BIM platforms already supporting AR tools
AR in Workflows for other data types
• Leveraging other types of information (non-3D) will require new tools and workflows
• Abstract data could be included in 3D model if geolocated correctly
ARTC Workshop
Augmented Reality in Transportation Construction (ARTC)
Held May 9th 2018, at Turner Fairbanks
Highway Research Facility
~40 participants
• FHWA / State DOT’s
• University / Research
• Contractors / Consultants
• AR Vendors / R&D
Goals:
• Educate participants in AR
• Identify new applications for AR
• Prioritize future applications of AR
ARTC Workshop
Conducted in Three Sessions
1: Overview of AR Technologies
• Current display technologies
• Characteristics of AR systems
• Challenges with AR tools for construction
2: Vendor technology presentations and hands-on demonstrations
• Participants were able to see and experience the AR tools
3: Brainstorming and participant polling of application opportunities
• Discussion and identification of potential AR applications
• Documentation of challenges for AR applications in construction
• Electronic ranking of top AR application concepts
ARTC Workshop
Key themes emerged in the brainstorming session on AR
application areas in construction
1: Visualization of design information
• Simplified 3D model components
• Abstract 2D information such as alignments or
