SEE: Synthetic Environment Enhancement for Unity
Vaughn Whisker, PhD
Synthetic Environment Applications Lab
Penn State Applied Research Lab
SOFTWERX Future Directions Forum
Unity Game Engine
• Unity is a commercially-available, “flexible and high-performance end-to-end development platform used to create rich interactive 2D, 3D, VR and AR experiences.”
• Personal, plus, and professional versions available.• Supports mobile platforms, PC-platforms,
embedded systems, and home-entertainment systems (XBox, PS4, etc)
• Multi-player framework for collaboration• Asset store with diverse first and 3rd party-
developed capabilities.• “5.5 million registered developers including large
publishers, indie studios, students and hobbyists around the globe”
Source: Unity3d.com
DIRE: Dynamic Immersive Runtime Environment• Penn State ARL-developed framework quick configuration of Unity
applications• Enhances application support for:
• VR HMDs: Rift, Vive, Hololens, Meta2, etc.• Xbox Gamepad, Oculus Touch Controllers, Vive Wands
• Stereo Displays• Motion Tracking Systems (ART, Optitrack, etc via VRPN or trackD)• Multi-screen displays
• Tools/Interactions: Compass, Query, Measuring Tape, Grab• Additional ARL-developed Modules:
• Data Processing – Read and interpret input files, create data maps, create simple sonifications
• Satellite – Generate and animate satellite orbits from TLEs• Characters – Adds support for quickly populating a scene with virtual people• Geodesy - Adds basic geographical support to Unity via Coordinate system
projections (Flat Earth and UTM) and geodetic component that allows objects to be placed with lat/lon
Synthetic Environment Enhancement
• UI Features• Terrain Import• Geospatial Feature Support• Geospatial Database
Integration• CSV/Excel Import Support• Location Data Ingest• Transit Data Ingest
User Interface Features
• Additional Tools• Range Finder, Highlight, Attach, Draw, Marker,
Waypoint Finder• MenuUI
• Mimics hand-held tablet interface• Support for Tools, Bookmarks, Map, Routes
• MegaLabels• Large labels to provide general situational
awareness • Visible only above a user-defined altitude
• Compass Bar• Static and dynamic object locations• Icons w/ Text Labels• Cardinal directions
Terrain Model Ingest• Automated import of photogrammetry mesh
tiles• Assigns collider and changes shader to unlit/texture
• Import commercial city models • Initially FBX with textures, > 10K objects• Worked with vendor to develop unitypackage
delivery• Automated runtime import of pre-built Unity
Asset Bundles• Investigating automating import of:
• Multi-LOD Collada models• FBX tiles
Geospatial Feature Support• Coordinate system/projection
support – UTM and Flat-earth• Place objects using Lat-Lon-alt vs x,y,z
• Shapefile rendering • Shapefile attribute display for
buffered regions• e.g. Display building names, road
names, language spoken, crime rate, etc
• Shapefile-based data management• Load/unload high-res and low-res
versions in a region• Open Layers web-based map
• Online or cached map tiles• Feature layers
Neighborhoods
Named Buildings
Streets
Shapefile Sources
Geospatial Data Integration with Unity• Unity + backend data/geospatial service = “virtual location services”
• Application reports user’s location to the server• Unity web requests enable query of nearby data to be displayed in 3D environment• User position and nearby data shown on virtual handheld map • System designed to support eventual AR application
Content Interaction
Places
Data/Analytics Service
Map Tiles Analytics
Data Integration
Multiple options for integrating external geospatial data into simulation
https://xxx.xxx.xxx
Data/Analytics Service
127.0.0.1
Data/Analytics Service
Online Server Standalone w/ Locally Cached Data
In-applicationFile Access
CSV/Excel Import Support
• CSV-based model placement• Place and ground-clamp models or
labels from file with name/lat/lonformat
• CSV-based foliage management/placement
• Support for multiple tree types –map attribute to model
• Load only trees around user within user-defined distance
Location Data Ingest• Cached online sources• Open Street Map Binary (*.PBF)• User-defined data
• Web pages for data entry• Supported Location types:
• Business, Education, Entertainment, Entity, Financial, Government Building, Healthcare, Miscellaneous, Parks and Recreation, Place of Worship, Residential, Shop, Sustenance, Transportation, Unknown, plus other application specific icons.
Transit data support• GTFS: Google Transit Feed Specification
(https://developers.google.com/transit/gtfs/) • Transit authorities worldwide publish GTFS data• Spec supports stop locations, routes, trip times,
calendars, fare attributes for a transit agency’s bus and rail services
• Published as a zip file
• SEE GTFS support• Schedule display at bus stops• Animation of bus routes
• Bus Model• Icons
SEE: Summary
Transit Info: Local Transit Agency
Place Info:OpenStreetMapOnline DatabasesUser InputOthers
Trees: Locality’s Geospatial DatabaseCyclomediaNDVI Technique
Buildings:Commercial VendorsLocal Geospatial Database
MiniMap:OpenStreetMapESRIMapnik/Leaflet
Text Overlays:Neighborhood NameBuilding NameApproximate Street AddressGPS PositionNearest IntersectionCardinal Directions
Compass Bar:Cardinal DirectionNearby Locations
Challenges and Lessons Learned • Data Wrangling
• Tradeoffs of using Commercial, Open Source, Govt. sources• Missing/incomplete data
• Filtering/Curating Data• Crowdsourced sources must be filtered significantly to remove irrelevant/extraneous data
• Volume of Data vs Visual Fidelity• Limit range of data being queried• Manage high-resolution data visibility to only show nearby data
• Insufficient Spatial Resolution• Challenge: Some placemarks appear inside of buildings, Trees in/on buildings• Solution: Use GIS tools/methods to adjust source data to make them visible
• Transportation Data Update Interval• GTFS schedule has widespread support from numerous transit agencies worldwide• Challenge: “Real-time” telemetry data not updated frequently enough for real-time simulation• Solution: Use supplied schedule and route data to interpolate tracks.
• How do we make the information more consumable?• Allow user to select relevant sources, user defines relevancy• Use filters or layer managers• Use familiar modalities for delivering information – phone/tablet, virtual GPS, etc.
Questions?