Hawaii Pacific GIS Conference 2012: Real-Time Data Acquisitions - Social Media Mapping: Unstructured Data Indexing & Geospatial Search

UNSTRUCTURED DATA INDEXING & GEOSPATIAL SEARCH

APAN - Social Media Mapping:

Ty Wooldridge – Pacific War Fighting Center Director

Jerry Giles – APAN Technical Director

Tim Gramp – APAN Technical Initiatives Manager

Will Yip – APAN Software Development Lead

Todd Hall – APAN Project Manager

Presented By Harley Powers Parks, GISP – Geospatial Visualization Web Developer

AGENDA

BLUF, Where We Are & The Way Forward

Why APAN Exists, The Users & The Site

Developing & The Team

Conclusion & Questions

3/14/2012 UNCLASSIFIED

2

What is a Geo-Tag and how is tagging automated?

Tag is a word or phrase used to make it easier for

people to find a relevant document while searching.

Geo-Tags are Geographic Locations

Unstructured data is analyzed to determine if phrases

or nouns are place names Attributes:

Place Name

Confidence % (relevant to content)

Anchor (location, length)

Place names are matched to Geographic Place

Names. Attributes:

Geographic Place Name

Point (lat, long)

Weight % (population)

Type (topo, hydro, political, etc.)

Category (city, state, park, lake, building, etc.)

https://community.apan.org

Improvements:

Maps display locations of

Social Media content.

Multi-Faceted Searches are

improved with a geographic

facet.

Geo-Tag data includes

attributes, like location and

reference (anchor) within a

source document, &

Geographic Name

BLUF:

APAN automated the geographic tagging (Geo-Tag) of social media content, to include blogs, wikis, forums, & media galleries. This GIS implementation demonstrates the interoperability Between APAN and a Geographic Text Searching & Referencing System.

3/14/2012 UNCLASSIFIED 3

https://community.apan.org

WHERE WE ARE – RICH EDITING IN A BLOG, MEDIA, FORUM, OR WIKI Place Names Identified: Tulagi, Florida Islands, Solomon Island, Lunga Point, & Guadalcanal. Other sites located beyond map extent were also identified.


Map is displayed when

editing post

By default, APAN will Geo-

Tag content unless the author

wants to exclude, create, or

remove Geo-Tag

Main post is processed for

Geo-Tag, while threads and

comments are not.

Authors Can Move or Edit the

Name of a Geo-Tag

WHERE WE ARE – READING BLOGS, SHARING DATA, DISCUSSING, TEACHING OR LEARNING. Place Names Identified: Savo Island, Lunga Point, & Guadalcanal


Map is displayed for Geo-

Tag content.

Main post is processed for

Geo-Tag, while threads and

comments are not.

The map placement and

some dynamic design is

possible at runtime.

WAY FORWARD – SEARCH RESULTS, MAP CHIPS

Search Term: “Torpedo or Scuttle”


Search results page view of map

or Map “chips” is a user “facet”

choice

Map “chip” indicates a Geo-Tag

search result

Map UI interaction allows for

spatial queries

WAY FORWARD – SEARCHING, MAP CENTRIC

Search Term: “Torpedo or Scuttle”


Map Centric Search Result

Callout Box summarizes

social media content

Clustering of points allow for

large index returns with no

impact on viewing

Symbology compliments facet

search options

WAY FORWARD – GROUP SOCIAL MEDIA MAP LIBRARY

Mapping Library allows users to create thematic maps

Standard spatial data formats & web service distribution

Attention to Branding

Group Customization

Intuitive, Visually Aesthetic

Standard Extension Path

Fluid Data Transitions


AGENDA






9

WHY APAN EXISTS... STABILITY

2007 – Secretary of Defense Report on the Directive 3000.5 Military Support For Stability, Security, Transition and Reconstruction (SSTR)

2011 – Defense Information Systems Agency funds APAN for All COCOMs & Partners


10

“To improve regional stability operations coordination and interoperability, U.S. Pacific Command sponsors… APAN, an unclassified internet-based collaboration portal for humanitarian assistance and disaster relief operations.”

THE USERS

30,000+ Registered Users


11

THE USERS – THE PEOPLE WE SERVE

Unclassified Information Sharing (UIS) and Collaboration with

APAN Stakeholder & Partners:

U.S. & Joint Military Forces

Multi-National Partners

Non-Government Organizations

U.S. Interagency

Medical Community

Civilian Authorities


12

THE USERS – USAGE

Key growth drivers 2010-2011

Haiti-related awareness

SOUTHCOM adoption

Training and exercise utilization

Continued real-world event response

Focused customer engagement (coaching)


13

-

200,000.00

400,000.00

600,000.00

800,000.00

1,000,000.00

1,200,000.00

1,400,000.00

1,600,000.00

1,800,000.00

2,000,000.00

Total Page Views - (2010 - 2011)

Panamax

233K page

views

1.7 million page

views (Tomodachi)

THE SITE

Single Sign On Home Page


Single Sign On

Join

Announcements

HA/DR

Exercises

Communities

Support

THE SITE – USER PROFILES

• Personal profiles

• highlighting training,

• professional experience

• qualifications


THE SITE – COMMUNICATION Communication via email, real-time multilingual chat, and private messaging


Private conversations

Browse colleagues

Email hosting (@apanmail.org)

Real-time, multilingual text chat

(peer-to-peer and group)

THE SITE – COLLABORATION Collaboration via blogging, discussion forums and community-managed wiki resources


AGENDA






18

DEVELOPING – REQUIREMENTS

APAN Users’ need to find relevant information within multiple facets, including geography, as part of APAN’s Social Media search capability.

Users will spend little time digitizing the geographic names contained within social media textual content.

Users who choose to map APAN Social Media, require intuitive tools.


19

DEVELOPING – DATA SOURCES & SERVICES

Google’s Map Services

APAN – Services (geo-text-search-referencing)

APAN Social Media – Unstructured Data

Blogs

Forums

Wikis

Media ~ limited size, file geotags & unstructured data

COCOM & Partner Data Sources

DEVELOPING – THE ENVIRONMENT

Virtualization

VMWare workstation or fusion

Windows 2008 servers Web Server & Systems

SqlServer

Windows 7

Team Foundation Server

Visual Studio C#

Javascript

Widget Studio -

JQuery

Velocity

API’s


21

DEVELOPING – SYSTEMS

Conceptual View


22

DEVELOPING – SUB SYSTEMS

Social Media Website – Chat, Forum, Blog, Media, Wiki, Search Telligent

Telligent Job Scheduler

Tomcat - Solr

RDBMS Microsoft SQLSever

Microsoft Content Portal - SharePoint

GIS ESRI ArcGIS Server

Google Maps

Geographic Search & Referencing Platform Metacarta Web Map

Debian Linux

Apache – Manifold Cf – Web Crawler

Scheduler

TomCat – application server

Solr – Search Engine

Tika – Document Parsing

UIMA – unstructured information management

PostGreSql


23

DEVELOPING – APPROACH

Concept Solution


24

Data

Solr Index

MetaCarta Index

GeoTag Data, Store, Schedule Solr Index

as Telligent Search Results.

DEVELOPING – PLANS

Planning is integral personal experience

Agile Approach

Sprints – 30 days

Daily Participation

GIS Lifecycle System Planning

Provides the overall iteration direction


25

THE TEAM Ty Wooldridge – PWC Director

Jerry Giles – APAN Technical Director

Tim Gramp – APAN Technical Initiatives Manager

Will Yip – APAN Software Development Lead

Todd Hall – APAN Project Manager

Staffing

10 Developers – Dan Warner, Jonathan Idica, & Harley Parks

6 Knowledge Managers – Alex Conway & Jenney Tagorda

Pacific Warfighting Center Ford Island

Simulation & Exercises Support Team, IA, UIS, etc. – (15)


26

CONCLUSION & QUESTIONS


1. APAN serves DOD U.S. Joint Forces & Mission Partners

2. APAN is for Unclassified Information Sharing & Collaboration

3. APAN supports 30,000 Users & a Site for blogging, discussing, creating

wikis, sharing media, mapping, and multi-facet searching

4. APAN is developing Social Media GIS interoperable applications

5. APAN has a Great Team of Leaders, Developers, Knowledge Managers

& support.

6. Bottom Line:

APAN automated the geographic tagging (Geo-Tag) of social media content, to include

blogs, wikis, forums, & media galleries. This GIS implementation demonstrates the

interoperability between APAN and a Geographic Text Searching & Referencing

System.

THE SPRINT – REQUIREMENTS Knowledge Managers Work With Developers


Automate the Metacarta Geotag

process when Social Media is

created, updated, or deleted

During the Solr indexing, provide

the stored geotag information

Allow users to delete or create

points

Allow users to modify the

location or the name of a geotag

THE SPRINT – USER STORIES Interaction From The Users Perspective


User Stories Detail’s the

experience when interacting

with the website UI

While suggesting design

decisions, the described

capability becomes the

requirement.

THE SPRINT – IMPLEMENTATION DETAILS Images, Proof of Concepts, Task details


Capture concepts that will

break down difficult tasks

into easily completed

activities

Data Requirements

Data Sources

It maybe necessary to pull

details into a separate

document

THE SPRINT – TASK SCHEDULE & TEST PLAN General Tasks with Schedule & Test Plan


Task Schedule or time needed to

complete the task is the primary

concern

As Product near a level of

completion, KMs will create test

plans

Test Plans list key functionality

and the expected UI interaction

AGENDA

BLUF, Product & Way Forward

Development & The Sprint

The Team, The Vision, Our Mission, Guidelines, Strategy, & The People We Serve

Why APAN Exists

The Users & The Site



32

THE VISION

APAN is a Community of Communities Portal that combines

the benefits of Unstructured Collaboration

(wikis, blogs, forums, SMS texting, & mapping) and Structured

Collaboration (SharePoint, file sharing, calendar, & GIS)

with the personalization of Social Networking.

3/14/2012

33

UNCLASSIFIED

OUR MISSION

Enable our Stakeholders to Connect,

Integrate and Collaborate while Building

Partnerships and Sharing Unclassified

Information.


34

GIS GUIDELINES

Build in Geospatial Data Management

Accommodate the overall strategy for APAN going forward

Provide Solutions that are Flexible and Scalable


35

GIS STRATEGY

Embrace security policies,

Harvest unstructured geographic data sources,

Utilize crowd sourcing,

Procure or establish data partnerships, and

Propose strategic directions while

Supporting continuous Agile development

Engage GIS Lifecycle Planning

Build It


36

GIS GOALS

Integrate technologies for real-time response

Connect organizations across the globe

Provide rescue and relief information for those in need

Surface instant photos of disaster areas

Present critical data and insight to mission commanders & partners


37

AGENDA




Why APAN Exists




38

WHY APAN EXISTS… DSCA INTERFACE

Defense Support of Civil Authorities

Incident Awareness and Assessment Systems (IAA) requires geospatial data acquisition

Handbook recommends…

“Ask what maps the civilian agencies are using and how you can obtain them… so that everyone is operating from the same map.”


39

WHY APAN EXISTS… OPEN GOVERNMENT Office of Management and Budget Issues Open

Government Directive DOD Open Government Plan, April 7, 2010

http://open.dodlive.mil

Sites APAN as part of DOD’s Open Government Plan Haiti Earthquake (1/12/2010) Relief

“DISA and the US Southern Command… quickly activated collaboration platforms… All Partners Access Network (APAN) …”

“Within days over 1,700 organizations and individuals…”


40

http://open.dodlive.mil/

http://open.dodlive.mil/

WHY APAN EXISTS… DISA FUNDING!

APAN Centrally Funded for all - Combatant Commands (COCOMs) to use with

Mission Partners in their respective areas of responsibility (See Map)

Defense Information Systems Agency Responsible for the management and funding of the

existing “All Partners Access Network”

APAN serves as the baseline for DOD UIS System


41

WHY APAN EXIST– OPERATION TOMODACHI Japan Earthquake & Tsunami 2011, Case Study


WHY APAN EXIST – AFRICA ENDEAVOR Exercises


WHY APAN EXIST – AFRICOM PARTNER NATION TRAINING

Training


AGENDA




Why APAN Exists




45

THE SITE – FILE SHARING Document hosting and sharing including pictures and video


GIS, as a principal work flow, has always been part of APAN

THE SITE – DISCOVERY Information discovery by enterprise search, keyword tagging, Really Simple Syndication (RSS) and email notifications


THE SITE – INFORMATION SHARING Improvements


Mobile and low-bandwidth

Accessibility -- Focus on

situational awareness &

contribution. Enhance

discovery through

improved navigation and

search.

Inference engine – helping

users find information before

they start looking for it (topical

and geospatial relevance)

Increased platform integration –

more robust REST web service API

and widget framework extensibility.

Enhanced SharePoint capabilities

Technology

Hawaii Pacific GIS Conference 2012: Real-Time Data Acquisitions - Social Media Mapping: Unstructured Data Indexing & Geospatial Search