1

NIEM 3.0 Data Analytics App

Dr. Brand NiemannDirector and Senior Data Scientist

Semantic Communityhttp://semanticommunity.info/

AOL Government Bloggerhttp://gov.aol.com/bloggers/brand-niemann/

May 5, 2013http://semanticommunity.info/NIEM_and_Thetus

2

Background

• Previous Stories and Analytics• NIEM Linked in Discussions• DoD Officially Adopts NIEM• NIEM 3.0 Alpha Review Request• Thetus Savannah Knowledge Base & Platform• CSDS: Cloud, SOA, Semantic, & Data Science

Conference and Pilots

3

Previous Stories and Analytics

http://semanticommunity.info/NIEM#Slide_1_Title_Slide

4

NIEM Linked in Discussions

URL

5

DoD Officially Adopts NIEM

http://semanticommunity.info/NIEM_and_Thetus#DoD_officially_adopts_NIEM

6

NIEM 3.0 Public Review

https://www.niem.gov/technical/coming-soon/Pages/Public-Review.aspx

7

NIEM 3.0 Public Review Steps• STEP 1 Download Alpha 2• STEP 2 Determine your Review Type and Access Materials

– Non-Technical Review MY NOTE: Why is this a non-technical review?– Reviewing the model in Excel allows you to see the elements and types in NIEM 3.0 without having to understand XML

schemas. To begin a non-technical review:– Locate the Microsoft Excel (.xlsx) file in the Alpha 2 package. MY NOTE: There are three Excel Files.– Open the Excel file. The file is a spreadsheet with several tabs. The first tab contains the content of NIEM Core. The other

tabs each contain an individual NIEM domain. All domains are included.• STEP 3 Conduct your Review

– We’re looking for your review and feedback on all things 3.0! If you have never taken part in a model review and/or not sure what to look for, a few suggested items to consider when reviewing NIEM 3.0 Alpha 2 are:

– Definitions: Do you see a definition that appears to be incorrect? Do any definitions appear to conflict with each other? Are definitions distinct and distinguishable (or at least different)?

– Grammar/spelling/punctuation: Do you see errors in the text of the model?– Technical Architecture: You might notice some things look different in NIEM 3.0, such as augmentations. NIEM 3.0 has

over a dozen technical improvements to architecture and structure. Because we are currently updating the NIEM Naming and Design Rules (NDR) for 3.0, we don’t expect that reviewers will be able to comment on all changes during Alpha 2.

– Harmonization: Do data element exists only once? Are they semantically precise and clear? Are they reusable and scalable across domains and the NIEM community?

– Code Lists: Are code lists up-to-date and relevant?– Other: Are there content or structural items or issues that need to be addressed?

• STEP 4 Submit your Feedback on NIEM.gov

8

NIEM 3.0 Alpha 2 Public Review

https://www.niem.gov/Pages/contact.aspx

9

niem-3.0-alpha2 File Inventory

10

niem-3.0-alpha2.xlsx: NIEM Core

http://semanticommunity.info/@api/deki/files/23929/niem-3.0-alpha2.xlsx

My Note: Why these yellow lines and Web links?

11

niem-3.0-alpha2.xlsx: Tabs1. Knowledge Base2. Consolidation: Number 3-14.3. NIEM Core4. Biometrics5. CBRN6. CYFS7. Emergency Management8. Immigration9. Intel10. Infrastructure Protection11. International Trade12. JXDM13. Maritime14. Screening15. Type Description16. Elts Sorted by Type17. Elts Sorted by Name18. Attributes19. Namespaces20. Class Type

12

niem-3.0-alpha2.xlsx: Consolidation

http://semanticommunity.info/@api/deki/files/23929/niem-3.0-alpha2.xlsx

13

niem-3.0-alpha2.xlsx: Spotfire

https://silverspotfire.tibco.com/us/library#/users/bniemann/Public?niem-3.0-alpha2-Spotfire.dxp

Navigation and Metadata

Inventory Data Set

Knowledge Base

Distribution of Types in Knowledge Base

99 Excel Tabs

112 XSDs

Consolidation Data Set

Distribution of NIEM Types

14

NIEM and Thetus Knowledge Base

http://semanticommunity.info/NIEM_and_Thetus

MashupFederated SearchWell-defined URLsSemantic Index (see next slide)

15

NIEM and Thetus Knowledge Base: Semantic Index

http://semanticommunity.info/@api/deki/files/23929/niem-3.0-alpha2.xlsx

16

niem-3.0-alpha2-codes.xlsx: Spotfire

https://silverspotfire.tibco.com/us/library#/users/bniemann/Public?niem-3.0-alpha2-Spotfire.dxp

17

Thetus Savannah: Knowledge Base & Platform

http://semanticommunity.info/NIEM_and_Thetus#Savanna_Solution_White_Paper

18

Thetus Savannah: All Source Analysis Capabilities

• Mindmap, an analysis process modeling interface and visualization.• Search interface with connectors to external data sources such as CIDNE, as well

as the local application repository.• My Stuff for organizing analysis assets, grouping them thematically and tracking

sources.• Link chart tool for creating link charts that can be the inputs to SNA analytics or

used as a network visualization.• Geospatial tool for creating high-quality maps and visualizing data with geo

coordinates.• Document viewer and annotation/markup for managing and synthesizing

unstructured text.• Virtual document to allow analysts to track content and capture insights.• Production tool to quickly assemble high quality output.• Structured data visualization for quick line, bar, scatter, pie, and bubble chart

visualizations.http://semanticommunity.info/NIEM_and_Thetus#ALL_SOURCE_ANALYSIS_CAPABILITIES

19

Conclusions and Recommendations

• This NIEM 3.0 Alpha Review Pilot Demonstrates:– Cloud Tools – MindTouch and Spotfire– SOA Services – Integrate Multiple Data Sources– Semantic Search – Facets and Context– Data Science – Extract meaning from data, create data

products, and tell a story that can be easily understood by non-practitioners.• Source: Wikipedia

• Thetus Savannah Is An Example of Mature Platform That Does These Things Well for the Intelligence Community.

• Which Is the Subject of Our Conference

20

CSDS: Cloud, SOA, Semantic, & Data Science Conference and Pilots

http://www.afei.org/events/3A03/Pages/default.aspx