Analytic Component System (ACS) - ncsu-las.org · Storyboard Scene 6: Workflow Capture and Analysis The constructs logged by the ACS provide a rich source of instrumentation data

Analytic Component System(ACS)

Matt Schmidt and Andrew Crerar

Presentation to LAS Weekly Research Meeting13 April 2015

Collaborators• Tonya Adelsperger (LAS-G)• Joe Aguayo (LAS-G)• Edward Browne (LAS-G)• Samuel Christie (NCSU)• Bill Elm (RCS)• Rob Johnston (Johnston Analytics)• Judy Johnston (Johnston Analytics)• James Kaiser (LAS-G)• Munindar Singh (NCSU)

• John Slankas (LAS-I2I)• Colleen Stacy (LAS-G)• Pankaj Telang (LAS-I2I)• Eli Typhina (NCSU)• Kathleen Vogel (NCSU)• Lori Wachter (LAS-G)• Alyson Wilson (NCSU)• Mark Wilson (NCSU)• Jorge Zuniga (RCS)

Outline

Motivation and OverviewTradecraft StoryboardAnalytic Component Interface

Background and MotivationApproachInitial ResultsBenefits

Analytic Computing EnvironmentBackgroundDemo #1: Execution of Composite WorkflowDemo #2: Inspection of WorkflowDemo #3: Modification of Workflow

Future Work and Summary

Goal of ACS

Provide analysts a library of analytic components that can becombined to create analytic workflows

Provide analysts a library of analytic components that can becombined to create analytic workflows

ProblemProblem• Analysts have access to a wide variety of

analytic tools and methods• Applying these tools and methods to a problem

can require a great deal of effort to:1. Learn about the tool/method and;2. Adapt the analyst’s workflow to

incorporate it

Desired OutcomeDesired Outcome

An organized set of analytic tools and methods(components) that an analyst could use toconstruct their desired analytic workflow

Technical Challenge:Component Integration

The costs of integrating different components into a singleworkflow can overwhelm any benefits

The costs of integrating different components into a singleworkflow can overwhelm any benefits

• Analytic components consume and producecertain types of information

• Integration occurs when one componentconsumes information produced by another

• Integration costs occur when there is a need totranslate the information before consumption

• Time• Cognitive Load• Mis-Communication

• Integration costs can arise at many differentplaces

• Component to Component• Component to Human• Human to Human

Integration CostsIntegration Costs

VS

Technical Challenge:Workflows are not Applications

Analytic workflows do not look like procedural applications andtherefore require a different computation environment

Analytic workflows do not look like procedural applications andtherefore require a different computation environment

• The analytic components used are not always known before execution

• Components can be computational, manual, or interactive

• Analysts may want to revisit and modify previously executed workflows

Technical Approach

The ACS effort consists of three related development effortsThe ACS effort consists of three related development efforts

GoalGoal

Component InterfaceComponent Interface Component LibraryComponent LibraryComputing EnvironmentComputing Environment

Enable compatible componentinterfaces

Enable execution of workflowscomposed of analytic components

Develop library of novel andexisting analytic components

Define standard analyticconstructs

ApproachApproachDevelop service-oriented

computing platformDecompose and wrap components

using ACI to execute in the ACE

Outline

Motivation and Overview Tradecraft Storyboard

Analytic Component InterfaceBackground and MotivationApproachInitial ResultsBenefits



ACS Technology-EnabledTradecraft Storyboard

Task of the analyst is to report on likely targets of theAnonymous hacktivist group

Task of the analyst is to report on likely targets of theAnonymous hacktivist group

BackstoryBackstory

• Single Analyst• Components in ACS plus front-end• Multiple open data sources• Collected reporting

QuestionQuestion

Who is the most likely next target of theAnonymous hacktivist group?

ResourcesResources

Storyboard Scene 1:Question to Data Source

Different components in ACS use different combinations ofcomputation and manual processes to generate constructsDifferent components in ACS use different combinations ofcomputation and manual processes to generate constructs

Scene DescriptionScene Description

ActionAction

Analyst selects a background report onAnonymous

Entities InvolvedEntities Involved• Analyst• Analytic Component from ACL

• Produces Data Source given aQuestion

• ACE• Sends question to component and

sends data source to next component• Background Report

Storyboard Scene 2:Data to Indicators

The ACI enables the data inspection component to generateconstructs understood by question resolution componentsThe ACI enables the data inspection component to generateconstructs understood by question resolution components


ActionsActions• Analyst reads line in background report

saying Anonymous members often havestrong libertarian beliefs

• Analyst creates an indicator question:Who are the most visible anti-libertarianactors right now?

Entities InvolvedEntities Involved• Analyst• ACL Components• ACI Constructs• Background report

Storyboard Scene 3:Indicators to Ranking

The ACL and ACE enables a question resolution workflow to beautomatically composed and executed

The ACL and ACE enables a question resolution workflow to beautomatically composed and executed


ActionActionAnalyst inspects automated workflow to see

how list of anti-libertarian actors wasproduced

Entities InvolvedEntities Involved• Analyst• ACL enables automated construction of

workflow to produce ranked list• ACE logs all constructs produced in

workflow• Front-end allows analyst to inspect

Storyboard Scene 4:Manually introduce new data source

The ACE enables analyst modification and control of anautomated piece of the analytic workflow

The ACE enables analyst modification and control of anautomated piece of the analytic workflow


ActionAction• Analyst adds a missing data source to the

workflow• Workflow automatically runs

Entities InvolvedEntities Involved• Analyst• ACE enables workflow modification• Front-end enables inspection of workflow

and addition of constructs

Storyboard Scene 5:New Hypothesis Alert

The ACS along with User Experience efforts supports transition ofanalysis effort to monitoring effort

The ACS along with User Experience efforts supports transition ofanalysis effort to monitoring effort


ActionActionAnalyst is alerted to new anti-libertarian

actor because of a new data being added tothe data source

Entities InvolvedEntities Involved• Front-end determines how to alert user• ACE enables streaming ingest and

analysis of data using workflows originallyfor historical data

• ACL consists of components that can berun in a streaming fashion

• ACI enables the use of either batch orstreaming methods of producingconstruct

Storyboard Scene 6:Workflow Capture and Analysis

The constructs logged by the ACS provide a rich source ofinstrumentation data for reporting and analysis

The constructs logged by the ACS provide a rich source ofinstrumentation data for reporting and analysis


ActionAction• Draft report generated automatically• Workflow becomes part of report• Analysis of workflow can provide estimate

of analytic rigorEntities InvolvedEntities Involved

• Elastic Search logs all analytic constructs• Auto-report generation uses constructs to

produce draft report• Analytic rigor maps production of

constructs to metrics of analytic rigor

Storyboard Slide 7:Develop new components

Previously captured workflows provide training data for thedevelopment of new components and workflow recommenders

Previously captured workflows provide training data for thedevelopment of new components and workflow recommenders


ActionActionResearchers use logged pairings of

constructs to develop automated method ofgenerating the constructs

Entities InvolvedEntities Involved• Researchers• Logged ACI constructs in Elastic Search• Analyst production of constructs

Related Efforts

Multiple LAS efforts can benefit from or contribute to the ACSMultiple LAS efforts can benefit from or contribute to the ACS

InstrumentationInstrumentation

ACI constructs define the resolution ofworkflow logs

Analytic RigorAnalytic Rigor

ACI constructs provide data from which toderive metrics on analytic rigor

User ExperienceUser ExperienceUser Experience will be required enableanalyst to observe and control workflows

executed in the ACS

Recommender SystemsRecommender Systems

Logged ACI constructs provide rich trainingdata source for recommender system efforts

Analytic Component DevelopmentAnalytic Component DevelopmentACS makes explicit the needs for new

components for existing components thatare inefficient

Auto-Report GenerationAuto-Report GenerationLogged ACI construct should provide

enough information to create a draft reportautomatically

Outline

Motivation and OverviewTradecraft Storyboard

Analytic Component Interface Background and Motivation

ApproachInitial ResultsBenefits



Analytic Constructs

An analytic construct is a piece of information either required orprovided by an analytic component

An analytic construct is a piece of information either required orprovided by an analytic component

• Analytic component produce one type of information using informationthat has been provided

• Examples• An analysts that provides a hypothesis for a given question• A clustering algorithm that provides a mapping to the different cluster

IDs for all of the provided data points

• The different types of information either provided or produced by acomponent are referred to as analytic constructs

Interfaces

In a modular workflow, analytic components are represented bytheir interfaces

In a modular workflow, analytic components are represented bytheir interfaces

InterfacesInterfaces ImpactImpact

BenefitsBenefits• Makes modular components easier to reuse

because the client only needs to understandthe interface

• Enables modular components to beinterchangeable as long as the share the sameinterface

• The method by which the component producesthe constructs has been abstracted

• From a systems perspective, the componentbecomes a black box

A component’s interface is the set of constructsthat are either provided or required by the

component

ExamplesExamples• Hypothesis Generation Component

• Required: Question• Provides: Hypothesis

• Clustering Component• Required: Vectorized Data Points• Provides: Mapping of Data Points to

Clusters

Compatible Interfaces

Integrating components with incompatible interfaces leads topotentially overwhelming integration costs

Integrating components with incompatible interfaces leads topotentially overwhelming integration costs

• Integrating two components meansgenerating the constructs required by onecomponent from the constructs provided byanother

• Components are compatible if theconstructs that are required by onecomponent are provided by another

IntegrationIntegration

• If two components are not compatible, then integrating those two components will requirean integration cost of transforming the provided constructs into the required constructs

• Paying this integration cost every time the workflow is executed can become anoverwhelming burden

Compatibility and Integration CostsCompatibility and Integration Costs

Goal of the ACI

Ensure compatible component interfaces by defining a standardset of analytic constructs

Ensure compatible component interfaces by defining a standardset of analytic constructs

Outline


Analytic Component InterfaceBackground and Motivation

ApproachInitial ResultsBenefits



Highest-Level Abstraction ofAnalysts’ Task

Analysts provide information about the world as it was (reflect), asit is (observe), or as it will be (imagine)

Analysts provide information about the world as it was (reflect), asit is (observe), or as it will be (imagine)

DataRepositories

and FeedsAnalyst

InformationRequest

InformationRequest

Did it rainyesterday? Query

yesterday.rainfall

yesterday.rainfall = 0“No, it did notrain yesterday”

Inference Task

Working Hypothesis: Inference from data, observations, andpredictions is the primary task of almost all analysis

Working Hypothesis: Inference from data, observations, andpredictions is the primary task of almost all analysis

DataRepositories

and FeedsAnalyst

InformationRequest

InformationRequest Will it rain

tomorrow?

Query ‘radar’

Query ‘pressure’“Yes, it will likelyrain tomorrow”

Query ‘humidity’

Query ‘temperature’

Other Domains of Inference

An abstract analyst inference process assesses evidence to createassessments of hypotheses likelihoods

This inference process is conceptually similar to statistical andcomputational inference processes

Hypothesis: Constructs in statistical and computational inferencewill have analogs in intelligence analysis workflows

Hypothesis: Constructs in statistical and computational inferencewill have analogs in intelligence analysis workflows

Outline


Analytic Component InterfaceBackground and MotivationApproach

Initial ResultsBenefits



Current Hypothesis for Constructs

This construct set is only a hypothesis and likely to changeThis construct set is only a hypothesis and likely to change

Construct Type DefinitionProposition Proposition A statement that can be either true or false

Truth Function Whether or not a proposition is consistent with the state of the world(s) being analyzedQuestion Request A request for propositions and an assessment of their truthAssertion Assertion A proposition that is true for all worlds being analyzed

Data Assertion Assertions that serve as the foundation for the analysis processQuery Request A request for data that satisfy a certain criteriaValidity Function Whether or not the proposition is of the type requested by the question

Hypothesis Proposition A proposition that is valid for the given questionDecision Assertion An asserted hypothesis of Q

Relationship Function The degree to which the truth of P1 is affected by asserting the proposition(s) P2Corollary Proposition A proposition that is strongly supported by the given proposition(s)Evidence Assertion An assertion that is relevant to the given hypothesis(es) or question(s)Indicator Request A question whose hypotheses can serve as evidence to the given question

Likelihood Function The truth of the hypothesis H when the evidence(s) E is assertedAssessment Function The relative likelihoods of the hypotheses of a question Q given the same set of evidence E

Model Function The set of all relationships between hypotheses of the set of questions QScenario Assertion A set of assertions that generated according to model M

Current Challenge toSpecifying Constructs

Need to find a mechanism by which to describe and validateconstructs across different fields

Need to find a mechanism by which to describe and validateconstructs across different fields

Description ACI Construct Statistical Analog Classification AnalogThe piece of information tobe inferred Question Response Variable Class Variable

All potential values for theinformation requested Hypothesis Set Response Variable Domain Label Set

The information determinedto be most likely Decision / Assessment Most Likely Estimate Assignment

Outline


Analytic Component InterfaceBackground and MotivationApproachInitial Results

BenefitsAnalytic Computing Environment

BackgroundDemo #1: Execution of Composite WorkflowDemo #2: Inspection of WorkflowDemo #3: Modification of Workflow


Benefit of Having Standard Constructs

Workflows can be described in terms of the constructs that will beproduced instead of the components that will be used

Workflows can be described in terms of the constructs that will beproduced instead of the components that will be used


Workflows can then be used to determine what componentsshould be integrated into the workflow

Workflows can then be used to determine what componentsshould be integrated into the workflow


Recursive workflows arise naturallyRecursive workflows arise naturally

Outline



Analytic Computing Environment Background

Demo #1: Execution of Composite WorkflowDemo #2: Inspection of WorkflowDemo #3: Modification of Workflow


Components and Services

Analytic components are the methods used to enact the service ofproducing analytic constructs

Analytic components are the methods used to enact the service ofproducing analytic constructs

ExampleExample Using ServicesUsing Services

• A service is a task being performed for others• A component is the method by which the task

is performed

Services can be composed in different ways toaccomplish larger tasks

ServicesServices

ComponentsComponents

Atomic Services

The enactment of atomic services only requires executing theunderlying component

The enactment of atomic services only requires executing theunderlying component

Composite Services

The ACE enables the enactment of composite servicesThe ACE enables the enactment of composite services

Outline



Analytic Computing EnvironmentBackground

Demo #1: Execution of Composite WorkflowDemo #2: Inspection of WorkflowDemo #3: Modification of Workflow


Application as a Composite Service

Demo #1 is an enactment of a composite service built to mirror thefunctionality of a previous monolithic application

Demo #1 is an enactment of a composite service built to mirror thefunctionality of a previous monolithic application

ACE: Providing Services

This pattern exists for each construct in the ACIThis pattern exists for each construct in the ACI

1. A request message is received by themanager

2. The manager identifies a component inthe component repository that canhandle the request

3. The manager instantiates the componentusing a thread from the thread pool

4. The handler executes

5. Upon completion, the manager runs thecallback function to return the constructproduced by the handler

6. The manager publishes the construct onthe response queue

ACE: Requesting Sub-Services

Temporary message queues enable handlers to receive constructsfrom other services

Temporary message queues enable handlers to receive constructsfrom other services

1. A request message is received by manager A

2. Manager A instantiates handler A

3. Handler A begins execution and sends requestmessage to Manager B

4. Manager B instantiates Handler B

5. Handler B executes and produces construct B

6. Manager B returns construct B on Handler A’stemporary queue

7. Handler A completes execution and producesconstruct A

8. Manager A responds with construct A

ACE: Choreographing Services

Changing the response queues allows composite services withouta centralized controller

Changing the response queues allows composite services withouta centralized controller

1. A request message is received byManager A

2. Manager A instantiates Handler A

3. Handler A completes and producesconstruct A

4. Manager A puts message on ManagerB’s request queue with construct A

5. Manager B instantiates Handler B

6. Handler B completes and producesconstruct B

7. Manager B puts message on Initiator’sresponse queue

Outline



Analytic Computing EnvironmentBackgroundDemo #1: Execution of Composite Workflow

Demo #2: Inspection of WorkflowDemo #3: Modification of Workflow


Workflow Observability

Logging the analytic components produced by an enactmentallows analyst to inspect automated portions of their workflowLogging the analytic components produced by an enactment

allows analyst to inspect automated portions of their workflow

• All messages are logged in an Elastic Search DB

• Through careful querying we can reproduce what messages weregenerated during the workflow and by which component

• It does not matter how the component produced the message

Outline



Analytic Computing EnvironmentBackgroundDemo #1: Execution of Composite WorkflowDemo #2: Observability of Workflow

Demo #3: Modification of WorkflowFuture Work and Summary

Sensitivity Analysis

Analysts can observe the sensitivity of the analytic product tomodifications of intermediate results

Analysts can observe the sensitivity of the analytic product tomodifications of intermediate results

• All services are designed to be independent of other services

• The only thing that ties the services together in a workflow is themessages sent between them

• Modifying the messages can modify the workflow and enable the analystto restart the workflow without re-executing unnecessary components

Outline





Ongoing Work

Ongoing work seeks to enable wider range of workflowssupported by ACS

Ongoing work seeks to enable wider range of workflowssupported by ACS

• (ACI) – Refine and validate constructs• Need more consensus on what the standard constructs should be

• (ACE) – Incorporation of a planner service• Planner service enables user or algorithm to compose a service as

determined by the problem

• (ACS) – Demonstrate interactive composition and execution of workflows• Interactive composition and execution of components is necessary

for ACS to support analyst workflows

Future Work

Future work will focus on enabling ACS to better support existingtools and tradecraft

Future work will focus on enabling ACS to better support existingtools and tradecraft

• (ACL) – Incorporate existing analytic components• Need to incorporate large set of existing manual and automated

analytic techniques

• (ACS) – Integrate with a User Experience framework• Need to determine how ACS supports an analyst’s user experience

• (ACS) – Demonstrate on mission exemplar• Need to determine how ACS affects mission tradecraft

Summary

Have made initial progress on a foundational development effortfor how analytic workflows are composed and executed

Have made initial progress on a foundational development effortfor how analytic workflows are composed and executed

• Proposed initial set of standard analytic constructs to decompose andmodularize analytic components

• Developed prototype service-oriented computing environment to enableanalytic components to be executed as services

• Demonstrated how an analytic tool can be transitioned to the ACS andthe benefits that come from doing so

Questions

Documents

Analytic Component System (ACS) - ncsu-las.org · Storyboard Scene 6: Workflow Capture and Analysis The constructs logged by the ACS provide a rich source of instrumentation data