SYST/OR 699 Master’s Project - Report
Design of a Budget Exhibit Analysis Tool for Analysis of the Department of Defense Research, Development, Test and Evaluation Budget Prepared for Systems Planning and Analysis, Inc.
Chris Anderson, Kris Cruz, Jhonnattan Diaz, Said Masoud 12-9-2016
i
TABLE OF CONTENTS
PROBLEM DEFINITION ........................................................................................ 1
SPONSORSHIP ..................................................................................................... 1
BACKGROUND .................................................................................................... 1
PROBLEM STATEMENT ........................................................................................ 3
PROBLEM AND NEEDS STATEMENT .................................................................... 3
OBJECTIVES AND PURPOSE ................................................................................. 4
STAKEHOLDERS ................................................................................................... 4
SCOPE ................................................................................................................. 6
ANALYSIS ............................................................................................................ 8
REQUIREMENTS ................................................................................................ 10
MISSION REQUIREMENTS ................................................................................. 10
SYSTEM REQUIREMENTS .................................................................................. 10
FUNCTIONAL REQUIREMENTS .......................................................................... 10
DATA EXTRACTION AND ANALYSIS ................................................................... 11
DELIVERABLES ................................................................................................... 13
BUDGET EXHIBIT ANALYSIS TOOL (BEAT) .......................................................... 13
R-1 BUDGET DATA EXPLORATION (PAST YEARS) ............................................... 14
R-2 BUDGET DATA EXPLORATION (FUTURE YEARS) .......................................... 16
FUTURE OPPORTUNITIES .................................................................................. 20
ACCOMPLISHMENTS ......................................................................................... 22
DECISION POINTS .............................................................................................. 23
PROGRAM MANAGEMENT................................................................................ 23
EARNED VALUE MANAGEMENT SYSTEMS ......................................................... 25
REFERENCES ...................................................................................................... 26
APPENDIX A – BUDGET ACTIVITY DEFINITIONS ............................................... A-1
ii
APPENDIX B - LIST OF ACRONYMS ................................................................... B-1
APPENDIX C – USE CASES ................................................................................ C-1
USE CASE 1: RUN R-1/R-2 LOAD BATCH JOB .................................................... C-1
USE CASE 2: RUN EXCEL GENERATOR JOB ........................................................ C-4
USE CASE 3: PERFORM R-1/R-2 DATA VISUALIZATION ..................................... C-7
APPENDIX D – SYSTEM ARCHITECTURE ........................................................... D-1
iii
Table of Figures
Figure 1. DoD Budget from 1950 – 2025 in FY16 dollar (future years estimated) ....................................... 1
Figure 2. Average annual growth of select countries’ R&D budget.............................................................. 2
Figure 3: Federal Spending on R&D .............................................................................................................. 3
Figure 4. Stakeholder Diagram...................................................................................................................... 6
Figure 5. Scope .............................................................................................................................................. 8
Figure 6. Use Case: Perform R-1/R-2 Data Visualization .............................................................................. 9
Figure 7. Sequence Diagram: OV-6C Data Visualization ............................................................................... 9
Figure 8. Requirements Diagram ................................................................................................................ 11
Figure 9. R-1 Budget Data for a specified President's Budget year ............................................................ 12
Figure 10. R-2 XML document example ...................................................................................................... 12
Figure 11. BEAT system R-1 graphing options ............................................................................................ 15
Figure 12. Spark Charts within BEAT (Smart Charts) .................................................................................. 15
Figure 13. DARPA drill-down to show PE's within ...................................................................................... 16
Figure 14. R-2 excerpts from PB13-PB17, identifying FY17 ........................................................................ 17
Figure 15. R-2 Graph of Electronic Warfare Technology ............................................................................ 17
Figure 16. BEAT (R-2) system user-interface .............................................................................................. 18
Figure 17. Advanced Aerospace Systems data for FY17 by PB submission year ($ Millions) ..................... 19
Figure 18. Multiple program elements graphed ......................................................................................... 20
Figure 19. Analysis ACROSS BAs, Services, Agencies .................................................................................. 21
Figure 20. Excerpt from PB17, showing FYDP+ ........................................................................................... 22
Figure 21. Excerpt from R-2, showing Accomplishment description .......................................................... 22
Figure 22. BEAT Work Breakdown Structure .............................................................................................. 24
Figure 23. Earned Value Management ....................................................................................................... 25
Figure 24. Use Case 1 - Run R-1/R-2 Load Batch Job ................................................................................. C-2
Figure 25. Sequence Diagram: OV-6c Run Data Load Batch Job................................................................ C-3
Figure 26. Use Case 2 - Run Excel Generator Job ...................................................................................... C-5
Figure 27. Sequence Diagram: OV-6c Run Excel Generator Job ................................................................ C-6
Figure 28. DIV-1 Conceptual Data Model ................................................................................................. D-1
Figure 29. OV-5b: Operational Activity Model .......................................................................................... D-2
Table of Tables
Table 1. Categories of Appropriation ............................................................................................................ 7
Table 2. Budget Activity Numbers and Names ............................................................................................. 7
Table 3. Use Case 1 - Success Criteria ........................................................................................................ C-1
Table 4. Use Case 1 - Scenario Extensions ................................................................................................. C-2
Table 5. Use Case 2 - Success Criteria ........................................................................................................ C-4
Table 6. Use Case 2 - Scenario Extensions ................................................................................................. C-4
Table 7. Use Case 3 - Success Criteria ........................................................................................................ C-7
Table 8. Use Case 3 - Scenario Extensions ................................................................................................. C-8
1
Problem Definition
Sponsorship
Systems Planning and Analysis, Inc. (SPA) is a professional services contractor located in Alexandria, VA.
SPA has permitted the Joint Air Force & Analysis Division (JTAAD), and specifically the Operational Support
Analysis Group (OSAG) to become the first sponsor of an educational capstone project. The problem
statement outlined below has been derived from conversations with OSAG, the official client of this
project. OSAG is trying to search out new analytical methods and approaches to categorizing and
analyzing past, present, and future budget submissions. The proposal offered by OSAG was selected as a
capstone project for our team
Background
Since 2000, the budget of the Department of Defense (DoD) has ebbed and flowed (See Error! Reference
source not found.) in line with national security concerns and overseas military activities. For example,
after 9/11, the DoD R&D budget increased by approximately 60% through FY2007, but declined 25%
between FY2010-FY2014, coinciding with the drawdown of American forces and presence in Iraq and
Afghanistan. (American Association for the Advancement of Science 2016)
Figure 1. DoD Budget from 1950 – 2025 in FY16 dollar (future years estimated)
Formally announced in 2014 by then Secretary of Defense Chuck Hagel, one major focus of the DoD has
been the third offset strategy. Third offset is an effort to maintain technological superiority; focusing on
such activities as but not limited to; Human-machine learning, Robotics, Undersea warfare, Hypersonics,
semi-autonomous weapons, resilient and/or alternative methods of communication, and cyber
Supplemental Funding Base Funding
2
capabilities (both defensive and offensive). There is also an emphasis on agility and sustainability in
manufacturing new systems, decreasing the logistical load of deployment by utilizing techniques such as
modular manufacturing and 3D printing.
In December of 2015, Deputy Secretary of Defense (DEPSECDEF) Bob Work stated that 2017 will be an
important year to lay the groundwork for this strategy. (American Association for the Advancement of
Science 2016)
Third offset strategy represents more than $18 billion of budget requests across the Future Years Defense
Program (FYDP), constituting an overhaul in US military operating concepts and technological
investments. The concepts and investments are intended to address emerging capability gaps as well as
deter potential near-peer adversaries, such as China and Russia. (.govini Accessed 2016).
These strategic goals, however, are being pursued in an increasingly constrained fiscal environment. The
Budget Control Act of 2011 created several mechanisms for enforcing stricter control of spending,
including sequestration, and its effects can be seen in the flat future budget projections of the final years
depicted in Figure 1. DoD Budget from 1950 – 2025 in FY16 dollar (future years estimated).
The US no longer has a monopoly on power projection, precision guidance, or the enabling infrastructure that made it the dominant military power since the end of the Cold War. China and Russia have observed the US’s capabilities and have been making strategic investments that counter US advantages. These efforts are evidenced by the average annual R&D growth of these two countries, which is significantly greater than that of the US (See Figure 2. Average annual growth of select countries’ R&D budget). (.govini Accessed 2016)
Figure 2. Average annual growth of select countries’ R&D budget
3
The domain of this project is limited to Research and Development (R&D) budgeting within the
Department of Defense. The DoD typically accounts for one-half of the federal Research and Development
budget – approximately $76.6 billion in 2016 (See Figure 3: Federal Spending on R&D).
Figure 3: Federal Spending on R&D
Problem Statement
Problem and Needs Statement
Despite the magnitude of DoD R&D investment, there is currently no known tool for housing and
visualizing budget data. The data are dispersed into many separate documents and locations. Manual
formatting of data is time-consuming, and there is no efficient way to isolate and visualize data meeting
a set of user-specified criteria.
Currently, if one wanted to visualize a forecasted budget trend over time using Future Years Defense
Program (FYDP) projected data, they would need to take the following steps:
Download one R-2 document from Defense Technical Information Center (DTIC) website for each year in the 5-year RDT&E budget cycle
o These documents are usually several hundred pages long, as they contain information on the progress and future goals of each Program Element (PE)
Find the PE of interest within each document
Export budget data to a program (such as Microsoft Excel) for visualization
Repeat process for multiple PEs if comparison is desired
4
This is a time-consuming process. So, while it is possible to track a PE budget over time by accessing budget
documents for each year, there is no known way to access that information rapidly or isolate PEs which
meet a set of user-specified criteria without searching for them manually.
The purpose of the Budget Exhibit Analysis Tool (BEAT) is to meet this need by allowing users to quickly
identify PEs of interest using historic data and visualize their individual budget trends over time using
Future Years Defense Program (FYDP) projected data.
Objectives and Purpose
The objective of this project is to develop a methodology to analyze past, present, and future budget
trends within the DoD S&T community. Different stakeholders will approach the DoD RDT&E budget with
different questions depending on their role and areas of interest. It is of paramount importance to the
team to develop a decision support system with parameters that can be set by the user to provide a
tailored analysis. This tool must also be able to be updated with new data as it becomes available. The
requirements of this tool as defined by the team, and agreed upon by the sponsor, are defined in the
Requirements section of this document.
Stakeholders
There are many stakeholders in the DoD budgeting process. Some of these stakeholders are decision
makers who are responsible for determining the budget, others are responsible for allocating it, and most
are the recipients of those allocated funds.
It should be noted that, with the diversity of interests at work within the budgeting process, not all of
them are of a nature such that they would consider this project of value. For example, someone who is
responsible for only one project and whose primary responsibilities are to document that project’s
progress and justify future funding may not be interested in trend analysis.
The primary stakeholders – those who would consider this project of value to them – are those who are
interested in budget trend analysis. This might include, for example, a company which is considering how
to best allocate its resources to projects which are likely to receive increased attention and funding in the
future.
The following is an overview of the stakeholders and their roles in the DoD budgeting process:
The President and Congress
The DoD submits an annual budget request to the President for review. The President then submits a
budget request to Congress. This request is reviewed and subjected to an iterative process of revisions
before it is returned to the President, at which point it is either signed into law or vetoed.
Understanding trends within R&D is vital for decision makers to be able to rally support for the
expenditures they support or against the expenditures they oppose.
5
Services (Department of the Navy, US Air Force, US Army)
The primary organizations through which military policy is carried out, the services each have research
laboratories which are organized under them. They are the principal end users of the technologies
developed, and are therefore essential for determining the usefulness of any technology.
Communities of Interest (COIs)
A network of technical groups that serve as a lasting structure to integrate the diverse technological
efforts throughout the DoD. COIs were established in 2009 as a mechanism to encourage multi-agency
coordination and collaboration in cross-cutting technology focus areas with broad multiple-component
investment (Research & Engineering Enterprise: EXCOM 2016).
Having a powerful analytical tool at their disposal will enable COIs to better facilitate the coordination of
R&D efforts across the DoD and monitor the general state of development for specific areas of technology.
Industry
A diverse collection of private entities whose interests and level of involvement with the DoD vary
significantly. Some are recipients of contracts for specific research projects, while others are contracted
to provide an array of consulting services.
Naturally, industry players are in competition with one another for the limited resources apportioned to
the services they provide. Being able to identify and predict future budgeting developments is invaluable
as companies seek to determine how to best allocate their resources and direct their future endeavors.
Federally Funded Research & Development Centers (FFRDCs), and University-Affiliated Research Centers
(UARCs)
Research partnerships in which the DoD is the sponsor of a research facility which is under the
administration of either a corporate entity (FFRDCs) or a university (UARCs).
Office of the Secretary of Defense (OSD)
The Secretary of Defense is the primary defense policy advisor to the President and is responsible for
formulating and executing policy through the DoD. The OSD, as the staff of the Secretary, shares in those
responsibilities by providing management and oversight for the numerous components of the DoD.
Leadership and decision making effectiveness can be improved with the development of decision support
tools.
The Assistant Secretary of Defense (Research & Engineering) – ASD(R&E) is one component of the OSD. It provides S&T leadership and strengthens coordination efforts
Science and Technology (S&T) Laboratories
In addition to the laboratories organized under the Army, Navy, and Air Force, the DoD allocates funding
to other laboratories, some of which are independent (directly under the DoD), and some of which are
within other DoD agencies such as DARPA.
6
Figure 4. Stakeholder Diagram
Lastly, it is important to grasp the profound impact that even slight improvements to information
accessibility and visibility of trends can have. It is easy, but incorrect, to look at each year’s budget as the
product of a brief decision process with a definite beginning and end. In actuality, determining the
future budgets for the many components of the DoD is an ongoing undertaking that many of the
different stakeholders participate in. Also, the description of stakeholders in this report is greatly
simplified, and there is a multidimensionality to the organizations represented, with organizations each
having several levels of decision makers interested in identifying budget trends. Given the interminable
nature of the budgeting process, and the sheer number of people involved in it, the benefit of increased
access to information and greater visibility of trends cannot be understated.
Scope
There are natural scoping mechanisms built into the budget nomenclature and taxonomy used to build
the budget documents. The first scoping mechanism is the separation of Research, Development, Test,
and Evaluation (RDT&E) funding (See Table 1. Categories of Appropriation) from the other types of
funding. There are five major categories of appropriation:
7
Table 1. Categories of Appropriation
Appropriation Category Scope Funding Policy
Research Development, Test, and Evaluation (RDT&E)
Activities and expenses related to research, development, test, and evaluation
Incremental
Procurement (PROC) End items >= $100K system unit cost, All centrally managed items, Initial spares, Labor for certain production-related functions (e.g., item assembly, quality assurance)
Full
Operations and Maintenance (O&M)
Replenishment spares, Fuel, Civilian salaries, Construction projects < $750K, Travel, Non-centrally managed end items < $100K system unit cost
Annual
Military Personnel Expenses (MILPER)
Military Personnel Expenses Annual
Military Construction (MILCON) Construction Projects >= $750K Full
S&T funding is primarily within the RDT&E funding component. We therefore chose to examine only that
component of the budget.
Furthermore, by examining the categories under RDT&E, we can further determine the budget categories
that are likely to determine the most important trends in DoD research funding. There are seven BA’s
that define the total of the RDT&E budget. (See Table 2. Budget Activity Numbers and Names)
Table 2. Budget Activity Numbers and Names
Budget Activity Number Budget Activity Name
1 Basic Research
2 Applied Research
3 Advanced Technology Development
4 Advanced Concept Development & Prototypes
5 System Development and Demonstration
6 RDT&E Management and Support
7 Operational System Development
The scope of this project will reduce the research and analysis of all seven budget activities to just three
of them; BA-2: Applied Research, BA-3: Advanced Technology Development, BA-4: Advanced Concept
Development and Prototypes.
The following BAs will be out of scope for this project:
BA-1: Basic Research does not immediately yield a technology that could be produced to assist the warfighter
BA-5: System Development and Demonstration include programs having passed Milestone B within the acquisition process leading to full rate production
BA-6: RDT&E Management and Support includes funds to sustain and/or modernize installations
8
BA-7: Operational System Development includes the developmental efforts to upgrade systems that have previously been fielded or received approval for full rate production
Figure 5. Scope
Analysis
After obtaining input from the project sponsor, a preliminary concept of operations (CONOPS) was
developed to elicit requirements for the BEAT system. In addition to the input from sponsors, use cases
were developed to formalize the CONOPS, and the viability of the system design.
We present a sample of a use case [Perform R-1/R-2 Data Visualization] to show the main success scenario
for displaying data to the analyst on demand. The included case has to do with the realization that the
add-on tool part has to be running concurrently with MS Excel. The extend case delineates what happens
when the tool does not find a Program Element in the query list (See Figure 6. Use Case: Perform R-1/R-2
Data Visualization). Other use cases were developed in order to account for the functionality of the BEAT
tool, and can be found in the appendix.
9
Figure 6. Use Case: Perform R-1/R-2 Data Visualization
In addition to the use cases, sequence diagrams were developed from the use cases to make sure that the
order of operations made sense with the dynamics of the tool (See Figure 7. Sequence Diagram: OV-6C
Data Visualization). The user launches the Smart Chart Tool, so the ExcelGeneratorJob can populate the
R-1, and R-2 data that the ExcelVisualizationTool can graph and display the data to the analyst depending
on the his/her input criteria for the query. Additional sequence diagrams for specific use cases can be
found in the appendix.
Figure 7. Sequence Diagram: OV-6C Data Visualization
10
Requirements
Mission Requirements
MR1: The BEAT system shall be able to ingest RDT&E budget information from R-1 and R-2 documents.
MR2: The BEAT system shall have the ability to ingest a minimum of five years of RDT&E budget exhibit documents, when data is available
MR3: The BEAT system shall visualize RDT&E budget data over a pre-determined time period
System Requirements
SR1: The BEAT system shall generate R-1 budget trend data by Program Element for budget activities 2, 3, and 4
SR2: The BEAT system shall generate R-2 budget trend data by Program Element for budget activities 2, 3, and 4
SR3: The BEAT system shall Aggregate budget data by Program Element for budget activities 2, 3, and 4
Functional Requirements
FR1: The BEAT system shall create a plot that visualizes “outliers” of R-1 budget data
FR2: The BEAT system shall identify “outliers” that exist within currently loaded data and newly loaded data
FR3: The BEAT system shall identify new Program Elements by first fiscal year reported
FR4: The BEAT system shall identify discontinued Program Elements by last fiscal year reported
FR5: The BEAT system shall Monitor a minimum of five RDT&E budget cycles, when data is available
FR6: The BEAT system shall aggregate Program Elements in order to conduct tailored analysis
11
Figure 8. Requirements Diagram
Data Extraction and Analysis
The data used for the purposes of this project is available to the public, and is obtainable at the website
of the Defense Technical Information Center (Defense Technical Information Center 2016). Two types of
documents were used to extract and analyze data. For the purposes of this project, only unclassified data
was retrieved.
R-1: The R-1 document contains budget information for all RDT&E programs undertaken by the branches of the DoD. For each fiscal year, the document lists the budget for the previous two fiscal years, as well as the budget for the current fiscal year. See Figure 9. R-1 Budget Data for a specified President's Budget year for an example of an R-1 document.
12
Figure 9. R-1 Budget Data for a specified President's Budget year
R-2: The R-2 document is specific to each program, identified by a “program element” (PE) number in the R-1. The R-2 serves as the item justification for each program, as the proposed budget for the next four fiscal years is listed. In addition, a detailed description of each sub-program covered by the PE is included. See Figure 10. R-2 XML document example for an example of an R-2 document.
Figure 10. R-2 XML document example
The R-1 and R-2 documents are available in Excel format and XML format, respectively, among others. The
data was extracted from the two file types via a Java batch application, and loaded into a MySQL database.
The R-2 documents all have the same schema; however, the format of the columns are slightly different
in the R-1 documents for each fiscal year, therefore some manual manipulation was required before the
documents were loaded via the batch application.
13
In the database, the data from the R-1 documents was loaded into a single table, using the PE number
and the fiscal year reported as the unique identifier for each row. The schema for the R-2 documents is
similar to the R-1 schema where the fiscal year reported and PE number are the unique identifiers for the
R-2 table.
Once the data was loaded into the database, the raw data was extracted and loaded into an Excel template
by the Java batch application. In order to visualize the respective trends, Excel graphs and other visuals
were developed and generated automatically using pre-defined Visual Basic for Applications (VBA) scripts
that were automatically executed after the raw data was loaded into the template. Within the Java
application, templates containing pre-formatted Excel sheets and VBA scripts were packaged together,
and, in the future, when new R-1 or R-2 documents are loaded into the database, the results can be loaded
into new files via the templates for analysis and visualization. The VBA scripts will automatically be
executed by the Java application, such that when the application has completed running, an end user can
simply open the Excel file created and instantly see the results.
In order to allow the sponsor to have the option to continue to use the application, a separate user
manual, which explains how to set up the database and Java batch application, was developed. This
manual contains a description of the Java batch application, as well as instructions to set up a MySQL
database, if desired.
Deliverables
The following items will be delivered to the stakeholder at the conclusion of this project:
Java Executable (.jar): The Java batch application will be compiled into an executable JAR file that can be readily executed, with the modification of the application properties
Java source code: The Java source code will be saved as an Eclipse project and compressed for more efficient delivery. In the event that the sponsor would like to modify or build on the code, the sponsor can download and set up Eclipse, then import the project directory.
MySQL database export: A MySQL database export will be provided so that the sponsor does not have to worry about setting up the database manually and trying to load the data that was already loaded during the course of this project. The export will be formatted such that if the sponsor chooses to use another type of database, the data can still be imported without much effort.
Excel templates and completed data sheets: The compiled data sheets used to develop the results of the analysis will be provided to the sponsor so that they may continue to perform analysis on the data. In addition, Excel templates will be provided in the Java JAR file and Eclipse project package so that when new data is available, they may create new Excel sheets for analysis.
User Manual: A user manual that will provide instructions on how to set up a MySQL database and implement the Java batch application.
Budget Exhibit Analysis Tool (BEAT)
BEAT was developed to aid in the identification of budget request trends within the RDT&E budget. The
BEAT system has the ability to upload new President’s Budget documents when they are released
(annually). The file formats capable of being uploaded are MS Excel (R-1) and XML (R-2). Previously
analysts would have to search on disparate sites for the specific documents in question, download them,
14
unpack (WinZip) then, and manually combine them into usable formats for their research. While the
downloading takes a long time, and will still be necessary with the BEAT system, the most time-consuming
event is in the compilation of the data.
The BEAT system is comprised of over 1,000 lines of code. It is this code that BEAT will use to extract,
transform, and load the documents in a user-friendly format into the BEAT system (MS Excel Application).
The system then runs “super-macros” (leading code that will run the pertinent routines in the proper
order) in order to finalize the transformation of the data using MS Excel VBA. This process happens for
both the R-1 and R-2 data.
BEAT is a tool that is used in two stages. The first stage utilizes the R-1 data from the President’s Budget
files. The R-1’s were collected from the Defense Technical Information Center (DTIC). Data collection
resulted in files from the year 1998 until 2017. As stated in the previous section, the R-1 data contains
budget information for the requested year, and the two previous years; hence will be used to identify
historic trends in budget requests.
The second stage utilizes the R-2 data from the President’s Budget. The R-2’s provide greater detail than
the R-1’s. The R-2’s provide the current year request, the two previous years of spending data, and four
years worth of projections for a specific program element. BEAT uses the future data to show users how
projections and requests change from fiscal year to fiscal year, over sequential President Budget
submissions.
In order to identify historic trends the BEAT system uses an Excel Add-in called Smart Charts. Smart Charts
was developed by a company called DevScope and downloaded from the Windows App Store. The cost
for Smart Charts was $5.00. Smart Charts allows for time-series data to be graphed in multiple manners
and includes a dynamic aspect. The charts selected can be filtered dynamically and a time-series video
can be played to show how the data changes over time; in this case the time factor is the President’s
Budget submission years.
R-1 Budget Data Exploration (Past Years)
The BEAT system can show data in multiple ways: Tree Maps, Pie Charts, Bar Graphs, and Bubble Charts
(See Figure 11. BEAT system R-1 graphing options). At a macro level, this tool is meant to show you how
the service’s or agency’s share of the budget changes over time. We also used this to show the annual
change of appropriated funds from year to year.
15
Figure 11. BEAT system R-1 graphing options
Identifying services or agencies of interest is based on the user’s research. Hovering over a service or
agency will show the user a spark chart of the data over time. Based on the filters set in Figure 12. Spark
Charts within BEAT (Smart Charts), the user is able to identify the absolute change in budget requests
from the President’s Budget submissions from 1998-2017. This hovering spark chart allows the user to
quickly scroll through the chart to identify the service or agency of interest.
Figure 12. Spark Charts within BEAT (Smart Charts)
16
By selecting DARPA, you will be able to see the program elements DARPA is funding. The filters were
changed in “Figure 13. DARPA drill-down to show PE's within” to show the absolute change in budget
requests, and includes each program element title. Again, you may hover over the program element
names in order to show a spark chart of the funding requests within that specific program element.
Selecting a program element will show that specific program element only, which may not be useful as it
will be the only item on the chart.
Figure 13. DARPA drill-down to show PE's within
R-2 Budget Data Exploration (Future Years)
After identifying a program element of interest, the user can begin stage two of the BEAT system by
delving into the projections of the R-2 budget documents. The R-2 portion of the BEAT system was
developed using MS Excel and VBA and provides visual charts to show the user how the projections for a
specified fiscal year change as the fiscal year approaches. As shown in “Figure 14. R-2 excerpts from PB13-
PB17, identifying FY17” the projections for fiscal year 2017 increased approximately 60% over 5 years;
from $15.839 Million in 2013 to $25.466 Million in 2017. These are the trends shown in stage 2 of the
BEAT system in “Figure 15. R-2 Graph of Electronic Warfare Technology”.
17
Figure 14. R-2 excerpts from PB13-PB17, identifying FY17
Figure 15. R-2 Graph of Electronic Warfare Technology
18
Identifying changes in projected funding can assist stakeholders in identifying the key technologies that
are gaining funding. This will assist companies trying to gain a foothold in federal contracting by giving
them the insights needed to identify key opportunities for their business units.
There are over 500 program elements to choose from, so it is important to make the tool user-friendly.
The R-2 section of the BEAT system allows the user to search and filter to see only the data requested.
“Figure 16. BEAT (R-2) system user-interface” shows the filtering options available to the user. The user
can filter the program elements based on the fiscal year of interest (default is 2017) which is the year
whose value will be shown as requested across President’s Budget submissions. The user can also filter
by Service or Agency (e.g. Army, DARPA, etc.), by budget activity (i.e. BA2 – BA4), and/or use the text
search capability. The listing of program elements will dynamically filter itself upon each selection
showing the user the program elements based on the filters selected.
Figure 16. BEAT (R-2) system user-interface
Once the user has selected the program element(s) to research further, the user presses the “OK” button.
That will populate the graph with the selected program elements using the data requested. As shown in
“Figure 17. Advanced Aerospace Systems data for FY17 by PB submission year ($ Millions)”, the user will
be presented with the graph, displaying a legend of the selected program elements, data markers to alert
the user to the amount depicted, and specific markers. The markers alert the user to the specific budget
19
activity of the program element (e.g. Square = Budget Activity 2, Square = Budget Activity 3, and Triangle
= Budget Activity 4). This will aid in discerning the data on the graph without cluttering it.
Null values have been taken out of the dataset, as they are translated as zero values within the dataset.
As such they were skewing the data, and could lead the user to believe that a budget request was made
within the President’s Budget for that year and no money appropriated to the program element. Values
that are actually zero, will be portrayed within the chart.
Figure 17. Advanced Aerospace Systems data for FY17 by PB submission year ($ Millions)
The BEAT system has the ability to graph multiple selections (i.e. program elements) simultaneously. This
is a first step toward the future work recommendations presented in the next section of this report. As
shown in “Figure 18. Multiple program elements graphed” filters were used to select items that were
alike. The graph on the upper-left portion of Error! Reference source not found.was filtered using the
text search functionality. The search was performed on “LCS” (Littoral Combat Ship), bringing up two
results: LCS mission modules and Littoral Combat Ship (LCS). Both selections are budget activity 3 and
Navy programs. The graph on the lower-right portion of Figure 18 used program elements selected using
multiple filters; the Service / Agency filter and the Budget Activity filter. The agency selected was DARPA
and the budget activity selected was BA 3. Out of the eight program elements remaining (post-filtering),
four were selected: Advanced Aerospace Systems, C3 Systems, Network-Centric Warfare Technology, and
Space Programs & Technology. While these are just examples of the types of graphs that can be created,
20
based on the user’s filtering preferences, more in-depth case studies were performed in order to provide
a proof of concept for the BEAT system.
Figure 18. Multiple program elements graphed
Future Opportunities
While we are able to graph the program elements from different services and budget activities on the
same chart, there are much more in depth and cross-program analyses that are possible. Building the
macros to allow such analysis would be a good future project for this class or could be done by the
sponsor. It allowed us to show a proof of principle (Figure 19. Analysis ACROSS BAs, Services, Agencies).
The future of this tool will be to take the analysis ACROSS budget activities and services / agencies.
21
Figure 19. Analysis ACROSS BAs, Services, Agencies
Program Elements are the top layer of the “analytical onion”. Underneath program elements, there are
projects listed within (one or more). Each project, in turn, has multiple accomplishments (one or more)
that comprise the project. This layering effect provides more detail and different information the further
you drill down. For an example, see below:
PE: 0603286E / Advanced Aerospace Systems (DARPA)
Project: AIR-01 / Advanced Aerospace Systems o Accomplishments:
Tactically Exploited Reconnaissance Node (TERN) Collaborative Operations in Denied Environments (CODE) Hypersonic Air-breathing Weapon Concept (HAWC) Tactical Boost Glide Advanced Aerospace System Concepts Technology for Enriching and Augmenting Manned-Unmanned Systems Vertical Take-off and Landing (VTOL) Technology Demonstrator Distributed Fires Advanced Full Range Engine (AFRE)
The data currently displayed within the BEAT system lends itself well to data analytics. Future work
involves the projection beyond the FYDP (four years in the future), (Figure 20. Excerpt from PB17, showing
22
FYDP+). There is an abundance of data that can be used as the training data to test the model or models
developed. Projections could be used to alert users to trends of interest (i.e. increase, decrease, or steady
funding profiles). The trends identified will be useful to government contractors looking to identify the
lines of business that represent opportunities to remain in, research the feasibility of getting in, and
research the feasibility of divesting. The opportunities identified relate directly to the trends of the
Science & Technology fields and whether they are increasing, decreasing, or remaining steady.
Figure 20. Excerpt from PB17, showing FYDP+
Another opportunity for future work would be delving deeper into the data within the R-2 budget exhibits
using machine learning techniques. The data within the R-2 budget documents below the program
element level contains more diverse data than at the program element level. The projects and
accomplishments have much more verbiage defining them. This verbiage lends itself to natural language
processing, support vector machine (SVM), and K-Nearest Neighbors (KNN) models (See Figure 21. Excerpt
from R-2, showing Accomplishment description). The output analysis of these types of models could alert
the user to specific projects and/or accomplishments relating to their areas of expertise, for further
research by the user.
Figure 21. Excerpt from R-2, showing Accomplishment description
Accomplishments
As the BEAT System team, we have been successful in many ways. We have been able to provide a proof
of concept tool for our sponsor that drastically reduces the amount of time spent parsing and compiling
the disparate files related to the R-1 and R-2 budget exhibits. The bulk of the “set-up” time will still be
the downloading of the documents. The R-1 documents are readily available in MS Excel format on
Comptroller’s website (http://comptroller.defense.gov/Budget-Materials/#detailed), but the time-
consuming portion will be the download of the R-2 budget documents. The R-2 budget documents are
housed in many locations. Services (i.e. Navy, Air Force, Army) are posted on the service specific
comptroller website grouped by budget activity, while the agencies (i.e. DARPA, MDA, etc.) are posted in
multiple documents indexed by their respective budget activity levels (Defense Technical Information
Center 2016). The documents to download are in .PDF format, with the appropriate .XML files attached
to the parent document as a .ZIP file.
23
Once the files are unzipped, and located on the computer of the analyst, they can easily be run through
the extract, transform, and load (ETL) phases. This portion of the BEAT system does the parsing and
compilation work of the analyst. This process is the MASSIVE time-saver that will allow the analyst to
concentrate his/her time on the analysis and research portion of their work.
Trends are easily identified within the BEAT system using the graphing capabilities and custom user-
interfaces within the tool. The analyst has many options, in terms of filters, to gain valuable information
quickly of the processed data. This allows the analyst to customize the work being completed, and back
up the analysis with graphs, solidifying their position within the analysis.
Decision Points
The BEAT system team has validated the intent of this project and is able to offer decision points for the
sponsor. The sponsor will need to identify the next analytic opportunity based on the decision points
chosen or creating their own vision of where to take this tool. Options for the continuation of this project
include:
Deep dive into the R-2 budget exhibits o Projects o Accomplishments o Plans
Data Models o Natural Language Processing (descriptions) o Forecasting (FYDP+)
Program Management
After requirements elicitation was completed with the customer, and agreed upon, a Work Breakdown
Structure (WBS) was developed to gauge the price of the contract. The Statement of Work (SOW) will
serve as a delimiter for the work to be performed.
The cost for the entirety of the project was estimated to be $66,446.81 and designated as a non-fixed cost
contract. Any changes by the customer after the SOW has been signed, such as additional work, change
of requirements, change in the schedule; will change the cost of the project, and a new SOW shall be
drafted, and signed. There are 30 hours allocated in the contract to meet with knowledge experts, and
consultants at a rate of $300.00 per hour. If extra consultant hours are needed, the developer will need
authority to proceed from the customer.
This is the Work Breakdown Structure developed to plan, control, and integrate the BEAT concept. There
are four major areas for the successful completion of the project as follows:
Analysis
Design
Implementation
Verification & Validation (V&V)
24
In the Analysis phase, special attention was devoted to the project overview in order to get a better idea
of what needed to be developed. Early collaboration with the sponsor was a major aid in the notion of
what the customer expects from the project team. Also, in this phase, being able to define the problem
with those who would eventually use the tool was beneficial in eliciting requirements.
There were three Customer meetings scheduled to review the progress being made by the developer of
the project. Customers are expected to interact with the developers for purposes of achieving the
requirements of the project.
Figure 22. BEAT Work Breakdown Structure
In the design phase, a CONOPS was developed and later on formalized with the aid of Use Cases derived
from the customer. Once the use cases were developed, formal Mission requirements emerged, and later
on systems requirements flowed naturally to the lower levels. Once a firm base of requirements was
established, the data model was developed to form the structure of the BEAT tool.
In the Implementation phase, a database was designed to hold the information researched from the
different Program Elements (PE) that will reside in the R-1 & R-2 data tables. The database was tested to
make sure its components worked correctly, and that CRUD operations could be performed. Also, the
visualization capabilities were developed, thereby satisfying one of the main components of the
requirements.
In the V&V phase, the operations of the database were tested with the data integrity and validation of
the visualization. After correct operation of the database was assessed as working as expected, the data
25
analysis tools were designed to make the reports and visualization user-friendly. Each of these
components was tested by the use cases.
Earned Value Management Systems
It was important for the group to perform EVMS throughout the life time of project to ensure that cost,
and schedule were track. At the beginning of the project, a non-fixed contract was accorded with the
customer for roughly $62,000 for a period of 15 weeks.
Figure 23. Earned Value Management
Although it was not required to implement EVMS for this project, the team found it imperative to have
transparency with the customer on how their money was spent, and that the project was on schedule in
the systems development life cycle.
A customer representative shall be designated to communicate with the customer and any questions shall
be directed to this person.
26
References
.govini. Accessed 2016. "US Department of Defense Third Offset Standard Market Taxonomy." Analyst
Report, Washington, D.C.
American Association for the Advancement of Science. 2016. Guide to the President's Budget Research &
Development FY2017. Washington, D.C.: AAAS.
Belasco, Amy. 2015. Defense Spending and the Budget Control Act Limits. US Defense Policy and Budget,
Washington, D.C.: Congressional Research Service.
Defense Technical Information Center. 2016. August 30. Accessed August 30, 2016.
http://www.dtic.mil/dodinvestment/#/home.
—. 2016. September 17. Accessed September 17, 2016.
http://www.dtic.mil/dodinvestment/html/Budget_Activity_RDDS.pdf.
2016. Research & Engineering Enterprise: EXCOM. Accessed October 3, 2016.
http://www.acq.osd.mil/chieftechnologist/COIs.html.
Rugy, Veronique de. 2013. High Levels of Government Spending Become Status Quo. Fairfax: Mercatus
Center. http://mercatus.org/sites/default/files/update-federal-spending-per-capita-analysis.pdf.
Systems Planning and Analysis, Inc. 2016. Systems Planning and Analysis, Inc. September 17. Accessed
September 17, 2016. www.spa.com.
A-1
Appendix A – Budget Activity Definitions
(Defense Technical Information Center 2016)
Budget Activity 1, Basic Research
Basic research is systematic study directed toward greater knowledge or understanding of the
fundamental aspects of phenomena and of observable facts without specific applications towards
processes or products in mind. It includes all scientific study and experimentation directed toward
increasing fundamental knowledge and understanding in those fields of the physical, engineering,
environmental, and life sciences related to long-term national security needs. It is farsighted high payoff
research that provides the basis for technological progress. Basic research may lead to: (a) subsequent
applied research and advanced technology developments in Defense-related technologies, and (b) new
and improved military functional capabilities in areas such as communications, detection, tracking,
surveillance, propulsion, mobility, guidance and control, navigation, energy conversion, materials and
structures, and personnel support. Program elements in this category involve pre-Milestone A efforts.
Budget Activity 2, Applied Research
Applied research is systematic study to understand the means to meet a recognized and specific need. It
is a systematic expansion and application of knowledge to develop useful materials, devices, and systems
or methods. It may be oriented, ultimately, toward the design, development, and improvement of
prototypes and new processes to meet general mission area requirements. Applied research may
translate promising basic research into solutions for broadly defined military needs, short of system
development. This type of effort may vary from systematic mission-directed research beyond that in
Budget Activity 1 to sophisticated breadboard hardware, study, programming and planning efforts that
establish the initial feasibility and practicality of proposed solutions to technological challenges. It includes
studies, investigations, and non-system specific technology efforts. The dominant characteristic is that
applied research is directed toward general military needs with a view toward developing and evaluating
the feasibility and practicality of proposed solutions and determining their parameters. Applied Research
precedes system specific technology investigations or development. Program control of the Applied
Research program element is normally exercised by general level of effort. Program elements in this
category involve pre-Milestone B efforts, also known as Concept and Technology Development phase
tasks, such as concept exploration efforts and paper studies of alternative concepts for meeting a mission
need.
Budget Activity 3, Advanced Technology Development (ATD)
This budget activity includes development of subsystems and components and efforts to integrate
subsystems and components into system prototypes for field experiments and/or tests in a simulated
environment. ATD includes concept and technology demonstrations of components and
subsystems or system models. The models may be form, fit and function prototypes or scaled models that
serve the same demonstration purpose. The results of this type of effort are proof of technological
feasibility and assessment of subsystem and component operability and producibility rather than the
development of hardware for service use. Projects in this category have a direct relevance to identified
military needs. Advanced Technology Development demonstrates the general military utility or cost
A-2
reduction potential of technology when applied to different types of military equipment or techniques.
Program elements in this category involve pre-Milestone B efforts, such as system concept
demonstration, joint and Service-specific experiments or Technology Demonstrations and generally have
Technology Readiness Levels of 4, 5, or 6. Projects in this category do not necessarily lead to subsequent
development or procurement phases, but should have the goal of moving out of Science and Technology
(S&T) and into the acquisition process within the future years defense program (FYDP). Upon successful
completion of projects that have military utility, the technology should be available for transition.
Budget Activity 4, Advanced Component Development and Prototypes (ACD&P)
Efforts necessary to evaluate integrated technologies, representative modes or prototype systems in a
high fidelity and realistic operating environment are funded in this budget activity. The ACD&P phase
includes system specific efforts that help expedite technology transition from the laboratory to
operational use. Emphasis is on proving component and subsystem maturity prior to integration in major
and complex systems and may involve risk reduction initiatives. Program elements in this category involve
efforts prior to Milestone B and are referred to as advanced component development activities and
include technology demonstrations. Completion of Technology Readiness Levels 6 and 7 should be
achieved for major programs. Program control is exercised at the program and project level. A logical
progression of program phases and development and/or production funding must be evident in the FYDP.
Budget Activity 5, System Development and Demonstration (SDD)
SDD programs have passed Milestone B approval and are conducting engineering and manufacturing
development tasks aimed at meeting validated requirements prior to full-rate production. This budget
activity is characterized by major line item projects and program control is exercised by review of
individual programs and projects. Prototype performance is near or at planned operational system levels.
Characteristics of this budget activity involve mature system development, integration and demonstration
to support Milestone C decisions, and conducting live fire test and evaluation and initial operational test
and evaluation of production representative articles. A logical progression of program phases and
development and production funding must be evident in the FYDP consistent with the Department’s full
funding policy.
Budget Activity 6, RDT&E Management Support
This budget activity includes research, development, test and evaluation efforts and funds to sustain
and/or modernize the installations or operations required for general research, development, test and
evaluation. Test ranges, military construction, maintenance support of laboratories, operation and
maintenance of test aircraft and ships, and studies and analyses in support of the RDT&E program are
funded in this budget activity. Costs of laboratory personnel, either in-house or contractor operated,
would be assigned to appropriate projects or as a line item in the Basic Research, Applied Research, or
ATD program areas, as appropriate. Military construction costs directly related to major development
programs are included.
Budget Activity 7, Operational System Development
This budget activity includes development efforts to upgrade systems that have been fielded or have
received approval for full rate production and anticipate production funding in the current or subsequent
fiscal year. All items are major line item projects that appear as RDT&E Costs of Weapon System Elements
A-3
in other programs. Program control is exercised by review of individual projects. Programs in this category
involve systems that have received Milestone C approval. A logical progression of program phases and
development and production funding must be evident in the FYDP, consistent with the Department’s full
funding policy.
B-1
Appendix B - List of Acronyms
ACD&P Advanced Component Development & Prototypes
ASD(R&E) Assistant Secretary of Defense (Research & Engineering)
ATD Advanced Technology Demonstration
BCA Budget Control Act
BEAT Budget Exhibit Analysis Tool
BLI Budget Line Item
COI Communities of Interest
CSV Comma Separated Values
DEPSECDEF Deputy Secretary of Defense
DoD Department of Defense
DoN Department of the Navy
DTIC Defense Technical Information Center
EVMS Earned Value Management System
FFRDC Federally Funded Research & Development Center
FYDP Future Years Defense Program
JTAAD Joint Air Force & Analysis Division of SPA
LCS Littoral Combat Ship
MILCON Military Construction
MILPER Military Personnel
O&M Operations and Maintenance
OCO Overseas Contingency Operations
OSAG Operations Support Analysis Group of SPA
OSD Office of the Secretary of Defense
PB President's Budget
PE Program Element
PROC Procurement
R&D Research & Development
RDT&E Research, Development, Test & Engineering
S&T Science & Technology
SDD Systems Development & Demonstration
SECDEF Secretary of Defense
SOW Statement of Work
SPA Systems Planning and Analysis, Inc.
SQL Structured Query Language
UARC University-Affiliated Research Center
USA United States Army
USAF United States Air Force
VBA Visual Basic for Applications
WBS Work Breakdown Structure
XML Extensible Markup Language
C-1
Appendix C – Use Cases
Use Case 1: Run R-1/R-2 Load Batch Job
Characteristic Information
Goal In Context: Load downloaded R-1/R-2 file into database via batch job
Scope: Java Batch Application, Database
Level: Sub-Functionality
Pre-Condition: R-1/R-2 file downloaded from DTIC website
Success End Condition: Batch job successfully, data loaded into database
Minimal Guarantees: Database is set up
Primary Actor: User: analyst interested in performing analysis on budget data
Trigger Event: R-1/R-2 file downloaded from DTIC website and moved to
staging directory
Main Success Scenario
Table 3. Use Case 1 - Success Criteria
Step Actor Action Description
1 User Execute command to R-1 batch job
2 R-1BatchJob Read R-1 batch job
3 R-1BatchJob Write R-1 job to Database
4 Database Send R-1 Job completion Receipt
5 User Execute command to R-2 batch job
6 R-2BatchJob Read R-2 batch job
7 R-2BatchJob Write R-2 job to Database
8 Database Send R-2 Job completion Receipt
C-2
Scenario Extensions
Table 4. Use Case 1 - Scenario Extensions
Step Sub Step Condition Action Description
Batch job fails because database services is not
running
1a
1a1 R-1 BatchJob Notify User of file execution failure
1a2 User Turn on database services
Unable to read file
2a
2a1 R-1BatchJob Notify User of read failure
2a2 User Close file
Figure 24. Use Case 1 - Run R-1/R-2 Load Batch Job
C-4
Use Case 2: Run Excel Generator Job
Characteristic Information
Goal In Context: User runs Excel Generator batch job to create data to be
exported to Excel
Scope: Database, Java batch application, Excel visualization tool
Level: Sub-Functionality
Pre-Condition: MS Excel must be loaded into machine. BVA tool kit must be
enabled
Success End Condition: Batch application successful, Excel file created, VBS script
executed
Minimal Guarantees:
Primary Actor: User: analyst who wants to extract data from database and
visualize the data
Trigger Event: User starts batch job
Main Success Scenario
Table 5. Use Case 2 - Success Criteria
Step Actor Action Description
1 User Start R-1 batch job
2 R-1BatchJob Retrieve R-1 data
3 Database Send R-1 File
4 R-1BatchJob Read R-1 Excel file
5 R-1BatchJob Send command to execute R-1 VBA Macro
6 ExcelGeneratorJob Execute R-1 VBA Macro
7 ExcelGeneratorJob Send R-1 File job completion
8 User Receive R-1 completion receipt
Scenario Extensions
Table 6. Use Case 2 - Scenario Extensions
C-5
Step Sub
Step
Condition Action Description
VBA Macro fails to execute
6a
6a1 ExcelGeneratorJob Notify User of VBA execution failure
6a2 User Reset Excel Macro capabilities
Figure 26. Use Case 2 - Run Excel Generator Job
C-7
Use Case 3: Perform R-1/R-2 Data Visualization
Characteristic Information
Goal In Context: User wants to graph and visualize R-1/R-2 data
Scope: ExcelGeneratorJob has the capability to graph and display R-1/R-
2 data
Level: Sub-Functionality
Pre-Condition: MS Excel add-in “SmartCharts” (by DevScope) must be installed
Success End Condition: Visualization successfully loads data and filters
Minimal Guarantees:
Primary Actor: User: interested in visualizing R-1 data and filters
Trigger Event: Visualization requests user to load data
Main Success Scenario
Table 7. Use Case 3 - Success Criteria
Step Actor Action Description
1 User Launch Smart Chart Tool
2 ExcelGeneratorJob Populate R-1 Data
3 ExcelGeneratorJob Populate R-2 Data
4 ExcelVisualizationTool Send data receipt to user
5 User Select data to be graphed
6 User Filter PE list
7 User Search for specific PE
8 ExcelVisualizationTool Graph R-1 data
9 ExcelVisualizationTool Graph R-2 data
10 ExcelVisualizationTool Display R-1 data in smart chart add-on to user
11 ExcelVisualizationTool Display R-2 data in smart chart add-on to user
C-8
Scenario Extensions
Table 8. Use Case 3 - Scenario Extensions
Step Sub Step Condition Action Description
PE is not in the query list
6a
6a1 R-1 Batch Job Notify User PE cannot be found
6a2 User Search for another PE