A Combat Support Agency

Defense Information Systems Agency

Forge.mil – DoD Collaborative Software Development

IT Innovators Award

IT Innovators Award

Guy Martin, Forge.mil Community Managercommunity@forge.mil

Richard Bullington-McGuire, Director, Technologyrichard.bullington-mcquire@threepillarglobal.com

Guy Martin, Forge.mil Community Managercommunity@forge.mil

Richard Bullington-McGuire, Director, Technologyrichard.bullington-mcquire@threepillarglobal.com

2

Forge.mil Vision

TODAY• Siloed development environments• Expensive and time consuming start-up • Limited exposure, sharing, or re-use• Duplication of effort

TODAY• Siloed development environments• Expensive and time consuming start-up • Limited exposure, sharing, or re-use• Duplication of effort

Developer

Tester

UserCertifier

Shared Test & Development Tools/Services/Environments

Shared Asset Libraries & Repositories

Developer

FORGE.mil• OSS best practices applied to internal projects • Cross-program sharing: software and services• Early and continuous collaboration • Integrated approach to development life cycle• Extensible platform to support delivery of partner capabilities

FORGE.mil• OSS best practices applied to internal projects • Cross-program sharing: software and services• Early and continuous collaboration • Integrated approach to development life cycle• Extensible platform to support delivery of partner capabilities

3

Forge.mil Services

Free, collaborative development environment for internal open-source and DoD community source software

Free, collaborative development environment for internal open-source and DoD community source software

On-demand, fee-for-service development environment for individual internal programs and projects

On-demand, fee-for-service development environment for individual internal programs and projects

Multi-Tenant Environment- Isolated project spaces for each

customer- Shared infrastructure

Multi-Tenant Environment- Isolated project spaces for each

customer- Shared infrastructure

Private Environment- Full environment dedicated to

your program

Private Environment- Full environment dedicated to

your program

4

Forge.mil Community Approach

DOD Acquisition Community

DOD Test and Evaluation Community

DOD IA Community

DOD NETOPSCommunity

DOD DevelopmentCommunity

Government, Industry & Academia

Collaborative Development & Test

Note: Forge.mil != DoD & External OSS Community

5

Requirements managementRequirements management

Forge.mil Key Features

Application lifecycle management (ALM) services

for the DoD Enterprise

Source code managementSource code management

Discussion forumsDiscussion forumsProject wikiProject wiki Document managementDocument management

Project management for distributed development teamsProject management for distributed development teams

Tasking & alertsTasking & alerts

Release managementRelease management

Real-time reportingReal-time reporting

Software development servicesSoftware development services

Bug, Issue TrackingBug, Issue Tracking

Share software, best practices, informationShare software, best practices, information

6

Community Challenges

– Hierarchical, process & command driven culture– Extreme risk aversion (with good reason!)– Heavy reliance on documents, in-person meetings, email– Skepticism of new processes/tools– Certification/accreditation of client-side tools (SVN, etc.)

7

Community Lessons Learned

– Government community building is a 'contact sport’– Both ‘carrot’ (grassroots) & ‘stick’ (top-down) needed– Categorization (project, artifact, etc.) VERY important– Documentation/process socialization critical to acceptance– Platform/tool MUST tie into email (notifications)– “You can’t forklift a revolution…”

8

Community Lessons Learned

– Don’t assume inquisitiveness– FAQ lists are important (even if not read the first time)– Seek out, support, & encourage community leaders– Grow community efforts first around existing tools/tech– Don’t be afraid to use chain of command to jumpstart things– Have realistic expectations

9

Community Victories

Sampling of Hosted Projects

Army• Apps 4 Army: Innovation contest to develop new ‘mashup’s of Army data• SOSCOE: System of Systems Common Operating Environment - tactical

middleware

Navy• Gargoyle: a network activity monitoring and analysis system• NEP-O: Naval Enterprise Portal Oceanography – Agile process used heavily• Vulnerator: Aggregation of security readiness findings for systems

Air Force• AF EIM: code supporting Air Force’s Enterprise Information Management • UAS TSPI Server: a common network interface to multiple UAS ground-stations

for sensors requiring real-time telemetry source

Marine Corps• DCGS-MC: Distributed Common Ground/Surface System – Marine Corps• MAGTF C2: Marine Air Ground Task Force Command and Control

Joint Chiefs & DISA• NSLDSS: National Senior Leadership Decision Support Service• DIB: Distributed Common Ground/Surface System (DCGS) Integration Backbone • CommunityCAC: CAC Utilities/Firefox plugin• DODBastille: RHEL STIG lockdown utilities

Sampling of Hosted Projects

Army• Apps 4 Army: Innovation contest to develop new ‘mashup’s of Army data• SOSCOE: System of Systems Common Operating Environment - tactical

middleware

Navy• Gargoyle: a network activity monitoring and analysis system• NEP-O: Naval Enterprise Portal Oceanography – Agile process used heavily• Vulnerator: Aggregation of security readiness findings for systems

Air Force• AF EIM: code supporting Air Force’s Enterprise Information Management • UAS TSPI Server: a common network interface to multiple UAS ground-stations

for sensors requiring real-time telemetry source

Marine Corps• DCGS-MC: Distributed Common Ground/Surface System – Marine Corps• MAGTF C2: Marine Air Ground Task Force Command and Control

Joint Chiefs & DISA• NSLDSS: National Senior Leadership Decision Support Service• DIB: Distributed Common Ground/Surface System (DCGS) Integration Backbone • CommunityCAC: CAC Utilities/Firefox plugin• DODBastille: RHEL STIG lockdown utilities

Initial Forge.mil capability (April 2009) supporting collaborative software development & reuse

Today (March 2011)• Over 2700 software releases available• ~47,000 software releases downloaded• Over 36,000 bugs/requirements tracked• Over 39,000 code checkins• Over 3500 discussion posts• ~ $160M in software asset reuse ROI

Initial Forge.mil capability (April 2009) supporting collaborative software development & reuse

Today (March 2011)• Over 2700 software releases available• ~47,000 software releases downloaded• Over 36,000 bugs/requirements tracked• Over 39,000 code checkins• Over 3500 discussion posts• ~ $160M in software asset reuse ROI

Mar 26

10

Forge.mil Implementation:Open Source Enabled Agility

How we achieved our mission by both producing and consuming

Open Source Software(while moving at a very rapid pace)

Disclaimer: These implementation notes reflect the experiences and opinions of the author, Richard Bullington-McGuire, the architect for the initial implementation phase of Forge.mil, and do not represent DISA’s official positions

Slides following marked with a “*” are contributed by Richard Bullington-McGuire

11

First Challenge: Secure it Fast *

– Implementation effort began on Nov. 10, 2008– First fielded in production (LOA) on Jan 23, 2009 (< 90 days)

– Biggest Roadblock: Security– Application Security and Development STIG Requires:

– PKI User Authentication– Support both soft cert and CAC hardware token– Clients: Web browser, Subversion, and SOAP API

– Encrypt system passwords at rest (including database user names and passwords)

– Solution: Extend open source elements of software stack

12

Forge.Mil Internal Architecture *

Key Concepts and Architecture• CollabNet TeamForge 5.x on

Red Hat Enterprise Linux 5.x• Open Source foundations:

Apache HTTP Server, mod_ssl, mod_python, JBoss, Tomcat, Subversion, Lucene, Apache James, PostgreSQL

• Key insight: extend security features by building on the open source components below CTF using Python and Java modules

• Key insight: enable CAC token authentication in Subversion clients by extending and distributing new client binaries

Deployment Architecture

12

software.forge.milApplication Server

svn.forge.milIntegration Server

Single Sign On (SSO) Database

Application Database

Forge.mil UserWith x509 Client Certificate (CAC/ECA)

13

Consuming Open Source *

– Server-side:– All custom extensions and amendments built on software

shipped with RHEL 5. Goal: keep system as vanilla as possible. Use RPM for packaging extensions.

– PKI enablement required updating stock mod_python with newer version, and writing user mapping logic tied to SSL

– Secure password storage required the m2crypto library for the mod_python extension layers, and extending JBoss with a module for reading the encrypted passwords using a key stored on a RAM disk. Password encryption functions : m2secret

– Client-side:– CAC-enabled Subversion with TortoiseSVN 1.5.5

14

Producing Open Source *

– Main Roadblock: lack of hardware token support in SVN clients. Only TortoiseSVN 1.5.5 had support for hardware token authentication with the CAC (we redistributed it)– We developed and distributed:

– A working Windows CLI svn client with CAC support– A modified Subclipse client with CAC support– A cross-platform Java SVN client with CAC support– Updated versions of TortoiseSVN with CAC support

– These efforts helped inspire broader support for hardware token-driven SVN clients (see the latest releases of: CollabNet Desktop for Eclipse and Visual Studio, which support hardware tokens)

15

Achieving Mission Success Fast *

We would not have been able to field this service so quickly without both producing and

consuming open source software– Leveraging Open Source let our small team make architecture-

level tweaks to vendor-supplied software stacks that would have taken the vendors much longer to make

– We focused on using OSS to solve real security issues, and worked within the DoD certification & accreditation process

– Licensing: we produced & consumed the software under the original terms of the various licenses (no new licenses introduced, no artificial hurdles in the reasonable distribution of software we modified under existing licenses)

16

BACKUP SLIDES

17

Upcoming Capabilities: Full Application Life-cycle Support

Development Zone

Validation/Pre-Production Zone

Dashboard, Reporting & Monitoring

Build Libraries & Code Repositories

ProductionZone

Agile SoftwareDevelopment

Unit, Integration &Regression Testing

Static CodeAnalysis

SystemTesting

IA Certification

IT & Data Standards

TodayToday

Release & OperationsManagement

Q2 11Q2 11

Q4 10Q4 10

Q4 11Q4 11

Q4 12Q4 12

Cloud Hosting Environment

18

Policy and Guidance

HR 2647, National Defense Authorization Act for Fiscal Year 2010, Sec. 804.

“The Secretary of Defense shall develop and implement a new acquisition process for information technology systems. The acquisition process developed and implemented pursuant to this subsection shall, to the extent determined appropriate by the Secretary —

. . . be based on the recommendations in chapter 6 of the March 2009 report of the Defense Science Board Task Force on Department of Defense Policies and Procedures for the Acquisition of Information Technology; and

. . . be designed to include —

1. early and continual involvement of the user;

2. multiple, rapidly executed increments or releases of capability;

3. early, successive prototyping to support an evolutionary approach; and

4. a modular, open-systems approach.”

19

Benefits of Using Forge.mil

FOR DEVELOPERS FOR MANAGERS FOR EXECUTIVES

• Access a full featured

development platform over the

web or directly from your IDE• Have fewer meetings and less

administration when

collaboration is part of everyday

development• Link to continuous integration

servers and provision build and

test servers in the cloud

whenever you need them

• Speed new project startup• Secure access to project assets• Enhance team productivity and

collaboration• Improve visibility into project

status• Reduce management and

administrative overhead• Access critical team assets via

the web or your Microsoft

Windows desktop

• Consolidate and centralize to

reduce administration, licensing,

and infrastructure costs• Establish governance and

regulatory compliance• Improve predictability of the

development organization• Integrate easily into existing

systems to extend return on

investments

Forge.mil reduces administrative costs, increases productivity, and improves visibility. Key benefits include:

20

Forge.mil PKE changes: HTTPD modules *

Application Database

Single Sign On (SSO) Database

software.forge.mil / svn.forge.milApplication Server or Integration Server

Red Hat Enterprise Linux 5Vmware ESXi

mod_python•sfauth (svn auth)•sf_sso looks up cert->user mappings in SSO db•sf_pki calls TeamForge login() method via SOAP using master password, redirects user through alternate login path accepting username + session ID

JBoss•On App server only•Web Rendering• SOAP Server•JAAS module: masterpassword.jar

Tomcat•James Mail•Lucene Indexes•SCM viewer (on integration server) PostgreSQL 8.2

DatabasesOn Separate RHEL 5 VMs

Apache HTTPD

Forge.mil User

Servicew/ x509 Server Cert,

Reused as Client Cert

Client → Serverhttps / TCP 443

Client Software•Web browsers (IE, Firefox)•Subversion clients (DAV over https)•Custom SOAP clients•All must use client cert auth.

httpproxy

+SOAP

Server → DatabasePostgreSQL / TCP 5432

JBOSS → Tomcat

Java RMI

PKE changes to baseline are listed in italics

21

Ongoing Challenges & Opportunities *

– Is Forge.mil the One True Path for DoD software development?– Walled garden protected by CAC & ECA certificates is a

disincentive to public participation. However, public participation is not the primary function of Forge.mil.

– Forge.mil is trying to solve the internal collaboration problems within the DoD using OSS methodologies.

– Not every project (Open Source or otherwise) will be a good fit for Forge.mil, but many projects will be a fit that would not be a good fit for a more public repository.

– Having this service available has helped shape the ongoing community discussion about what normative behavior should be regarding producing and consuming Open Source in the context of DoD projects.