Open Source Software Frameworks

Interagency Modeling and Analysis Group (IMAG)

Peter Lyster, Ph.D.

Program DirectorNational Institute of General Medical Sciences

Tuesday, June 7 (2005)

A programmatic goal is to promote open and interoperable software, and to have teams engage in concurrent software development efforts.  The benefits of these practices are expected to include, but are not limited to, improved functionality by linking disparate but scientifically appropriate software, reduction of redundant software efforts, efficient software reuse, and improvement in quality of software by opening the development process to more scientists. 

Frameworks: a la MSM NSF 04 607

Each investigator should include in the project description the software development practices they intend to use to achieve these goals.  Proposers should also include a dissemination plan that describes in detail how software to be developed and data to be generated will be made available for incorporation into the work and the explorations of other members of the community, including interoperability of software components between the proposed project's software and others, and ability to merge data produced by the project with other workers' databases and datasets.   The program is expected to promote these goals initially through the development of a knowledge base through which each investigator will contribute regular updates of their project.  Investigators will be expected to interact with cognizant program officials in the creation of this knowledge base.  The Consortium of Investigators and program officers will then be able to utilize this knowledge base to promote interaction between investigators.  As the program matures, it is expected that mature software engineering practices will be developed that will enable concurrent development, interoperability, and the sharing of software. Investigators are expected to budget for and plan software repositories, and associated software engineering efforts. 

There is no prescribed license format for software produced in this project.  However IMAG does have goals for software dissemination, and reviewers will be instructed to evaluate the dissemination plan relative to the following goals: 1) The software must be freely available to researchers and educators in the non-commercial, non-profit sector, such as institutions of education, research institutes, and government laboratories. 2) The terms of software availability must permit the commercialization of enhanced or customized versions of the software, or incorporation of the software or pieces of it into other software packages. 3) The terms of software availability must include the ability of researchers outside the Consortium to modify the open source code and to share modifications with other colleagues as well as with the Consortium.  A Consortium member must take responsibility for creating the original and subsequent "official" versions of a piece of software, and must provide a plan to manage the dissemination or adoption of improvements or customizations of that software by others. This plan must include a method to distribute other users' contributions such as extensions, compatible modules, or plug-ins.

IP Issues

Post Award

We expect the work funded under this solicitation to form a scientific Frameworks community for concurrent software development and dissemination.  This may involve development of community SourceForge repositories.  Among other activities, a post award Consortium meeting will be a forum for planning and executing the Frameworks.  This will apply, as appropriate, to software developed under the current solicitation, or even relevant software that has already been developed and deployed in the community.

Open, Concurrent Software Development using CVS

The really old way: The dictatorThe old way: Check out and lock (or worse)The CVS way: Check out, update/merge, check in

• Freedom from errors• Freedom from problem developers• Freedom from conflict• Enables Open, Concurrent software development

Open versus Closed

Concurrent versus Sequential

Software Interoperability and Reuse

• Seek to use same component in multiple contexts, or multiple components in the same context (plug and play)

• Separate software from context• Distinguish between syntactic and semantic interoperability (see

caBIG compatibility guidelines)• Pipelines

Open, Concurrent Software development• IP issues• Multiple developers (at geographically distant sites) working

on the same software• Federated or single software repositories?

A Framework is a component-based software environment that promotes reuse and concurrent software development

Roadmap National Centers for Biomedical Computing (NCBC)

Software and Data Integration Working Grouphttp://www.na-mic.org/Wiki/index.php/SDI:_Agenda_for_July_8_meeting_in_Bethes

da

• Proposed Meeting Goals • 1a. To identify the elements and activities the Centers

believe describe a broad national biomedical computing framework--this includes interactions among the Centers as well as with other related efforts in biomedical computing;

• 1b To identify the role of the Centers in this broader framework;

• 2a To identify steps necessary to exploit opportunities and address gaps for achieving a national biomedical computing framework

• 2b. To identify how the Centers may exercise their national leadership as NIH NCBCs (collectively and individually) to achieve these steps and foster a robust and translationally relevant biomedical computing framework

• a. Explore NIH-forge, Frameworks, or Clearinghouse. A substantive effort may involve the development of common federated software repositories and the associated community software building effort that such repositories entail. These may be used to foster software reuse and/or concurrent software development. The key to federation is that control of software repositories is retained at the developer sites, while a lightweight overarching management structure means that the broad developer community is not confronted with balkanized access, change, test, and build procedures.

• b. Data repositories: issues include the temporal order of creation, maintenance, and their independence/dependence.

• c. Interoperability: the broad issues are developed in the caBIG Compatibility Guidelines (https://cabig.nci.nih.gov/guidelines_documentation). One suggestion is the use of interoperability demonstrations as a driver. The Centers have already identified areas of common domain overlap (these will be described briefly on July 8), and these will be a logical choice for meaningful interoperability demonstrations.

• d. The use of Pipelines (e.g., LONI Pipeline (http://www.na-mic.org/Wiki/index.php/LONIPipelineSummary)). This may be used to foster software reuse and flexible composition of applications.

4. To define players, milestones and timelines for implementing these efforts

3. To identify specific new or ongoing efforts in which the Centers and

other NIH projects may participate to advance biomedical research in the

area of software and data integration, e.g:

Summary Issues/Threads

• Simplicity, Reuse, Extensibility, Maintainability

• Open source development (open, concurrent)

• Interoperability, compatibility (syntactic, semantic)

• Frameworks, NIH-forge

THE END

Linux: Cathedral or Bazaar?(http://firstmonday.org/issues/issue5_3/kuwabara/)

• Torvalds works closely with the so-called "Inner Circle" of technical advisors, or "lieutenants," immediately around him. They are core developers who have established their status as competent programmers and proven their expertise valuable to the project through years of involvement. Opinions vary, however, as to who actually constitute the Inner Circle:

• "There are a number of well trusted individuals like Alan Cox who could be said to exist in the inner circle. I'm not sure if there is a defined inner circle or not. Most likely it's a group of individuals who know the kernel inside and out and have similar goals in mind."

Cont.

• Because of the increasing volume of development in general and the technical details of numerous internal components of the kernel in particular, Torvalds encourages developers to submit their ideas and patches of code to the primary maintainer of the component first:

• "What happens is that if I get a patch against something where I'm not the primary developer (against a device driver or a [file system] I have nothing to do with, for example), I am less likely to react to that patch if it doesn't come from the person who is responsible for that particular piece of code.

• If I get a networking patch, for example, it's a lot more likely to get a reaction if it is from Alan Cox, and then it usually goes in with no questions asked. Similarly [...], if it as a patch against the SCSI subsystem, I much prefer to have it from somebody I know works on SCSI“

• the Inner Circle and other primary maintainers of subsystems mediate Torvalds and the development community, providing an effective filter to reduce the load reaching Torvalds - effective to the very extent that, while Torvalds still insists that he reviews every line of the code he applies to the kernel, some people feel that this is unnecessary [66] (one person regards it "probably the most significant Linux bottleneck" [67]), suggesting the general reliability of the decentralized development below Torvalds.

• In the Linux project, the lack of formal structure invested with decision-making procedures resonates with the absence of central planning from Torvalds. Indeed, it is his deliberate effort to avoid imposing long-term plans and visions on the community: