Slide 1

Sociotechnical production systems for software in science James Howison and Jim Herbsleb Institute for Software Research School of Computer Science Carnegie Mellon University School of Information University of Texas at Austin http://james.howison.name/pubs/HowisonHerbsleb2011SciSoftIncentives.pdf

Slide 2

How does a a cubic km of ice become a scientific paper?

Slide 3

First find some ice Image Credit: NASA

Slide 4

Build a big drill Image Credit: IceCube

Slide 5

and some Digital Optical Modules Image Credit: IceCube

Slide 6

Combine Image Credit: IceCube

Slide 7

Collect and filter data Image Credit: IceCube

Slide 8

Store and analyze it Image Credit: http://www.flickr.com/photos/theplanetdotcom

Slide 9

Simulate light in ice Photo credit: http://www.flickr.com/photos/rainman_yukky/

Slide 10

Simulate Atmosphere Image Credit: NASA

Slide 11

Model

Slide 12

Analyze

Slide 13

Plots

Slide 14

Publish

Slide 15

Software is everywhere

Slide 16

Enhancing reproducibility and correctness Saving money Driving innovation Coalescing into widely used software platforms All linked to software as information artifact: Re-playable Re-useable Extendable A appealing vision of software

Slide 17

Yet software also has constraints Maintenance (avoiding bit rot) Software must be maintained (synchronization work Kept in sync with complements and dependencies Coordinated Rapid development and changes can lead to breakdown Path dependencies Easy to start, hard to architect for widespread use

Slide 18

How to achieve the Software Vision? Better technologies? Better engineering methods? Leadership/Norms/Ethics? Policy? Rewards?

Slide 19

A sociotechnical understanding Understand software work in existing institutions of science Specific Research Questions: What software is used? How created and maintains it? What incentives drive its creation? Why is it trusted?

Slide 20

Method: Data Route into complex practice Chose paper as unit of analysis: Focal Paper Trace back from paper to work that produced it Semi-structured interviews Supported by artifacts (e.g., paper/methods and materials) Elicit workflow, focus on software work Identify software authors/sources, and seek introductions Qualitative analysis Phenomenological exhaustion

Slide 21

Case 1: STAR Image Credit: RHIC

Slide 22

Our focal paper

Slide 23

Workflow

Slide 24

Software Production 1.Employed Core Software development Professional software developers ROOT4STAR framework 2.Core simulation code Scientists undertaking service work 3.Analysis code to get the plots Locally written, frozen at publication

Slide 25

Case 3: Bioinformatic microbiology Image Credit: http://www.flickr.com/photos/grytr

Slide 26

Studying the nitrogen cycle Image Credit: Focal Paper

Slide 27

A field revolutionized by software

Slide 28

Personal software infrastructure Power user scripts Personal competitive advantage that is something that most biologists cant do. period. Share methods but not personal infrastructure code or actively support others Methods and materials section should provide enough information, if not hell fix it. But not going to do their homework for them

Slide 29

Publishing on software Tools potentially useful to others described in separate publications, Software pubs Ambivalence: Can you make a career out of this? Definitely But: hes known for his software rather than his science hes known for facilitating science rather than and some people have that reputation Advise a student to do this? Yes, but if you happen to get a publication out of it and it becomes a tool thats widely used, then great, thats fantastic, better props for you but theres a danger Tool developers are greatly under-appreciated

Slide 30

Algorithm people Self-described member of the algorithm people as distinguished from biologists Muscle: biology == strcmp() Builds from scratch (avoid tricky dependencies) Obvious that they dont collaborate Credit accrues to the original publications Little credit in perceived incremental improvements Politics of improvement acceptance at the mercy of Competition is appropriate and productive

Slide 31

Software Production systems Practice that is similar on four aspects: 1.Incentives for the work 2.The type of artifacts produced 3.The way it is organized 4.The logic of correctness

Slide 32

Context: Academic reputation system

Slide 33

Software as support

Slide 34

Collaboration service-work

Slide 35

Academic credit: Incidental software

Slide 36

Academic credit: Parallel software practice

Slide 37

Systemic threats to software vision The type of software work needed to realize the cyberinfrastructure vision is poorly motivated Invisible work (Star and Ruhlender) Especially, little incentive to collaborate Project owned by initial creators Initial publications receive citations Extension dominated by fork-and-rename

Slide 38

Academic reputation and integration James Howison and Jim Herbsleb (2013) Sharing the spoils: incentives and integration in scientific software production. ACM CSCW

Slide 39

Where to for science policy? Exhortations? Training? Forcing open source through funding lever? Risk of substituting logics of correctness Kleenex code as open source? Risk of undermining appropriate competition Turn scientists into open source community managers? When there is little reward for this work?

Slide 40

Scientific Software Network Map But, you know, imagine it as a live, dynamic data set!

Slide 41

Techniques for measuring use Software that reports its own use Instrumentation Analysis of traces in papers Mentions, citations Characteristic artifacts Analysis of collections of software On supercomputing resources (TACC, NICS) Through workflow systems (Galaxy, Pegasus, Taverna)

Slide 42

Contact James Howison http://james.howison.name jhowison@ischool.utexas.edu This material is based upon work supported by the US National Science Foundation under Grant No. #0943168.