User-Based Innovation & User-Based Innovation & Communities Drive Commercial Communities Drive Commercial

Systems SoftwareSystems Software

James HamiltonJames HamiltonGM SQL Server WebData DevelopmentGM SQL Server WebData Development

& SQL Security Architect& SQL Security Architect

http://research.microsoft.com/~JamesRHhttp://research.microsoft.com/~JamesRH

JamesRH@Microsoft.comJamesRH@Microsoft.com

2004.04.152004.04.15

22

IntroductionIntroduction I’m an unrepentant commercial software guy :-)I’m an unrepentant commercial software guy :-)

Have done some non-commercial S/W work in spare timeHave done some non-commercial S/W work in spare time Ported g++, gdb, and Taylor UUCP to AIX 1.3Ported g++, gdb, and Taylor UUCP to AIX 1.3 Ran a UUCP site for yearsRan a UUCP site for years

System Software FocusedSystem Software Focused 11 years at IBM11 years at IBM

Ada & C++ Development ManagerAda & C++ Development Manager Lead Architect DB2 UDB DatabaseLead Architect DB2 UDB Database

7 years at Microsoft7 years at Microsoft Windows2000 Base DevelopmentWindows2000 Base Development SQL Server development team in various rolesSQL Server development team in various roles

–Relational System Development ManagerRelational System Development Manager–Security ArchitectSecurity Architect–General Manager WebData DevelopmentGeneral Manager WebData Development

Interested in better understanding & harnessing Interested in better understanding & harnessing user community contribution to S/W systemsuser community contribution to S/W systems

What follows are my observations & opinions andWhat follows are my observations & opinions and do not necessarily represent a Microsoft positiondo not necessarily represent a Microsoft position

33

AgendaAgenda

Driving innovation: users or manufacturer?Driving innovation: users or manufacturer? Revolutionary change often not user drivenRevolutionary change often not user driven

Examples from DB & TP world with which I’m Examples from DB & TP world with which I’m most familiarmost familiar

Redundant Array of Inexpensive Disks (RAID)Redundant Array of Inexpensive Disks (RAID) Relational databaseRelational database

Users prime drivers of product evolutionUsers prime drivers of product evolution Community drives commercial S/W successCommunity drives commercial S/W success

Commercial S/W has always been community Commercial S/W has always been community dependentdependent

Why not go open source & gain the community Why not go open source & gain the community contribution?contribution?

Parallels between commercial & non-commercial Parallels between commercial & non-commercial s/w effortss/w efforts

44

Revolutionary change: Revolutionary change: often not user drivenoften not user driven

Users do drive incremental & evolutionary changeUsers do drive incremental & evolutionary change Some examples from DB world:Some examples from DB world:

Online index createOnline index create Online re-orgOnline re-org XML datatypeXML datatype Automated multi-system administrationAutomated multi-system administration ……

Revolutionary change examples from systems Revolutionary change examples from systems world:world: RAID & Relational DBRAID & Relational DB

“…“…Well-managed companies that have their Well-managed companies that have their competitive antennae up, listen astutely to competitive antennae up, listen astutely to their customers, invest aggressively in their customers, invest aggressively in new technologies, and yet still lose market new technologies, and yet still lose market dominance”dominance”

“…“…Well-managed companies that have their Well-managed companies that have their competitive antennae up, listen astutely to competitive antennae up, listen astutely to their customers, invest aggressively in their customers, invest aggressively in new technologies, and yet still lose market new technologies, and yet still lose market dominance”dominance”

The fundamental changes are often not The fundamental changes are often not user drivenuser driven

The fundamental changes are often not The fundamental changes are often not user drivenuser driven

55

Redundant Array of Inexpensive DisksRedundant Array of Inexpensive Disks

Disks were expensive, IBM Disks were expensive, IBM dominated, & growing at sub-dominated, & growing at sub-Moores law ratesMoores law rates

Commodity disk much less Commodity disk much less reliable & 1/5 to 1/10 the reliable & 1/5 to 1/10 the capacity of enterprise diskcapacity of enterprise disk

““A Case for Redundant A Case for Redundant Arrays of Inexpensive Disks Arrays of Inexpensive Disks (RAID)”(RAID)” – Patterson, Gibson, Katz– Patterson, Gibson, Katz

Base observation: Commodity Base observation: Commodity disks with redundancy can be disks with redundancy can be combined to produce larger & combined to produce larger & more reliable storagemore reliable storage

SIGMOD, June 1988SIGMOD, June 1988

66

EMC Symmetrix DMX3000EMC Symmetrix DMX3000

84 TB Storage capacity84 TB Storage capacity 576 3 ½” commodity Seagate (usually) SCSI Disks576 3 ½” commodity Seagate (usually) SCSI Disks 256 GB Memory & 100 1 Ghz PowerPC CPUs256 GB Memory & 100 1 Ghz PowerPC CPUs 10x to 15x storage cost premium over commodity disk10x to 15x storage cost premium over commodity disk EMC annual revenue: $6.24B EMC annual revenue: $6.24B (2003 10K filing)(2003 10K filing) RAID overall annual revenue: $13B RAID overall annual revenue: $13B (1998 Disk/Trend Report)(1998 Disk/Trend Report)

77

Relational DatabaseRelational Database ““A Relational Model of Data for Large A Relational Model of Data for Large

Shared Data BanksShared Data Banks”– E. F. Codd CACM 13,6 ”– E. F. Codd CACM 13,6 (June 1970)(June 1970) ““Some industrial motivation and then Some industrial motivation and then

straight to the math”straight to the math” – Irv Traiger– Irv Traiger

Broadly debated: CODASYL vs Broadly debated: CODASYL vs RelationalRelational SIGFIDET and SIGMOD conferencesSIGFIDET and SIGMOD conferences

First IBM System R relational DB user: First IBM System R relational DB user: MIT Sloan School of Management MIT Sloan School of Management (System R (System R

Phase 0 proto 1975)Phase 0 proto 1975) System R code base later became IBM System R code base later became IBM

SQL/DSSQL/DS later DB2 for VM & VSE — product still later DB2 for VM & VSE — product still

availableavailable Manufacturer research rather than Manufacturer research rather than

user community pulluser community pull

Relational DB Theory: E.F. CoddRelational DB Theory: E.F. Codd

SQL Language: Don ChamberlinSQL Language: Don Chamberlin

88

Relational Database MarketRelational Database Market Overall annual Relational DB market: $8BOverall annual Relational DB market: $8B Gartner expects 86% of DB revenue to be Relational by 2005Gartner expects 86% of DB revenue to be Relational by 2005 Many DB & TP industry innovations driven by user partnershipsMany DB & TP industry innovations driven by user partnerships

IBM TPF: American Airlines SABER IBM TPF: American Airlines SABER Project started 1959Project started 1959

IBM IMS: Rockwell-NASA Apollo ProgramIBM IMS: Rockwell-NASA Apollo Program Released 1969Released 1969

IBM CICS: Partnership with Public UtilitiesIBM CICS: Partnership with Public Utilities PUCICS released 1968PUCICS released 1968

The innovation required user involvement but they didn’t drive itThe innovation required user involvement but they didn’t drive it

Source: Gartner 2001Source: Gartner 2001

99

AgendaAgenda Driving innovation: users or manufacturer?Driving innovation: users or manufacturer?

Revolutionary change often not user drivenRevolutionary change often not user driven Examples from DB & TP world with which I’m Examples from DB & TP world with which I’m

most familiarmost familiar Redundant Array of Inexpensive Disks (RAID)Redundant Array of Inexpensive Disks (RAID) Relational databaseRelational database

Users prime drivers of product evolutionUsers prime drivers of product evolution Community drives commercial S/W successCommunity drives commercial S/W success

Commercial S/W has always been community Commercial S/W has always been community dependentdependent

Why not go open source & gain the community Why not go open source & gain the community contribution?contribution?

Parallels between commercial & non-commercial Parallels between commercial & non-commercial s/w efforts?s/w efforts?

1010

Communities Drive Commercial S/WCommunities Drive Commercial S/W

Product SupportProduct Support User community support is the best way to provide User community support is the best way to provide

scalable, available, high quality product supportscalable, available, high quality product support It’s difficult to invest enough in a dedicated support It’s difficult to invest enough in a dedicated support

team to replace a community based programteam to replace a community based program Examples form engineering team in which I work:Examples form engineering team in which I work:

Participation required of all team membersParticipation required of all team members Newsgroups, customer presentations, feedback Newsgroups, customer presentations, feedback

sessions, service team internships, customer sessions, service team internships, customer requirements DB, work with MVP community, sample requirements DB, work with MVP community, sample programs, …programs, …

Marketing and Sales also heavily community drivenMarketing and Sales also heavily community driven ISV & reseller sales model commonISV & reseller sales model common Shareware & community sales sitesShareware & community sales sites

1111

Community Example:Community Example: www.boatdiesel.comwww.boatdiesel.com

1212

Why not use Community Development?Why not use Community Development?

S/W business driven by important, difficult to reproduce S/W S/W business driven by important, difficult to reproduce S/W aggregationsaggregations Separates lack-luster profit from the truly impressiveSeparates lack-luster profit from the truly impressive S/W business cost of entry is very close to zeroS/W business cost of entry is very close to zero

Profit of most entrants unexcitingProfit of most entrants unexciting Large S/W systems with critical mass very valuable:Large S/W systems with critical mass very valuable:

SAP, Windows, Oracle, DB2, …SAP, Windows, Oracle, DB2, … Large S/W aggregations are actively protectedLarge S/W aggregations are actively protected

But they are open in many waysBut they are open in many ways Most support interfaces to allow 3Most support interfaces to allow 3rdrd party extension party extension Source is typically available in a controlled waySource is typically available in a controlled way

Open-source S/W systems also actively protectedOpen-source S/W systems also actively protected However, branches are possible and sometimes succeedHowever, branches are possible and sometimes succeed

Commercial S/W open at key interfaces but user source code Commercial S/W open at key interfaces but user source code mods typically not supportedmods typically not supported Both commercial and non-commercial S/W systems dependent Both commercial and non-commercial S/W systems dependent

upon community for successupon community for success

1313

SummarySummary

Most product innovation is driven by usersMost product innovation is driven by users Improvements in speeds, feeds, & featuresImprovements in speeds, feeds, & features Applications of existing technology to new domainsApplications of existing technology to new domains

Revolutionary changes often not user drivenRevolutionary changes often not user driven Fundamental new approaches to existing problemsFundamental new approaches to existing problems Especially approaches that are:Especially approaches that are:

Not backward compatibleNot backward compatible Don’t adequately solve the entire breadth of the Don’t adequately solve the entire breadth of the

problem domainproblem domain All products, whether commercial or not, depend All products, whether commercial or not, depend

upon user community for support & innovationupon user community for support & innovation User driven support only affordable & effective optionUser driven support only affordable & effective option User Community source of most innovationUser Community source of most innovation User development community typically the driver of even User development community typically the driver of even

non-open source product successnon-open source product success

MicrosoftMicrosoft