Upload
matt-evans
View
107
Download
2
Tags:
Embed Size (px)
DESCRIPTION
Presented at GTAC 2010 conference, Hyderabad, India
Citation preview
Crowd Source Testing Mozilla Community Style
Matt EvansMozilla QA Director
Wednesday, November 10, 2010
about:talk
• Background and Scale of Mozilla
• Crowdsourcing: Crowd vs. Community
• History of Mozilla Community testing
• How Mozilla succeeds with Community testing
Wednesday, November 10, 2010
• I have been in the testing biz for most my career
• 9 mos as QA Director @ Mozilla
• Manage a team of ~30 Global Test Engineers
about:me
Wednesday, November 10, 2010
about:mozilla
Wednesday, November 10, 2010
about:mozilla
Mozilla’s Mission:
is to promote openness, innovation, and opportunity on the web
Wednesday, November 10, 2010
I’m still getting my head around scale at Mozilla
• The scale of information flow
• The scale of the user base
Wednesday, November 10, 2010
Mozilla Firehose
Wednesday, November 10, 2010
Mozilla Firehose• Many sources of information
• bugzilla (200+bugs/day)
• forums (50+)
• blogs (50+)
• tweets (100s/day)
• wikis (1000s pages)
• irc (50+ channels)
Wednesday, November 10, 2010
Mozilla Scale
400+Developers
10,000+Nightly Testers
2,500,000Beta Testers
400,000,000Users
400,000,000
350,000,000
300,000,000
250,000,000
200,000,000
150,000,000
100,000,000
50,000,0000Wednesday, November 10, 2010
Story Time
Wednesday, November 10, 2010
Big decision: Ship or No-Ship to Beta
A Recent Firefox product team meeting
Wednesday, November 10, 2010
Protector of the tester community
Wednesday, November 10, 2010
Wednesday, November 10, 2010
Wednesday, November 10, 2010
Get feedback from tester community
Wednesday, November 10, 2010
Wednesday, November 10, 2010
We were both wrong right!
Wednesday, November 10, 2010
It was a big change for my perspective
Wednesday, November 10, 2010
Respecter of the Tester Community as QA
Team Partner
Wednesday, November 10, 2010
We now work much better together
Wednesday, November 10, 2010
Crowd Sourcing: It involves a crowdworking towards a common goal
Wednesday, November 10, 2010
What is new, the virtual crowd is enabled by internet technologies
Wednesday, November 10, 2010
Wednesday, November 10, 2010
• Building a global community of testers ~30k+
• Stewardship of the test community is their secret sauce
Community approach to software testing
Wednesday, November 10, 2010
Mozilla Labs Crowdsource Crowdsourcing Project
Wednesday, November 10, 2010
Key Findings• there are two types users: heavy
users and casual users. • 90-9-1 Principle
• crowd motivation is separated into two forms: intrinsic and extrinsic
• crowdsourcing relies on people, and thus works when participants are treated as people.
Mozilla Labs Crowdsource Crowdsourcing Project
Wednesday, November 10, 2010
VS
Community Crowd
Wednesday, November 10, 2010
Mozilla uses a hybrid of crowd based feedback mechanisms and community
management
“Crowd sourced testing Mozilla Style”
Wednesday, November 10, 2010
The History of Mozilla Testing Community
Wednesday, November 10, 2010
Mozilla.org is launched
by Netscape
1998 2002
Mozilla 1.0 is released
Phoenix 0.1released
2004
Firefox 1.0released
2005-2009 2010
Firefox 1.5-3.5released
Firefox 4.0expected release
Pioneer Community Testers
Community Testers, The Next Generation
1 2 3
Wednesday, November 10, 2010
1998 2002 2004 2005-2009 2010
Pioneer Community Testers
Community Testers, The Next Generation
1 2 3
1998 Netscape open sourced the Netscape browser source code.
Wednesday, November 10, 2010
History of Mozilla’s Tester Community
•A lot of passionate people without a development background wanted to get involved.
Asa Dotzler First Mozilla Community Tester
Wednesday, November 10, 2010
The Rise of the Nightly Build
History of Mozilla’s Tester Community
Wednesday, November 10, 2010
History of Mozilla’s Tester Community
•A great success
•The community of testers grew to the 1000s
Wednesday, November 10, 2010
History of Mozilla’s Tester Community
Too much of good thing
Wednesday, November 10, 2010
History of Mozilla’s Tester Community
•As tester community members increased so did the noise
Wednesday, November 10, 2010
1998 2002 2004 2005-2009 2010
Pioneer Community Testers
Community Testers, The Next Generation
1 2 3
• 2002+ • The rise of Firefox code base• Split from AOL/Netscape• 2003 Mozilla Foundation is formed
Wednesday, November 10, 2010
1998 2002 2004 2005-2009 2010
Pioneer Community Testers
Community Testers, The Next Generation
2
Wednesday, November 10, 2010
1998 2002 2004 2005-2009 2010
Pioneer Community Testers
Community Testers, The Next Generation
1 2 3
• 2007+ • The rise of Continuous Integration
and unit tests
Wednesday, November 10, 2010
Continuous Integration
~3.3 million automated tests executed for every committed code change
•The largest CI system built on Buildbot and Tinderbox technologies
• For every code change
• 11 full builds of the browser across supported platforms
•~300k unit/perf tests per build
Wednesday, November 10, 2010
Minefield is the result of Continuous Integration
Continuous Integration and Unit Testing is the Foundation of Today’s Mozilla
Community Testing Success
Wednesday, November 10, 2010
Community Feedback
Wednesday, November 10, 2010
Mozilla Runs On Bugzilla
Wednesday, November 10, 2010
Crash Data Analysis
Wednesday, November 10, 2010
Crash Data Analysis
Wednesday, November 10, 2010
Crash Data Analysis5 years of crash data
FF1.5 FF2.0 FF3.0 FF3.5
FiledFixedDups
Wednesday, November 10, 2010
Firefox 4 Beta Feedback
Wednesday, November 10, 2010
Firefox 4 Beta Feedback
LCD Screens of Feedback and Twitter mentions are displayed throughout the Mozilla Office
Wednesday, November 10, 2010
Engaging the Mozilla Tester Community
Wednesday, November 10, 2010
Manual TestingLitmus - Test Case Management
Wednesday, November 10, 2010
Test and Bug Days
Wednesday, November 10, 2010
Mozmill Test Automation Framework
Community Focused Automated Testing
• Javascript framework for testing gecko based applications• Includes an IDE• Installs as an Firefox
extension• Better suited for
community contribution
Wednesday, November 10, 2010
Mozilla Community Based Automated Testing
Wednesday, November 10, 2010
Mozilla Community Based Automated Testing
Wednesday, November 10, 2010
Overview of How we succeed with Community Testing
• Continuous Integration is the foundation of today’s community testing
• Utilizing Community Feedback
• Engage in Manual Testing efforts
• Engage Community focused automated testing in the near future
Wednesday, November 10, 2010
Community Stewardship
Wednesday, November 10, 2010
Community Stewardship
Wednesday, November 10, 2010
Community Stewardshipquality.mozilla.org
Wednesday, November 10, 2010
Reaching Out to the Communityquality.mozilla.org
quality.mozilla.org
Wednesday, November 10, 2010
A call out to you!
Wednesday, November 10, 2010
Thank you
Questions?
Wednesday, November 10, 2010