Marketing & Sales Roundtable

Beta Program Best Practices

November 2004

2©2004 Rosemary Remacle

11:30 – Introductions/Objectives

11:45 - Beta Program Best Practices: Presentation and Discussion

1:15 – Summary and Wrap-up

Agenda

3©2004 Rosemary Remacle

What Do We Mean: “Beta Test”?

• Testing of a working product performed by real users in a production environment

• Last test before formal product launch, production shipments begin (GA)

• A test of all elements and process required for successful GA product

The Beta program definition will be driven by product category and type of users – HW, SW, Services

4©2004 Rosemary Remacle

Objectives for a Beta Program

Product

• Gain deeper understanding of how product works in a ‘real world’ environment

• Stress products more completely and in different ways than in a lab setting

• Fine-tune product/service functionality (tweak vs. major overhaul)

• Test configuration planning, installation and support procedures and documentation (including technology and delivery partners where applicable)

• Test specifications and metrics to ensure quality levels of the specifications in a range of environments (e.g. usability, reliability, performance, etc.)

Marketing

• Test corporate and product positioning platform, message architecture and plans for customer segment programs

• Develop a systematic way for capturing customer information and input

• Build key early customer relationships, generate references and referrals

• Understand weaknesses in company infrastructure required to support all customers’ requirements (total product)

5©2004 Rosemary Remacle

Beta Program Development

• Collect and analyze user

feedback

• Identify, prioritize

issues to be resolved

• Develop integrated functional response plan

• Refine beta testing process

• Summary beta report doc

• Define beta testing process, participants

andobjectives

• Produce integrated functional plan

- Engineering- Marketing- Sales

- Professional Services

- Support- Partners

• Develop criteria for beta test customers

• Identify and Recruit beta customers and set expectations

• Create beta program

documentation

• Implement beta program

- Product Distributed to Beta Users

- Survey Vehicles in

place

- Reporting Process

- Timelines, Milestones

6©2004 Rosemary Remacle

Sample Criteria for Selecting Beta Customers

Criteria (Hi – Med – Lo)

Member of identified market entry customer segment(s)

Does their application test stress the aspects of the product that need it?

Match of company’s solution to perceived customer critical need

Motivated to make fast decision within market window/Beta program manager in place

Early adopter technology adoption cycle profile: high tolerance for Beta problems

Referenceability: Big name, technology leader and/or key influencer in market entry segment, willing to be reference

Existing business relationships with key members of Beta company team, users

Internal Champion

Geographic proximity for ease of support

Technology or market leader

7©2004 Rosemary Remacle

Beta Program Best Practices: The Short List

• Don’t abuse the generosity of beta customers with non customer-worthy product – develop a “phased beta approach” to address risks of putting beta product in big influential prospective customer

• Choose and recruit your beta customers, users to match market entry customer segment(s)

• Drive for confidentiality (NDAs, Beta agreements, etc.)• Manage the program and beta users, communicate (bi-directional) regularly

(program managers in both your company and customers)• Provide a simple, easy method for beta users to give feedback, make them feel like

part of the your team• Encourage beta users to communicate with each other• Respond quickly to all requests and problems • Document, document, document

8©2004 Rosemary Remacle

What Motivates Beta Testers ?

• Early adopter technology adopter cycle value profile• Ability to contribute to product development process, be perceived as expert• ‘Community’ participation• Exclusive access to new product/Head start on resolution of their business problem• ‘Free’ product/Incentives• Career advancement

Where do You Find Them?• Sales Force/Executive Team relationships• Partners (Channel, complementary products, etc.) • Internet groups/Blogs/Bulletin Boards and Lists• “Dead prospects”• Customers’ competitors• Other?• User Groups/SIGs

9©2004 Rosemary Remacle

Elements of Beta Program Research Plan

• [Realistic] sampling strategy• - # of customers • - # of participants• - Functional representation

• Efficient user feedback (technical and marketing) collection tools (on-line, web page)

• Third party tools (e.g. web surveys - Survey Monkey, Zoomerang) for simple, measurable inputs

• - In-depth, qualitative (e.g. telephone/in-person interviews) for exploration of issues/requirements and positioning strategy validation

• - Forum discussions (open and hosted: web forum, conference calls, WebEx)• Surveys: design for structured analysis!

• - Topics to be covered by function• - Technical and marketing metrics• - Qualitative ‘discussion’ topics

10©2004 Rosemary Remacle

Beta Test Plan

• Definition- Project parameters, goals, activities and participants- Internal resources required, responsibilities identified-Foundation for beta report/summary

• Table of Contents- Product definition (in current state)-Target customer segment(s) – “user spec” (time, HW, etc.)- Parameters (# of participants, timeline, etc.)- Goals (UI, support requirements, testimonials, etc.)- Incentives (as appropriate to situation)- Beta team action items and responsibilities- Etc.

11©2004 Rosemary Remacle

Post-Beta Test

• Make customer feedback accessible, institutional customer intelligence (Analyze customer feedback in structured way and document all issues)

• Prioritize issues (severity and repetition) and develop an integrated cross-functional action plan

• Incorporate beta user feedback into market-driven product roadmap• Ensure all company infrastructure requirements identified have been fully

addressed• Review beta program process and execution performance and develop

improvement plan• Send out incentives (where appropriate) and thank individual Beta users

personally (phone call, email, etc.) in a timely fashion

12©2004 Rosemary Remacle

Some Tips for Managing a Beta Program

• Establish well-integrated cross-functional team with representation from all stakeholders (executive, marketing, sales, engineering, support, partners, etc)

• Assign program owner (generally marketing) and define clear team member roles and responsibilities

• Agree cross-functionally on criteria for choosing a Beta site • Promote importance and value of beta test to all functional

organizations and share the beta information with all who need it

13©2004 Rosemary Remacle

Key Beta Test Mistakes

• “Wrong” beta testers accepted• Recruiting family or friends skews perspective• Poorly managed beta testers and/or test data• Prohibitive time commitment, resulting in drop-outs• Viewing Beta program as revenue source• Beta test period too short or too long• Too few or too many beta users (difficult to manage)

“Choosing the wrong beta testers is the best way

I know to provide useless results”

14©2004 Rosemary Remacle

Summary: A Successful Beta Program

• Product works as advertised • Benefits are documented, validated and measured

(overall, feature-specific)• Product configuration, installation and customer support

processes and documentation meet customer expectations

• Unanticipated or un-addressed total product requirements are identified and prioritized with level of criticality determined

15©2004 Rosemary Remacle

Presenters

• Rosemary Remacle, ConsultantMarket Focus408-244-0412rosemary@mktfocus.net