Upload
azia
View
81
Download
0
Embed Size (px)
DESCRIPTION
MGS Testing. A High Level Overview of Testing in Microsoft Games Studio Joe Djorgee – Test Lead. Today’s Topics. Staffing Philosophy Testing Roles Basic Org Structure Specialized Test L a bs On-Boarding Program Why Quality Test Talent is Important Working with Partners - PowerPoint PPT Presentation
Citation preview
MGS Testing
A High Level Overview of Testing in Microsoft Games Studio
Joe Djorgee – Test Lead
Today’s Topics
» Staffing Philosophy» Testing Roles» Basic Org Structure» Specialized Test Labs» On-Boarding Program» Why Quality Test Talent is Important» Working with Partners» Test's Role in the Cycle» Automation
Staffing Philosophy » Involve the right Testing at the right time:
» Emphasize early involvement of Test in product development… “Test Upstream”
» Keep a smaller team of FTE in place longer, ramp up when RTM is more predictable, hold off on using large Testers resources until end of cycle
» Harness benefits of centralization by sharing technologies and methods, moving resources as needed, focusing training, and removing redundancies
» Minimize overtime and avoid “carrying the standing army” due to late project slips
» Encourage small onsite Testing for external development partners – they act as first line of Testing for BVT’s, milestone drops, and as GTO liaisons
Testing Roles» Full Time Test Manager
» Manages the teams for multiple projects» Project Test Lead
» Responsible for Test efforts for a specific project» Software Dev Engineer (Shared)
» Spend ~85% of time coding/scripting tools/automation for specific title
» Software Dev Engineer in Test (SDET)» Spend ~25% of time coding/scripting, own specific Test areas and
report results from Test Passes» Testers (STE/TA)
» Creates and executes Test Cases, play through game, runs specified Test Passes
» Can be full time or temporary Testers» Core, Core Assist, Play Testers
» Specialized Labs» PC Compatibility Lab» Multiplayer/Network Lab
Basic Org StructureFull Time Test Manager
Specialized Labs
Project Test LeadArea Lead
STE/SDET
STE/SDET
STE/SDET
Area Lead
STE/SDET
STE/SDET
STE/SDET
Area Lead
STE/SDET
STE/SDET
STE/SDET
Shared Dev Team Full Time Lead of Temp Testers
Core Temp Testers
Core Assist Testers
Play Testers
Specialized Labs» PC Configuration (Config) Lab
» Testing on the highest possible market share percentage of computer hardware and OEM systems
» Extensive, in depth product Testing under all supported operating systems against a vast array of hardware driver versions
» Work with other internal teams (Beta, PSS) to track, reproduce and regress all real world bugs
» Extensive coverage on input devices as well (game controllers, carious keyboards, mice, etc.)
» Muliplayer/Networking Labs» Provides a clean lab environment to Test multiplayer
functionality» Scheduled Passes on various connections and hardware» Focuses on technical network configurations and
connection issues
Temporary Tester On-Boarding Program
» Formal on-boarding is provided to all new Testers in their first week on assignment
» The on-boarding program contains a lot of information» Helps our temporary Testers learn the tools and processes required to
effectively Test our titles» How do I do this?» What is this?» Why am I doing this?» What does success mean?
» Areas covered during this process include» An overview of the game development cycle» Bug writing» General Testing principles
» Assessment» Facilitator is present to assist new temporary Testers and assess skill
level» Evaluations/final exam are required at the end of each section
Why Quality Test Talent is Important
» Testing of titles is growing increasingly complex and challenging, and a high performing and proactive team is necessary for predictable RTM’s
» Test teams are doing more than just verifying reliability and functionality
» Higher skilled teams arguably reduce costs in other more expensive functions (i.e. dev)
Working with Partners» Establish strong working relationship with Developer
point of contact(s)» Daily/weekly status» Direct phone/video conferences
» Sync up on project issues by using a single bug database
» Test Pass summaries – Milestones, Performance, Certification
» Provide gameplay and Design feedback » Share Test tools» Acclimate Partner to MS processes
Test's Role in the Cycle» Testing is an end to end process occurring
throughout the product cycle» Early Test involvement exposes issues early,
when they are least costly… “Testing Upstream”» Test assessment is critical to accurate
forecasting, planning and production» A portion of the Testing complements the efforts
of other disciplines - Localization, Play balancing, Usability, Audio, Fun, etc.
Automation
» Automation helps to improve Test coverage and efficiency
» Microsoft has a long history of engineering Test solutions
» Test and Dev teams work to implement hooks to aid Test efforts
Automation
» Infrastructure Examples» Distributed automation frameworks» Build recognition, replication, and
distribution tools» Build creation and compile tools» Coding our own debug hooks» Test Case managers» Test Case generators and optimizers
Automation
» Administrative/Productivity Examples» Game event tracking and coverage
tools» Bug filing and reproduction tools» Automated bug frequency tracking
used to help determine fix priorities» Visual reporting techniques» Consistent Task Management system
for Testers» Resource usage reporting
Automation Examples
Automation Examples
Automation Examples
Questions?