WHQL Code Coverage WHQL Code Coverage Prototype ProgramPrototype ProgramAndy WenAndy Wen

22

AgendaAgenda What is Code Coverage Prototype Program?What is Code Coverage Prototype Program?

A prototype program to address hardware independent A prototype program to address hardware independent driver signingdriver signing

How does Code Coverage Work?How does Code Coverage Work? Hardware Dependent Drivers versus Hardware Dependent Drivers versus

Hardware Independent DriversHardware Independent Drivers Code Coverage NotesCode Coverage Notes Code Coverage TestingCode Coverage Testing

Submission ProcessSubmission Process Code Coverage ToolsCode Coverage Tools Best PracticesBest Practices

Call To ActionCall To Action Questions and AnswersQuestions and Answers

33

What Is Code Coverage What Is Code Coverage Prototype ProgramPrototype Program A new innovative solution to test & sign driversA new innovative solution to test & sign drivers Hardware independent driversHardware independent drivers

Display Mirror Driver Display Mirror Driver Virtual CD DriverVirtual CD Driver Sound enhancement driverSound enhancement driver

Hardware dependent driversHardware dependent drivers Device hardware does not fall under any existing WHQL programsDevice hardware does not fall under any existing WHQL programs Standalone Windows 2000 driver for hardwareStandalone Windows 2000 driver for hardware

Current Status of Code Coverage Prototype ProgramCurrent Status of Code Coverage Prototype Program Target goal is 70% of code covered during testingTarget goal is 70% of code covered during testing 35 companies participated, 58 drivers (including updates) 35 companies participated, 58 drivers (including updates) Average request for test result parsing per driver submission to achieve Average request for test result parsing per driver submission to achieve

70% is 4 times70% is 4 times

44

Code Coverage NotesCode Coverage Notes No Source code is exposed to MicrosoftNo Source code is exposed to Microsoft

Because we “stamp” a compiled driver, we do not see Because we “stamp” a compiled driver, we do not see source codesource code

Legal agreement is required for Code Coverage Legal agreement is required for Code Coverage participationparticipation Uses technology under development, not ready for wide Uses technology under development, not ready for wide

distributiondistribution Not covered under existing agreementsNot covered under existing agreements

Code coverage is not the ultimate solution for Code coverage is not the ultimate solution for ensuring driver qualityensuring driver quality It is a testing metric that can be applied to all driversIt is a testing metric that can be applied to all drivers Not all code is executedNot all code is executed

Conditional branch error pathsConditional branch error paths

55

What Is Code Coverage What Is Code Coverage Prototype ProgramPrototype Program BenefitsBenefits

Measure code testing effectiveness of existing test toolsMeasure code testing effectiveness of existing test tools Use your own test toolsUse your own test tools You control your own testing scheduleYou control your own testing schedule Get driver digitally signed by WHQLGet driver digitally signed by WHQL Clear measurement of what is being tested in the driverClear measurement of what is being tested in the driver Easier to test the appropriate sections of updated driverEasier to test the appropriate sections of updated driver Enable you to easily select test cases for the future submissionsEnable you to easily select test cases for the future submissions Help developer to identify redundant/unused functions Help developer to identify redundant/unused functions

in the driverin the driver

66

How Does Code Coverage Work?How Does Code Coverage Work?

The driver binary must be recompiledThe driver binary must be recompiled WHQL provides information (instructions and additional compile and WHQL provides information (instructions and additional compile and

link flags)link flags) Kernel-mode driver binary onlyKernel-mode driver binary only No Source code is exposed to MicrosoftNo Source code is exposed to Microsoft

Developer follows instructions and buildsDeveloper follows instructions and builds An instrumented kernel-mode driver binary used in testingAn instrumented kernel-mode driver binary used in testing A “retail” driver that can be signed and distributedA “retail” driver that can be signed and distributed

Developer sends instrumented kernel-mode driver binary to WHQLDeveloper sends instrumented kernel-mode driver binary to WHQL WHQL “stamps” the instrumented kernel-mode driver binaryWHQL “stamps” the instrumented kernel-mode driver binary WHQL returns “stamped” driver binary and provides monitoring toolWHQL returns “stamped” driver binary and provides monitoring tool Developer tests the “stamped” driver binary and sends encrypted Developer tests the “stamped” driver binary and sends encrypted

log to WHQLlog to WHQL WHQL parses log and sends coverage report to DeveloperWHQL parses log and sends coverage report to Developer If target coverage 70% is met or exceeded, Developer makes a If target coverage 70% is met or exceeded, Developer makes a

WHQL submission with “retail” version of driver for signingWHQL submission with “retail” version of driver for signing

77

Code Coverage TestingCode Coverage Testing Preliminary requirementsPreliminary requirements

Contact Code Coverage Prototype Program (Contact Code Coverage Prototype Program (WHQLCCP@Microsoft.comWHQLCCP@Microsoft.com) )

The Code Coverage Prototype Program is offered through both The Code Coverage Prototype Program is offered through both the WHQL U.S.A. and the WHQL Japan labs and is available only the WHQL U.S.A. and the WHQL Japan labs and is available only if the following criteria are metif the following criteria are met The driver is NOT one of the following types The driver is NOT one of the following types

Device driver stack (bus driver or bus filter-driver) Device driver stack (bus driver or bus filter-driver) File system filter driver File system filter driver Transport Driver Interface (TDI) driver Transport Driver Interface (TDI) driver 64-bit driver64-bit driver

The driver is installed with a device setup Information file (INF)The driver is installed with a device setup Information file (INF) The driver runs in kernel modeThe driver runs in kernel mode

Accept Code Coverage AgreementAccept Code Coverage Agreement

88

Code Coverage TestingCode Coverage Testing Submission ProcessSubmission Process

1.1. Contact WHQL Code Coverage Prototype Program Contact WHQL Code Coverage Prototype Program whqlccp@microsoft.comwhqlccp@microsoft.com2.2. Accept Code Coverage AgreementAccept Code Coverage Agreement3.3. Establish a Winqual Account on Establish a Winqual Account on https://https://winqual.microsoft.comwinqual.microsoft.com

Click "How to Establish an Account" and follow the steps; Click "How to Establish an Account" and follow the steps; Your company must obtain a VeriSign Code-3 Signing ID, which can take 2 - 4 weeksYour company must obtain a VeriSign Code-3 Signing ID, which can take 2 - 4 weeksWhile waiting, proceed to step 4While waiting, proceed to step 4

4.4. Download HCT 11.2 at Download HCT 11.2 at http://www.microsoft.com/http://www.microsoft.com/whdc/hwtest/device/default.mspxwhdc/hwtest/device/default.mspx. . You must use a logo'd system running Windows XP Professional to test devices;You must use a logo'd system running Windows XP Professional to test devices;For a USB device, you need a USB 2.0 host controller and an external USB 2.0 For a USB device, you need a USB 2.0 host controller and an external USB 2.0 self-powered hub between the test system and the test device; HCT documentation self-powered hub between the test system and the test device; HCT documentation describes the test environmentdescribes the test environment

5.5. Instrument driverInstrument driver6.6. Run TestsRun Tests7.7. Return coverage result to WHQL Code Coverage Prototype ProgramReturn coverage result to WHQL Code Coverage Prototype Program8.8. After the test result meets all the requirements, you can make a submission by using After the test result meets all the requirements, you can make a submission by using

the account that you establish from #3the account that you establish from #3

99

Hardware Dependent DriversHardware Dependent Drivers ExampleExample

Personal Digital Assistants (PDAs)Personal Digital Assistants (PDAs) Cell-phone connectivity adaptersCell-phone connectivity adapters Biometric security devicesBiometric security devices Hardware-based software protection devicesHardware-based software protection devices

Device follows the Windows Logo Program requirementsDevice follows the Windows Logo Program requirements Driver follows the Windows DDK for the target OSDriver follows the Windows DDK for the target OS Pass HCT 11.2 or later version of Unclassified\Universal kitPass HCT 11.2 or later version of Unclassified\Universal kit Use your own test cases and test toolsUse your own test cases and test tools Meets or exceeds 70% of code coverageMeets or exceeds 70% of code coverage 2 hardware samples are required for submission2 hardware samples are required for submission Up to 30 days of submission turnaround time (Goal of 2 weeks)Up to 30 days of submission turnaround time (Goal of 2 weeks) Driver is digitally signedDriver is digitally signed ““Designed for Windows” logo for hardware is issuedDesigned for Windows” logo for hardware is issued Windows Catalog listingWindows Catalog listing

1010

Hardware Independent DriversHardware Independent Drivers ExampleExample

Network VPN driversNetwork VPN drivers Virtual CD driversVirtual CD drivers Sound enhancement driversSound enhancement drivers

Driver follows the Windows DDK for the target OSDriver follows the Windows DDK for the target OS Pass all the applicable test tools from HCT 11.2 or later versionPass all the applicable test tools from HCT 11.2 or later version Use your own test cases and test toolsUse your own test cases and test tools Meets or exceeds 70% of code coverageMeets or exceeds 70% of code coverage Provide WHQL a copy of the software uses the driverProvide WHQL a copy of the software uses the driver Up to 48 hours of submission turnaround timeUp to 48 hours of submission turnaround time Driver is digitally signedDriver is digitally signed No “Designed for Windows” hardware logo will be issuedNo “Designed for Windows” hardware logo will be issued May be eligible for “Designed for Windows” software logo processMay be eligible for “Designed for Windows” software logo process

1111

Code Coverage TestingCode Coverage Testing Your own test cases and test toolsYour own test cases and test tools Core Driver Quality HCT TestsCore Driver Quality HCT Tests

Public ImportPublic Import SignabilitySignability ChkINFChkINF Driver VerifierDriver Verifier Device Path ExerciserDevice Path Exerciser

Core Functionality HCT TestsCore Functionality HCT Tests ACPI StressACPI Stress Enable/Disable Device I/OEnable/Disable Device I/O

Core Bus HCT TestsCore Bus HCT Tests PCI Specific test toolsPCI Specific test tools USB Specific test toolsUSB Specific test tools 1394 Specific test tools1394 Specific test tools

1212

Code Coverage TestingCode Coverage Testing Core Device Specific HCT TestsCore Device Specific HCT Tests

DVD Annex – DecodeDVD Annex – Decode NDISNDIS DirectMusicDirectMusic

Best PracticesBest Practices

1313

Best PracticesBest Practices Multiple Processor environment makes it easier to find weak codeMultiple Processor environment makes it easier to find weak code

Use a Multiple Processor (or Hyper-Threading) system for testingUse a Multiple Processor (or Hyper-Threading) system for testing Use two or more test devices simultaneously (in the same system)Use two or more test devices simultaneously (in the same system) Test on fastest processorTest on fastest processor Enable Driver Verifier while running all test toolsEnable Driver Verifier while running all test tools

Enable Driver Verifier prior to running any HCT Tools and your own Enable Driver Verifier prior to running any HCT Tools and your own test toolstest tools

Don't leave "KdPrint()" in your retail driver binariesDon't leave "KdPrint()" in your retail driver binaries Remove this, and any other debug code from your retail driver binaries Remove this, and any other debug code from your retail driver binaries

before you begin testingbefore you begin testing Device Specific ApplicationsDevice Specific Applications

IMPORTANT: If a device requires an application, ensure that the appIMPORTANT: If a device requires an application, ensure that the app Calls RegisterDeviceNotification on the device handleCalls RegisterDeviceNotification on the device handle Listens for WM_DEVICECHANGE in its message loopListens for WM_DEVICECHANGE in its message loop On receiving this message with type DBT_DEVICEQUERYREMOVE, it On receiving this message with type DBT_DEVICEQUERYREMOVE, it

closes the device handle so the device can be properly removedcloses the device handle so the device can be properly removed

1414

Call To ActionCall To Action Contact Contact WHQLCCP@microsoft.comWHQLCCP@microsoft.com to to

participate the programparticipate the program Refer to Windows DDK and Windows Logo Refer to Windows DDK and Windows Logo

Program requirements when developing new Program requirements when developing new devices and driversdevices and drivers

Subscribe WHQL Weekly Newsletter for the Subscribe WHQL Weekly Newsletter for the program updates program updates http://www.microsoft.com/whdc/newsreq.mspxhttp://www.microsoft.com/whdc/newsreq.mspx

Ask The Expert – Thursday 05/06/2004 from Ask The Expert – Thursday 05/06/2004 from 12:00 P.M. to 1:30 P.M.12:00 P.M. to 1:30 P.M.

1515

Community ResourcesCommunity Resources WHQL WebsiteWHQL Website

Device and System testingDevice and System testing http://www.microsoft.com/whdc/hwtesthttp://www.microsoft.com/whdc/hwtest Winqual Online SubmissionWinqual Online Submission http://winqual.microsoft.com http://winqual.microsoft.com

Attend a free chat or WebcastAttend a free chat or Webcast http://www.microsoft.com/communities/chats/default.mspx http://www.microsoft.com/communities/chats/default.mspx http://www.microsoft.com/seminar/events/webcasts/default.mspx http://www.microsoft.com/seminar/events/webcasts/default.mspx

Locate a local user group(s)Locate a local user group(s) http://www.microsoft.com/communities/usergroups/default.mspx http://www.microsoft.com/communities/usergroups/default.mspx

Non-Microsoft Community SitesNon-Microsoft Community Sites http://www.microsoft.com/communities/related/default.mspx http://www.microsoft.com/communities/related/default.mspx