How to Manage a Quality Device Driver

Donald D. BurnChief Software Architectburn @ acm.orgReliable Technologies, Inc

Session OutlineSession Outline

Fundamentals

Documentation

Development

Maintenance

Session GoalsSession Goals

Attendees should leave this session with the following:

A better understanding of the driver development process

Approaches to driver development that canImprove the quality of device drivers

Speed time to market

Reduce long-term costs

FundamentalsFundamentals

Driver development is different than application work

Recognizing the differences is the best way to improve driver efforts

Fundamentals – Drivers Are a SpecialtyFundamentals – Drivers Are a Specialty

Driver writers need training

Driver development continually changesContinue to send your developers to the Microsoft Windows Driver Developers Conference

Watch the Microsoft Windows Hardware and Driver Central website for new information

Experts can be cheaper

There are many types of drivers and no developer is a specialist in all of them

Fundamentals – Drivers & Product Fundamentals – Drivers & Product CreationCreation

Engage driver development at the product design phase

Involve driver development with hardware design

Start driver planning early

Schedule lots of time for testing

Fundamentals – No More Win9xFundamentals – No More Win9x

You cannot design for just a single processor x86 with < 4GB memory

We now have:Multiprocessors everywhere

Low-cost memory for more than 4 GB

64-bit processors

Headless and/or multi-user systems

Hot-plug devices widespread

Fundamentals – Fundamentals – Drivers Are Part of WindowsDrivers Are Part of Windows

Build the driver using the DDK’s tools

Install the driver following Windows best practices

Related Session – “DMI: Driver Install Frameworks Tools and Strategies”

Test the driver to WHQL standards and beyond

Validate the driver on a variety of platforms

Submit the driver to WHQL

Fundamentals – Windows DriversFundamentals – Windows Drivers

Drivers are derived from existing driversBugs can be propagated to other drivers

Windows Driver Foundation addresses this reuse

Drivers typically need small teamsRequires good teamwork

Adding manpower to a late software project makes it later (The Mythical Man-Month, Fred Brooks)

Fundamentals – Driver CostFundamentals – Driver Cost

Rule of thumb: System software costs twenty times the salaries of the developers for the life of the product

Typically two thirds of the cost of software is in maintenance and support

Poor quality can push support costs to a much higher percentage

Rule of thumb: If you are lucky 10% of your dissatisfied customers will let you know they are unhappy!

DocumentationDocumentation

Critical for tomorrow

Gives you a baseline

Allows for fix priced quotes

Can be an incremental effort

Can be contracted

Documentation – Hardware DocumentsDocumentation – Hardware Documents

Hardware GuidelinesSpecify common criteria for good hardware

Create your guidelines from:Windows Logo Program Requirements

Industry standards such as: PCI, USB, etc

Your company’s past experience

Driver developers need to give input

Hardware SpecificationDocument programming the device

Joint effort of hardware and software

Documentation – Coding GuidelinesDocumentation – Coding Guidelines

Styles for interfaces and security

Rules on undocumented and deprecated functions

Diagnostic model

Standard tool set for building drivers

Build options for reliability and testing

Documentation – Driver SpecificationDocumentation – Driver Specification

Define all interfaces and their security

Note the Installation Model

Specify projected performance goals

Document deviations from the guidelines and standards

Identify the ancestry of the driver source code

Explain the design for unique features

May be part of a system architecture / larger product

Documentation – Test DocumentsDocumentation – Test Documents

Test Guidelines

Engineering Verification Test Plan

QA and Certification Plan

Beta Test Plan

DevelopmentDevelopment

Cost to Fix Errors

Design Build Debug Test QA

$

Rule of thumb: A defect that costs $1 to fix on the programmer's desktop costs $100 to fix once it is incorporated into a complete program and many thousands of dollars if it is identified only after the software has been deployed in the field. (Building a Better Bug Trap - The Economist June 2003)Microsoft provides tools for finding many errors when the cost is low!Perform code reviews to improve quality

Development – Driver Development KitsDevelopment – Driver Development Kits

Use the latest Device Driver Kit (DDK) Works for all systems back to Windows 2000

Most up to date APIs, tools, samples and documentation

It is FREE!!!

Use the compiler and tools from the DDKWindows itself is built with these tools

Development – Source ControlDevelopment – Source Control

Source control is a debugging tool

Encourage checking in source often – as much as multiple times a day

Track individual changes to isolate errors

Provides an undo capability for development

Use the source control data to feed the code review

Nothing is done until it is under source controlSet your milestones based on when a feature is tested and checked in

Use the above as the basis for determining if the project is on schedule

Development – Debugging EnvironmentDevelopment – Debugging Environment

A test machine separate from the development system

The test machine should be multi-processor or at least hyper-threaded

Higher end systems find more bugs than lower end systems

Consider hot-plug drives for easy OS changes

Additional test machines:Other CPU architectures and other HAL’s

Large multiprocessor (8 or more if possible)

Large memory > 4GB

Hot-plug support

Development – Compile for ReliabilityDevelopment – Compile for Reliability

Compile with /WALL /WX

Build with PREfast

Enable Deprecated Function checking

Use C_ASSERT – compile time assertions

Consider PC-Lint and custom checking

Validate INF files with ChkINF

More tools are comingStay up to date with the latest tools

Provide time for your developers to learn the new tools

Development – Execute for QualityDevelopment – Execute for Quality

Debug drivers under the Checked Build of Windows

Run drivers under Driver Verifier

Enable the Call Usage Verifier

Consider compiler runtime checks

Use ASSERT – runtime assertions

Conditionally instrument the driver

Allocate and free memory with tags

Development – Test for ValidityDevelopment – Test for Validity

Engineering Verification TestBasic tests for most functionality

Quality Assurance and Certification TestsTest drivers with HCT

Custom tests

Integration tests

Stress tests

Regression tests

Development – Profile Development – Profile

Code coverageIdentifies code with little or no utilization

Shows where tests can use improvements

Performance ProfilingIdentify performance hotspots to improve

Can indicate errors in a driver

Use the above data to perform a code review of the driver

Development – Code ReviewDevelopment – Code Review

Use walkthroughs during development

Have a formal inspection before the driver leaves development

Consider a review by an outside firm

Code reviews find twice the number of bugs as testing!

Development – Threat AnalysisDevelopment – Threat Analysis

Perform a threat analysis on your driversIdentify all possible avenues of a attack on your driver

Writing Secure Code by Howard and LeBlanc is an excellent reference

Drivers need secure interfacesSecure device objects

Check all data accessible from user space

Be careful with IOCTL’s

Add additional security checks for dangerous operations

Do not develop “all-in-one” interfaces that can be abused

MaintenanceMaintenance

Provide reporting and diagnostics in your driver

Take advantage of Online Crash Analysis

Create a feedback model to stop bugs from recurring

Maintenance – Customer ReportsMaintenance – Customer Reports

Event LoggingDon’t log the ordinary!

Use unique error log entries

WMI performance data

Maintenance – Diagnostics Maintenance – Diagnostics

WPP tracingExtremely low performance impact

Not decodable by users

WinDBG extensions

Custom dump data

Maintenance – OCA DataMaintenance – OCA Data

Windows Error Reporting – provides direct customer data about Windows crashes

Microsoft’s Online Crash Analysis database –prioritizes problems with your driver

Maintenance – Defect FeedbackMaintenance – Defect Feedback

Let the developer know

Fix all the drivers that use the code

Create a regression test so the bug does not reappear

Call To ActionCall To Action

Update your practices to improve driver quality

Involve your driver developers early in the product cycle

Create a culture of quality for your drivers

Community ResourcesCommunity Resources

Windows Hardware & Driver Central (WHDC)www.microsoft.com/whdc/default.mspx

Technical Communitieswww.microsoft.com/communities/products/default.mspx

Non-Microsoft Community Siteswww.microsoft.com/communities/related/default.mspx

Microsoft Public Newsgroupswww.microsoft.com/communities/newsgroups

Technical Chats and Webcastswww.microsoft.com/communities/chats/default.mspx

www.microsoft.com/webcasts

Microsoft Blogswww.microsoft.com/communities/blogs

Additional ResourcesAdditional Resources

Web Resources:Windows Hardware and Driver Centralhttp://www.microsoft.com/whdc

Related SessionsThe WDK for Engineering Managers and Product Planners: An Introduction

Windows Driver Foundation: An Introduction

DMI: Driver Install Frameworks Tools and Strategies