File Systems_IFS FAQ.pdf

8/14/2019 File Systems_IFS FAQ.pdf

1/45

7/12/13 File Systems:IFS FAQ

www.osronline.com/article.cfm?article=17 1/45

Kernel Debugging &Crash AnalysisLABSanta Clara, CA9-13 Sep 2013

Windows Internals andSW DriversLABSanta Clara, CA16-20 Sep 2013

Writing WDF DriversLABBoston/Waltham, MA7-11 Oct 2013

Developing File SystemsSeattle, WA5-8 Nov 2013

Windows Debugger

Checked BuildDownloads29-Apr-10

Debugging Symbols

WDK Documentation

Windows WDK

Need to develop aWindows file systemsolution?

We've got a kit for that .

Need Windows internalsor kernel driver expertise?

Bring us your mostchallenging project - w e can

help!

Everything Windows Driver Development

Fri, 12 Jul 2013

99237 members

Abo ut This Sit eWh at's New?OSR Dev BlogThe NT Insider The BasicsFile SystemsDownloadsListServer / ForumDriv er J obsSto re

Express Link s UMDF V2 -- It's KMDF

Com patible!

OSR Seminar Schedu le

Wdf Send: Are There

REALLY Three Usef ulVariants?

Turning a Breakpoint int o aBusy point Inves tigating a NULL

Pointer D ereference

FILE SYSTEMS

IFS FAQOSR Staff | Published: 14-Nov-01| Modified: 23-Oct-07

Q1 How difficult is it to port a Windows 9x based file sy stem or file sys temfilter driver to Windows NT/2000/XP?

Q2 Is there a WDM model for file systems or file system filter drivers?

Q3 How does a file system or file system filter driv er handle PnP, Power Management, and WMI in Windows 2000/XP?

Q4 How do file systems get loaded on Windows NT/20 00/XP?

Q5 How is the file system's device object found?

Q6 What's the right way to cancel a CREATE request in my filter drive r?

Q7 How do I deal with file-sharing issues in my filter driver?

Q8 What's the proper way to install my file system or fil e system filter dr iver?

Q9 Does WHQL logo file systems or file s ystem filter dr ivers?

Q10 When my customers install my file system o r filter driver product, will theyget that nasty pop-up saying "this is not a signed driver...Microsoftrecommends that you do not continue"??

Q11 How is cache coherency handled when a file is opened for "ordinary"(cached) I/O and also opened memory mapped?

Q12 Can I leave integration with the cache manager out of my product tosimplify things? What's the impact of doing this?

Q13 Must I support Fast I/O in my file system or filter, and where is Fast I/Odocumente d?

Q14 Filt ering file systems doesn't look that hard... I have a free sample Idownloaded off the web. What are the limitations of the currently available filesystem filter driver samples?

Q15 Is Rajeev Nagar's book accurate, and should I use it as a reference?

Q16 Are there any other books on NT/2K/XP file system or file sys tem filter development? What other resources are available to help me??

Q17 What are the primary differences between filtering rdr and filtering a localfile system?

Q18 In general, when can I use the IFS kit's mini-rdr model, and when is it bestfor me to write my own rdr from scratch?

Q19 Does Microsoft offer support for developing file systems or file systemfilters?

Post Your Comments.

Print this article.

Email this article.
http://www.osronline.com/article.cfm?id=610http://www.osronline.com/article.cfm?id=611http://www.osronline.com/article.cfm?id=612http://www.osronline.com/article.cfm?id=612http://www.osronline.com/article.cfm?id=613https://www.osronline.com/custom.cfm?name=index_fullframeset.cfm&pageURL=https://www.osronline.com/store/index.cfmhttp://www.osronline.com/page.cfm?name=ListServerhttp://www.osronline.com/section.cfm?section=17http://www.osronline.com/section.cfm?section=16http://www.osronline.com/section.cfm?section=45http://www.osronline.com/page.cfm?name=OSR%20Online?http://www.osronline.com/custom.cfm?name=login_joinok.cfmhttp://www.osronline.com/login.cfmhttp://www.osronline.com/login.cfmhttp://www.osronline.com/article.cfm?article=17#Q15http://www.osronline.com/article.cfm?article=17#Q15http://www.osronline.com/article.cfm?article=17#Q14http://msdn.microsoft.com/en-us/windows/hardware/gg463028http://www.osronline.com/article.cfm?id=259http://www.osronline.com/article.cfm?article=17#Q12http://www.osronline.com/article.cfm?article=17#Q11http://www.osronline.com/article.cfm?article=17#Q10http://www.osr.com/fsd.htmlhttp://www.osronline.com/article.cfm?article=17#Q10http://www.osr.com/wdf.htmlhttp://www.osronline.com/article.cfm?article=17#Q7http://www.osr.com/swdrivers.htmlhttp://www.osr.com/swdrivers.htmlhttp://www.osronline.com/article.cfm?article=17#Q4http://www.osr.com/debug.htmlhttp://www.osronline.com/article.cfm?article=17#Q3http://www.osr.com/debug.htmlhttp://www.osronline.com/article.cfm?article=17#Q2http://www.osronline.com/article.cfm?article=17#Q1http://www.osronline.com/article.cfm?article=17#http://www.osronline.com/index.cfmhttp://www.osronline.com/index.cfmhttp://www.osronline.com/adClick.cfm?id=73http://www.osronline.com/article.cfm?article=17#http://www.osronline.com/custom.cfm?name=articlePrint.cfm&id=17http://www.osronline.com/articleCommentPost.cfm?id=17#commentshttp://www.osronline.com/article.cfm?article=17#Q19http://www.osronline.com/article.cfm?article=17#Q18http://www.osronline.com/article.cfm?article=17#Q17http://www.osronline.com/article.cfm?article=17#Q16http://www.osronline.com/article.cfm?article=17#Q15http://www.osronline.com/article.cfm?article=17#Q14http://www.osronline.com/article.cfm?article=17#Q13http://www.osronline.com/article.cfm?article=17#Q12http://www.osronline.com/article.cfm?article=17#Q11http://www.osronline.com/article.cfm?article=17#Q10http://www.osronline.com/article.cfm?article=17#Q9http://www.osronline.com/article.cfm?article=17#Q8http://www.osronline.com/article.cfm?article=17#Q7http://www.osronline.com/article.cfm?article=17#Q6http://www.osronline.com/article.cfm?article=17#Q5http://www.osronline.com/article.cfm?article=17#Q4http://www.osronline.com/article.cfm?article=17#Q3http://www.osronline.com/article.cfm?article=17#Q2http://www.osronline.com/article.cfm?article=17#Q1http://www.osronline.com/section.cfm?section=20http://www.osronline.com/article.cfm?id=610http://www.osronline.com/article.cfm?id=611http://www.osronline.com/article.cfm?id=612http://www.osronline.com/article.cfm?id=583http://www.osronline.com/article.cfm?id=613https://www.osronline.com/custom.cfm?name=index_fullframeset.cfm&pageURL=https://www.osronline.com/store/index.cfmhttp://www.osronline.com/section.cfm?section=31http://www.osronline.com/page.cfm?name=ListServerhttp://www.osronline.com/section.cfm?section=27http://www.osronline.com/section.cfm?section=20http://www.osronline.com/section.cfm?section=26http://www.osronline.com/section.cfm?section=17http://www.osronline.com/section.cfm?section=16http://www.osronline.com/section.cfm?section=45http://www.osronline.com/page.cfm?name=OSR%20Online?http://www.osronline.com/custom.cfm?name=login_joinok.cfmhttp://www.osronline.com/login.cfmhttp://www.osronline.com/adClick.cfm?id=73http://www.osronline.com/index.cfmhttp://www.osronline.com/index.cfmhttp://www.osronline.com/index.cfmhttp://www.osronline.com/index.cfmhttp://www.osr.com/consulting.htmlhttp://www.osr.com/toolkits.htmlhttp://msdn.microsoft.com/en-us/windows/hardware/hh852365http://msdn.microsoft.com/en-us/library/windows/hardware/ff557573(v=vs.85).aspxhttp://msdn.microsoft.com/en-us/windows/hardware/gg463028http://www.osronline.com/article.cfm?id=259http://www.microsoft.com/whdc/devtools/debugging/default.mspxhttp://www.osr.com/fsd.htmlhttp://www.osr.com/wdf.htmlhttp://www.osr.com/swdrivers.htmlhttp://www.osr.com/debug.htmlhttp://www.osronline.com/login.cfmhttp://www.osronline.com/index.cfm


2/45



System hangs/crashes?

We've got a specialdiagnostic team that'sstanding by .

Visit the OSR CorporateWeb site for moreinformation about howOSR can help!

Q20 Does NT/2K/XP have anything like a VFS interface?

Q21 Are the sources for NTFS available for general reference?

Q22 Can any part of a file system or a file system filter driver be pageable?

Q23 How does the defragmentation API work? Is there anything special that afilter driver must do to handle it correctly?

Q24 How are file IDs and Object IDs used in the file systems? In my filter driver, how do I deal with them?

Q25 What is the difference between cached I/O, user non-cached I/O, andpaging I/O?

Q26 How do I map devices back to drive letters in Windows NT/2000/XP? Can Iuse the mount manager to do this? If so, how do I do this?

Q27 An application opened the file using the short name. How do I retrieve thelong name?

Q28 I see I/O requests with the IRP_MN_MDL minor function code? What doesthis mean? How should I handle it in my file system or filter driver?

Q29 Who is responsible for maintaining the 'file pointer' (CurrentByteOffsetfield)? When an application does an append to the end of the file, how is thispresented to the file system?

Q30 I never see any calls to several fast I/O operations. Does this mean I don'tneed to filter them? What happens if I do need to filter them?

Q31 Issues calling an FSD from a completion routine

Q32 Obtaining a drive letter assignment

Q33 Handling FILE_COMPLETE_IF_OPLOCKED in a filter driver.

Q34 Opening files during IRP_MJ_CREATE processing

Q35 How do I retrieve the "user name" for the user performing a givenoperation?

Q36 How do I detect reentrancy back into my filter driver?

Q37 How do I call a user-mode function from my kernel driver?

Q38 What is structured exception handling? How should I use it? Why do I getSTOP code 0x1E (KMODE_EXCEPTION_NOT_HANDLED)? How do I deal withthis?

Q39 I am using a completion routine in my filter. What am I allowed to do?What am I NOT allowed to do? What alternatives do I have to performing thework in my completion routine?

Q40 How do I force files to be closed from my file system/filter driver?

41 Can I rely upon the RelatedFileObject field in the FileObject? How should Iuse this information?

Q42 How do I deal with the "recycle bin"? Is this some special directory in thefile system?
http://www.osronline.com/article.cfm?article=17#Q42http://www.osronline.com/article.cfm?article=17#Q41http://www.osronline.com/article.cfm?article=17#Q40http://www.osronline.com/article.cfm?article=17#Q39http://www.osronline.com/article.cfm?article=17#Q38http://www.osronline.com/article.cfm?article=17#Q37http://www.osronline.com/article.cfm?article=17#Q36http://www.osronline.com/article.cfm?article=17#Q35http://www.osronline.com/article.cfm?article=17#Q34http://www.osronline.com/article.cfm?article=17#Q33http://www.osronline.com/article.cfm?article=17#Q32http://www.osronline.com/article.cfm?article=17#Q31http://www.osronline.com/article.cfm?article=17#Q30http://www.osronline.com/article.cfm?article=17#Q29http://www.osronline.com/article.cfm?article=17#Q28http://www.osronline.com/article.cfm?article=17#Q27http://www.osronline.com/article.cfm?article=17#Q26http://www.osronline.com/article.cfm?article=17#Q25http://www.osronline.com/article.cfm?article=17#Q24http://www.osronline.com/article.cfm?article=17#Q23http://www.osronline.com/article.cfm?article=17#Q22http://www.osronline.com/article.cfm?article=17#Q21http://www.osronline.com/article.cfm?article=17#Q20http://www.osr.com/http://www.osr.com/analyze.htmlhttp://www.osr.com/consulting.html


3/45



Q43 I need to access the file, but it is locked for exclusive access. How do Iget around this?

Q44 I need to read a range of the file but it has a byte range lock on it. Howcan I bypass these byte range locks?

Q45 I need to build my own IRP. How do I do this?

Q46 How do timestamps work on files? What is the "change time" versus the"modify time"?

Q47 How does dismount work? How does this differ from media removal?Device removal?

Q48 What happens if I mix memory mapped I/O with regular file I/O? What if the file I/O is non-cached?

Q49 I am getting a PFN_LIST_CORRUPT STOP code. What does this mean?What could I be doing wrong? How do I work around this problem?

Q50 What are the rules for my file system/filter driver for handling paging I/O?What about paging file I/O?

Q51 I am getting NO_MORE_IRP_STACK_LOCATIONS as a stop code. Howdo I fix this?

Q52 What are the obsolete calls in Windows 2000? In Windows XP?

Q53 How do I enumerate the contents of a directory from kernel mode?

Q54 I am building a filter driver where I must change the directory information.How do I do that?

Q55 I see the user close the file. My filter receives an IRP_MJ_CLEANUP. ButI never see the IRP_MJ_CLOSE? Why not?

Q56 What are the rules for managing MDLs and User Buffers? How do Isubstitute my own buffer in an IRP?

Q57 What are the issues with respect to IRQL APC_LEVEL? What does it do?Why should I use (or not use) FsRtlEnterFileSystem?

Q58 How do I determine if the FILE_OBJECT represents a file or a directoryfrom my filter driver? Can I rely upon the FILE_DIRECTORY_FILE bit?

Q59 How do I determine if the IRP is coming from a local process or over thenetwork?

Q60 How should I deal with Fast I/O in my file system? In my filter driver?

Q61 I am suffering from stack overflow issues. How do I deal with this?

Q62 What is the difference between EOF and AllocationSize? Why is the AllocationSize the same for a file AFTER it is compressed?

Q63 What is the IFS Kit? How do I get it? I'm not in the US/Canada. Can I stillbuy it? Can I buy it from a retail distributor? With a purchase order?

Q64 I open a file but later when I try to use the handle I get back an error indicating an invalid handle (or invalid object type) error. What am I doingwrong? How can I use my handle?

Q65 When can I rely upon the file name in the FileObject structure?
http://www.osronline.com/article.cfm?article=17#Q65http://www.osronline.com/article.cfm?article=17#Q64http://www.osronline.com/article.cfm?article=17#Q63http://www.osronline.com/article.cfm?article=17#Q62http://www.osronline.com/article.cfm?article=17#Q61http://www.osronline.com/article.cfm?article=17#Q60http://www.osronline.com/article.cfm?article=17#Q59http://www.osronline.com/article.cfm?article=17#Q58http://www.osronline.com/article.cfm?article=17#Q57http://www.osronline.com/article.cfm?article=17#Q56http://www.osronline.com/article.cfm?article=17#Q55http://www.osronline.com/article.cfm?article=17#Q54http://www.osronline.com/article.cfm?article=17#Q53http://www.osronline.com/article.cfm?article=17#Q52http://www.osronline.com/article.cfm?article=17#Q51http://www.osronline.com/article.cfm?article=17#Q50http://www.osronline.com/article.cfm?article=17#Q49http://www.osronline.com/article.cfm?article=17#Q48http://www.osronline.com/article.cfm?article=17#Q47http://www.osronline.com/article.cfm?article=17#Q46http://www.osronline.com/article.cfm?article=17#Q45http://www.osronline.com/article.cfm?article=17#Q44http://www.osronline.com/article.cfm?article=17#Q43


4/45



Q1 How difficult is it to port a Windows 9x based file system or filesystem filter driver to Windows NT/2000/XP?

In general one does not "port" a Windows 9x (Windows 95, Windows 98, andWindows Me) based file system to Windows NT, Windows 2000, or WindowsXP. This is because the two file system models are quite a bit different. Thus,in general, "porting" consists more of re-implementing the file system.

This is particularly true for "filter drivers" where Windows 9x provides the IFSManager "hook" mechanism and Windows NT uses the file system filter driver model. The two models are incompatible with one another.

It is possible for a file system to be written in an "OS independent" fashion, butin general such undertakings are substantial projects and are not normallyconsidered "ports" of a file system (or filter driver) from one OS to another OS.

There are numerous differences between a Windows NT and Windows 98 filesystem that make porting a difficult task. They include:

(a) A substantially different I/O Model for file sys tem operations. Windows 9xuses the "IFS Manager" interface. Windows NT uses the I/O Manager (IRP-based) interface.

(b) Windows NT is a re-entrant operating system, while Windows 98 is not.Thus, calls can re-enter the storage stack in Windows NT. This introducesmore complex locking and synchronization semantics than are present inWindows 98.

(c) Windows NT makes considerable use of the demand paged virtual memorysystem with respect to file systems. Windows 9x file systems do not makeuse of the virtual memory system - such use is entirely different!

Of course, this list isn't intended on being comprehensive, but rather todemonstrate some of the "big" differences that developers find when portingfrom one OS to the other.

Q2 Is there a WDM model for file systems or file system filter drivers?

The Windows Driver Model (WDM) does not have a model for file systems or file system filter drivers. Thus, it is not possible to construct a file system or filesystem filter driver that is cross-platform compatible using the standardWindows Driver Model mechanism. This is because the model for file systemsis substantially different between the two operating systems.

Q3 How does a file system or file system filter driver handle PnP, Power Management, and WMI in Windows 2000/XP?

Plug and Play, Power Management, and Windows ManagementInstrumentation (WMI) are normally handled in a minimal fashion in theWindows 2000/XP file systems. Of course, this does not mean that filesystems are prohibited from handling such differences, only that as a matter of course most file systems do not need to be aware of all the PnP, Power Management, and WMI operations available.

For physical file systems, and file system filter drivers for those file systems,the most significant PnP operations are those that handle removable devices(which is not to be confused with removable media). A removable device is adevice that can be dynamically removed from the system, such as a storagedevice connected via the USB or bus. In this case, the file system follows arather specific pattern of behavior (which is not the same as the behavior


5/45



described in the DDK for the physical storage drivers). Thus, when an FSDreceives a plug and play request inquiring as to the state of the device for removal (IRP_MJ_PNP, with IRP_MN_QUERY_REMOVE) it might ensure thatno critical files are located on the device (paging files, registry hive, etc.) and if not it would then pass the request through to the underlying media device.When the device is removed (IRP_MJ_PNP, with IRP_MN_REMOVE_DEVICE)the file system waits to ensure that the underlying media has beensuccessfully removed. Once that's done, it then logically "dismounts" thevolume (this is internal to the file system) and cleans up its own internal state.

After all, the volume that was contained within the removable device is gone.

Most other plug and play operations would not normally apply to a physicalmedia file system. This can be seen in the FAT file system code distributed aspart of the IFS Kit.

A network file system may also be interested in the appearance or removal of specific protocol stacks, and thus may become involved in the monitoring of plug and play events. However, the physical file systems are tied directly to themedia device via the volume parameter block, but the network file systems arenot directly tied to the underlying protocol stacks and thus they cannot relyupon the same IRP passing mechanism. Instead, a network file system wouldnormally utilize a user-mode service to monitor state changes in the protocolsby registering for notification of such events. Then, when a protocol has beenloaded (or is being unloaded) the user-mode service can indicate that statechange to the underlying driver using its private interface into the driver.

Power management is not supported by any of the existing physical media filesystems because there is no need to do so. Prior to powering down thesystem, the OS ensures that the file systems are called to flush any dirty databack to their media (which is, of course, the primary concern of most filesystems). It is possible to support power management within a file system, butat the present time there are no examples of this in the IFS Kit.

Windows Management Instrumentation (WMI) for file systems is the same as itis for normal device drivers. The examples in the IFS Kit do not use the WMImechanism for control, configuration or statistics gathering. Instead, they relyupon the Windows NT 4.0 mechanism (IOCTL calls) for retrieving suchinformation. Thus, there are no standard information formats supported by WMIfor file systems.

Q4 How do file systems get loaded on Windows NT/2000/XP?

File systems are loaded via the I/O Manager on Windows NT, Windows 2000and Windows XP, based upon information stored in the registry, or by explicitrequests to load the driver via the ZwLoadDriver API (which is documented inthe Windows XP IFS Kit).

While it is possible to build a Windows 2000-style file system driver (whichutilizes an AddDevice entry point) none of the existing Windows 2000 filesystem function in this manner. Thus, they are neither installed using INF

scripts, nor does the plug and play manager start them.Q5 How is the file system's device object found?

Each time a user performs an open operation (via ZwCreateFile) they musteither specify an absolute path name or they must specify a path name relativeto an existing open file or directory.

In the case of a relative open, the I/O Manager "knows" to which device itshould send the requests, because the existing open file or directory alreadyreferences the correct file sys tem device instance. Thus, the request can besent directly to that device.


6/45



In the case of an absolute open, the I/O Manager must first start by parsing thename via the Object Manager. The Object Manager resolves a portion of thename that leads to a device object and then passes the balance of the name(the portion that has not yet been resolved) back to the I/O Manager, along witha pointer to the device object it located during its portion of name resolution.

The I/O Manager then examines the device object. If the device object has avalid pointer to a volume parameter block (VPB), the I/O Manager uses theDeviceObject field of the VPB to find the correct file system device object. If thedevice object does not have a valid VPB pointer, then the I/O Manager sends

the IRP_MJ_CREATE request directly to the device driver for the device objectpassed in by the object manager.

For example, a physical media file system volume is located via the VPB. If, for instance, an application program attempts to open some file, say"c:\fred\bob.txt", this name is translated into its Windows NT/2000/XPequivalent name (via the Win32 CreateFile API call's implementation)"\DosDevices\c:\fred\bob.txt". This name, when passed to the I/O Manager must then be passed off to the object manager in order to locate the correctphysical media device object. The object manager begins by attempting tolocate the "DosDevices" entry. This entry is a symbolic link to the ?? directory(which, in turn, is a special directory, whose contents can vary on a per-process basis). This causes the name to be reconstructed using the newname. Parsing that new name then begins again. Thus, this name becomes"\??\C:\fred\bob.txt". The parsing for ?? locates the correct directory (the onethat can vary on a per-process basis) and then in that directory it finds the "c:"entry. This is yet another symbolic link. The contents of this will vary from onecomputer to another, depending upon its configuration, but it would typically besome new name such as (this from a real Windows XP system)"\Device\HarddiskVolume1". Thus, the name being parsed has now beenreworked so that it is "\Device\HarddiskVolume1\fred\bob.txt" . Initiating a parsewith this name we find the Device directory and then the HarddiskVolume1device object within that directory.

This device object, along with the balance of the name, is then passed into theI/O Manager. The I/O Manager then examines the device object in question.Using the DeviceTree utility (available from OSR, and also scheduled for inclusion in the Windows XP DDK), we can see that this device object has aVPB structure that in turn points to the correct file system device instance.Thus, the name passed by the object manager to the FSD in this instance was"\fred\bob.txt" and the file system's device object found via the VPB will bepassed to its IRP_MJ_CREATE dispatch entry point.

Had we performed the same analysis using a different drive letter, such as the"I:" drive, which maps to a network share, we would have followed similar parsing logic until we found the device name, which in this case the name onour sample system was "\Device\LanmanRedirector\;I:0\server\c$". However,when the I/O Manager examines the device object (for "\Device\LanmanRedirector") it will not find a VPB. Thus, it will pass thebalance of the name to the IRP_MJ_CREATE handler of this device directly.

Note that the name passed in this case would be quite different:"\;I:0\server\c$\fred\bob.txt".

Q6 What's the right way to cancel a CREATE request in my filter driver?

A file system filter driver will sometimes need to pass an IRP_MJ_CREATErequest to the underlying file system and determine (in its completion routinefor IRP_MJ_CREATE) whether or not the operation should be successful. Theissue becomes the fact that the file object has now been declared to theunderlying FSD. If the filter driver fails the IRP_MJ_CREATE operation it willcause a memory leak because the FSD will not know the file object is nolonger in use.


7/45



For Windows 2000 and Windows XP a file system filter driver can use theIoCancelOpen API. For earlier versions of Windows NT, equivalent functionalityis achieved by:

- Setting the FO_HANDLE_CREATED field in the file object; and- Sending an IRP_MJ_CLEANUP to the underlying FSD; and- Sending an IRP_MJ_CLOSE operation to the underlying FSD.

These I/O operations should be performed synchronously. Once these have allbeen completed, the status of the original IRP must be set to an error value

and the I/O operation completed.Note: this technique does not imply that it will undo the side effects of IRP_MJ_CREATE .? To determine if an? IRP_MJ_CREATE will have side effectsyou must?examine?the Disposition field.? The FILE_SUPERSEDE,FILE_OPEN_IF, FILE_OVERWRITE and FILE_OVERWRITE_IF operations canhave definite side effects, some of which are extremely difficult to "undo" (e.g.,FILE_SUPERSEDE) .? If your filter needs to "undo" such operations, you mustfirst check to find the state of the underlying file sys tem before allowing theoperation to proceed.

Q7 How do I de al with file -sharing issues in my filter driver?

File sharing semantics are often an unforeseen complication for file systemfilter drivers. This is because filter drivers often wish to create separate handlesagainst the file so that they can perform operations against the file,independent of the application-initiated request to access the file. For example,an on-access virus scanner might wish to read an executable in order to scanit - this requires read access, while execution does not require read access.

Regardless of the reason, the "problem" that many file system filter drivershave is that when they attempt to open the file, they must be able to handlefile-sharing issues. Specifically, if another application opens the file, it mightspecify exclusive access to the file. It may be critical to the function of the filter driver that it be allowed to access the file even though normal file sharingsemantics would reject the access request. There are a few techniques that a

file system filter driver can use to "circumvent" file sharing semantics:

- The filter driver can implement the file sharing internally in the filter driver; inthis case it always asked the underlying FSD for full-sharing of the file andenforces the sharing at its level; or - The filter driver can use memory mapping to access the file; or - The filter driver can build IRPs and access the file directly;

While it is possible for a file system filter driver to implement the file sharingsemantics internally within the file system filter driver; it is often the mostdifficult situation because of the complex nature of the sharing semantics. Toproperly track sharing state, the file system must track (on a per-file basis) thecurrent sharing state of the file. Typically, this is done to mirror the behavior of the FAT file system example in the IFS Kit. Thus, when the file is opened, thesharing is managed by using IoCheckShareAccess and/or IoUpdateShareAccess. When the user closes the file (as specified by theIRP_MJ_CLEANUP operation) the filter removes the share access(IoRemoveShareAccess) so that subsequent operations to access the filesucceed.

In computing share access there are two important elements:

- The access requested for the specific open instance of the file, which isrecorded in the file object; and- The share access allowed by existing applications using the file.

The actual implementation of this can be somewhat complicated, which is why


8/45



it is generally best to allow the FSD to manage this, or to use the I/O Manager routines.

A second approach is to keep in mind that the section object for a file canalways be used to memory map the file.

A third approach is to build read and write IRPs directly. Because securitychecks (including whether or not a read/write is allowed) are done in the I/OManager before the operation is passed to the file system. Thus, the filter driver can perform I/O operations against a file object that was not opened for read or

write access, simply by constructing read and write IRPs within the filter driver itself.

Q8 What's the proper way to install my file system or file system filter driver?

Typically, a file system driver is a legacy driver. As such, it is installed byusing the Service Control Manager, or directly making changes to the registry.

A file system filter driver normally installs in a separate fashion, although thiswill change as the Windows XP team begins to add WHQL certificationmechanisms for certain types of file system filter drivers. For such drivers, theywill be required to utilize INF scripts.

In addition, it is possible to build a plug-and-play "type" file system. In such acase, it is possible to build an INF file for such a driver. However, there are noexisting samples of how to achieve this.

Q9 Does WHQL logo file systems or file system filter drivers?

Not at the present time. However, WHQL is working on a plan for the future thatwill provide testing and certification for certain types of file system filter drivers.The first target file system filter drivers for certification will be anti-virus filters.

Q10 When my customers install my file system or filter driver product,will they get that nasty pop-up saying "this is not a signeddriver...Microsoft recommends that you do not continue"??

Not at the present time. Once Microsoft begins to implement a protocol for signing file system filter drivers this pop-up would be displayed for any filesystem filter driver that registered using one of the defined class GUIDs butwas not signed.

This pop-up dialog box does not apply for devices that register using a GUID for which Microsoft does not have a signing program, which includes file systemsand file system filter drivers.

Currently, Microsoft is working on developing a signing program for the filter part of anti-virus products containing a file system filter driver. Once that

program is in-place, a driver developer will need to submit their driver toMicrosoft so that the results can be certified and the resulting driver signed byMicrosoft. In the future, Microsoft may extend this program to include other filesystem filter drivers as well, although it seems unlikely there will ever be a"generic" file system filter driver certification process, given the broad range of possible behaviors from file system filter drivers.

File systems do not presently have a program for certification. However, intheory, it should be possible to submit a file system for certification, providedthat it passes all of the relevant tests (that is, those tests used to ensureproper behavior of FAT and NTFS). We note that this is "in theory" becausesuch certification is not a standard program and would require acceptanceunder a case-by-case review by Microsoft.


9/45



Q11 How is cache coherency handled when a file is opened for "ordinary" (cached) I/O and also opened memory mapped?

The Windows NT/2000/XP file systems all utilize the virtual memory system for caching the file system data. The task of managing the file system data is theresponsibility of the cache manager. The cache manager accomplishes this bymemory mapping the files - which in turn is done by the Memory Manager.

An application (such as Notepad) that uses memory-mapped files does so by

memory mapping the files, again via the Memory Manager.What this means is that if an application program reads data from the filesystem (using the standard Win32 ReadFile API call) the file system satisfiesthis request by calling the Cache Manager (via CcCopyRead, for example) andthe Cache Manager then, in turn, copies the data from virtual memory.

The advantage of this approach is that there is only a single copy of the file -the one in virtual memory. Thus, both read/write and memory mapped accessto the file are coherent.

Q12 Can I leave integration with the cache manager out of my productto simplify things? What's the impact of doing this?

While it is possible to construct a file system that does not integrate with thecache manager, in general there are a number of potential problems of which adeveloper should be aware:

(a) Integration with the Memory Manager (versus the Cache Manager) isseldom optional. This is because any memory-mapped file access requiressupporting the Memory Manager. A file system that does not support theMemory Manager will not work with Notepad in Windows 2000, nor will programexecution function properly.

(b) A file system that does integrate with the Memory Manager but does notintegrate with the Cache Manager must be aware that it may not be possible to

maintain cache coherency between the Memory Manager and the file system.If a file is memory mapped by an application program, it is not possible for a filesystem to indicate that the cached (memory mapped) data has changed. Thus,the application program will continue to use the old information.

To implement a file system so that it does not "integrate" with the cachemanager, it is sufficient for the file system to refrain from calling any of the Cc*routines. However, to support memory mapped files (executables, notepad,etc.) it is necessary for the FSD to create the storage for theSectionObjectPointers structure (a field to which the FileObject points). TheMemory Manager will utilize this storage to track the section objects used aspart of supporting a file-backed section object.

An FSD then controls the contents of the memory manager state by usingMmFlushSection and MmForceSectionClosed in order to advise the memorymanager of changes to the section, or to force the memory manager to tear down references to the section. Note that both of these calls are optional and itis quite possible that neither of them will lead to the desired results - thus, theFSD must be written to handle cases where these calls return errors.

Q13 Must I support Fast I/O in my file system or filter, and where is FastI/O documented?

Fast I/O is optional for a file system. For a file system filter driver, it is requiredif any file system it filters supports fast I/O. Because all of the standardWindows NT/2000/XP file systems support fast I/O, essentially it is required of


10/45



all filter drivers.

Fast I/O is a mechanism that was introduced into Windows NT in order tooptimize the handling of certain I/O operations. In addition, fast I/O routines arealso used for a variety of communications channels between the FSD andvarious kernel components, including the I/O Manager, the File SystemRuntime Library, and the CIFS File Server. Unfortunately, at the present timeFast I/O is not documented, although future versions of the IFS Kit will includedocumentation on how the Fast I/O routines are used.

Q14 Filtering file systems doesn't look that hard... I have a free sample Idownloaded off the web. What are the limitations of the currentlyavailable file system filter driver samples?

The examples we have seen often serve as a good "starting point" for a filesystem filter driver project. Unfortunately, most of these examples are usually"passive" filters - that is, they examine an I/O operation but do not attempt tomodify the I/O or the flow of control to the underlying FSD.

Thus, the more closely the samples fit your intended file system filter driver,the more applicable they will be. If, however, your primary goal is to modify thebehavior of the underlying file systems, such as might be the case with an on-access virus scanner or an on-the-fly encryption program, then these samplesserve as little more than that - a sample of how to implement a file system filter driver.

Q15 Is Rajeev Nagar's book accurate, and should I use it as areference?

Rajeev Nagar's book can serve as a good reference text for background, as wellas for a discussion of many of the issues involved in writing a file systemand/or a file system filter driver for Windows NT. The most significant limitationto the book is that much of it was done against Windows NT 3.51, although itwas released just after Windows NT 4.0 became available. Thus, some of thedetailed information is no longer applicable. However, since it is the only text

available on this subject, i t stil l plays an important role in the reference shelf of a file systems developer.

Q16 Are there any other books on NT/2K/XP file system or file systemfilter development? What other resources are available to help me??

Obviously, one important source of information for those developing WindowsNT/2000/XP file systems is the NTFSD mailing list (for which you can s ign upat the OSR Website . List members may submit questions [email protected].) In addition, there are books on Windows NT devicedrivers that can provide much of the background for drivers in general:

? Viscarola, Peter G. and W. Anthony Mason, Windows NT Device Driver

Development, Macmillan Technical Publishing, 1998.

? Dekker, Edward N and Joseph M. Newcomer, Developing Windows NTDevice Drivers: A Programmer's Handbook, Addison-Wesley PublishingCompany, 1999.

? Baker, Art, The Windows NT Device Driver Book: A Guide for Programmers,Prentice Hall, 1996.

? Cant, Chris, Writing Windows WDM Device Drivers: Covers NT 4, Win 98,and Win 2000, CMP Books, 1999.

? Oney, Walter and Forrest Foltz, Programming the Microsoft Windows Driver
http://www.osr.com/


11/45



Model, Microsoft Press, 1999.

? McDowell, Steven, Windows 2000 Kernel Debugging, Prentice Hall Computer Books, 2001.

? Solomon, David A and Mark Russinovich, Inside Microsoft Windows 2000(Third Edition), September 2000.

? Nebbett, Gary, Windows NT/2000 Native API Reference, New RidersPublishing, 2000.

? Schreiber, Sven, Undocumented Windows 2000 Secrets, Addison-Wesley,2001.

There are other resources as well, including the Microsoft IFS Kit and OSR'sThe NT Insider .

Q17 What are the primary differences between filtering rdr and filteringa local file system?

In general, filtering a local file system differs somewhat from filtering aredirector in a number of key areas:

(a) File state tracking. Network file sy stems frequently defer operations in order to minimize the network traffic. For example, CIFS defers opening files untilnecessary. This can lead to changing the "FsContext" value of the file object,for instance, which complicates the tracking for file system filter drivers.

(c) Security. Network file systems routinely restrict operations that can beperformed by the system process using standard system credentials. Thus filesystem filter drivers may need to utilize advanced techniques, such asimpersonation, etc. in order to circumvent these security considerations.

(d) Private APIs between user mode service and kernel mode driver. Some of the processing needed by the network redirector is implemented in a user mode service. The user mode service and kernel mode driver interact using a

private IOCTL-based API.

(e) Support for UNC naming. The Universal Naming Convention (UNC) consistsof names that begin with a prefix (e.g., "server" and "share" in CIFS) and thenthe balance of the name. A filter must be able to handle both UNC names and"volume based" names.

(f) Complex naming issues. Names are not unique for redirectors. For example,the volume based name and the UNC name are quite different, yet they caneach be used to reach the same file.

(g) Some operations do not translate to redirectors. For example, "open by ID"works with CIFS, but the file ID cannot be obtained directly from all versions of

the CIFS redirector - instead, a "dummy value" is returned by the CIFSredirector.

Many operations are identical between the two, but there are subtle behavior differences that can dramatically impact the development of a file system filter driver - beware!

Q18 In general, when can I use the IFS kit's mini-rdr model, and when isit best for me to write my own rdr from scratch?

The IFS Kit's mini redirector model is most suitable for CIFS-like network filesystem clients, although it can be adapted to a broad range of file systems,including file systems that are not "redirectors" in the traditional sense. The
http://www.osr.com/http://www.microsoft.com/ddk/ifskit


12/45



primary advantage of the mini-redirector model is that it simplifies theinteractions with the virtual memory system.

Writing any file system, including a redirector, "from scratch" is a substantialundertaking and will typically involve many months of diligent work. It is mostappropriate when the file system being developed does not fit into a standardmodel (such as the mini-redirector model) or when the file system requires theability to control the file system/virtual memory interactions.

Q19 Does Microsoft offer support for developing file systems or file

system filters?

Microsoft's support offerings for file systems currently include making the IFSKit available. Newer versions of the IFS Kit include documentation on how touse the kit. At the present time (October, 2001) Microsoft does not offer development support for developing file systems or file system filter drivers,although that may be subject to change.

Q20 Does NT/2K/XP have anything like a VFS interface?

The Virtual File System (VFS) interface is a UNIX abstraction originallydeveloped in order to simplify the porting of the Network File System (NFS) to

various UNIX platforms. It provides a model where there is a "standard" set of operations that are supported by a file system. The operating sys tem then, inturn, works with the file system to maintain a "vnode" cache (where a "vnode"is a virtual file system information element) and to provide core OSmanagement services for working with the OS.

The Windows NT/2000/XP platform also has a "standard" set of operations thatit expects to be supported by file systems. The operating system then invokesthe relevant file system services by calling upon those standard services.

However, the details of the implementation of the UNIX VFS interface are quitedifferent than the IRP Major dispatch entry points a Windows NT file system.Because of this, a UNIX VFS-based file system does not "port" in any trivial

fashion to Windows NT/2000/XP.Q21 Are the sources for NTFS available for general reference?

The Installable File Systems (IFS) Kit contains a number of real file systemsthat are distributed with Windows NT/2000/XP. This includes the FAT filesystem and the CDFS file system. However, Microsoft does not distribute theNTFS file system as part of the IFS Kit. While it may be possible to licenseother components of Windows NT/2000/XP from Microsoft, this is notsomething that is available via the IFS Kit mechanism.

Q22 Can any part of a file system or a file system filter driver bepageable?

Depending upon the type of file system, some, or all of the file system can bepageable. In addition, there are some special issues for physical media filesystems that support paging files (these are special files used by the OS tostore pageable data when it is not in use).

First, it is worth noting the underling issue that motivates these rules aboutpaging - specifically, about the ability of the Windows operating system tohandle an arbitrary number of stacked page faults.? The stack in the kernel islimited - typically 12KB - and any reentrant behavior, including page faults, cancause stack overflow conditions. Thus, in general the rule is that at most twopage faults may be "stacked".? The second page fault will occur whenattempting to resolve the first page fault.? That second fault is actually subject
http://www.microsoft.com/ddk/ifskit


13/45



to some pretty stringent rules.

For any arbitrary code within a file system, or file system filter driver, that canbe invoked while handling paging activity (that is any I/O operation where theIRP_PAGING_IO bit is present in the IRP) the file system must ensure that itsown code necessary for handling that I/O can always be fetched safely. For example, the NTFS and FAT file systems must ensure their code is locked inmemory because if it were paged in from the disk it might be paged in from anNTFS or FAT volume. However, since the code necessary to handle thatpaging I/O would not be present, there would be no way for the page fault to be

resolved. This, of course, applies to file system filter drivers that filter NTFS or FAT, as well. If you are writing a file system, and your file system code is never stored on your file system, then you can make your code paths pageable.

For all file systems (even NTFS or FAT) you can make data accessed whilehandling paging activity pageable. Thus, you can store file system data inpageable memory, as well as use file-backed sections (which are "demandpaged"). Even if the file system is processing a page fault, a second page faultcan be handled. In that case, however, processing that second page fault mustnot generate yet another page fault because that would then constitute thedreaded "three faults". Thus, there is SOME data within the file system thatmust be in non-paged memory. Typically, this is any data required to processthe paging activity to satisfy an IRP_MJ_READ operation, since that is requiredfor retrieving data from a file.

For the paging file, there are special rules because access to the paging filecannot safely perform any operation that will generate a page fault. This isbecause the memory manager may be retrieving the contents of paged poolfrom the file system. Since we have already described the paths where the filesystem may be handling a stacked page fault any new page fault wouldconstitute a triple-fault situation, which is catastrophic. Thus, the rules for paging file access are very restrictive. No access to pageable code or data.Use only of OS routines that are safe at DISPATCH_LEVEL (even though theOS is only running at APC_LEVEL when processing these page faults). TheMemory manager controls all serialization for paging operations (to eliminatepotential deadlock conditions).

Because of these restrictions, the Memory Manager makes certain guaranteesto file systems. Perhaps the most important is that it does not perform"extending write" operations. Thus, the portion of a file system that handlesspace allocation need not be resident in memory.

Note that the rules for paging files do not apply to file systems that do notsupport paging files. A file system can recognize an attempt to open or createa paging file because the SL_OPEN_PAGING_FILE bit will be indicated in theFlags field of the I/O stack location for the IRP_MJ_CREATE operation.

In Windows XP and Windows Server 2003 the routine FsRtlIsPagingFile maybe used to determine if a given file object represents the paging file.

Note that a file system filter driver that filters any file system supporting pagingfiles must obey the same rules.

Q23 How doe s the de fragmentation API work? Is there a nything specialthat a filter driver must do to handle it correctly?

The defragmentation API is a mechanism that was introduced in Windows NT4.0 to allow application programs to defragment files on the local disks.Perhaps the best source of information about how defragmentation works inWindows 2000 is the Platform SDK, which describes the use of FSCTL_GET_VOLUME_BITMAP, FSCTL_GET_RETRIEVAL_POINTERS, andFSCTL_MOVE_FILE. These three operations are used to defragment a givenfile. In addition, the IFS Kit includes the FAT file system source code and its


14/45


15/45



????????????return STATUS_INVALID_PARAMETER;????????}????}

And thus this will not work if the file ID is 128 bits rather than 64 bits. Only theNTFS file system supports opening a file using both its file ID and its object ID.

The most substantial impact of the use of file IDs for file system filter drivers isthe inability to extract a name for the file. For CDFS, a filter can query the filesystem for the name of the file (using IRP_MJ_QUERY_INFORMATION or

IoQueryInformationFile and querying the FileNameInformation attribute of thefile) and the file system will always return a name. However, for NTFS there arecertain cases when it cannot return a path to the file (where the caller thatopened the file does not have traverse privileges, there is no mechanism for NTFS to determine if it is allowed to return the path to the file). Further, even inthose cases where it does return a path to the file it is important to note thatthere may be multiple paths to the file (via hard links) and that the path namereturned is only one of the possible names.

Q25 What is the difference between cached I/O, user non-cached I/O,and paging I/O?

In a file system or file system filter driver, read and write operations fall intoseveral different categories. For the purpose of discussing them, we normallyconsider the following types:

- Cached I/O. This includes normal user I/O, both via the Fast I/O path as wellas via the IRP_MJ_READ and IRP_MJ_WRITE path. It also includes the MDLoperations (where the caller requests the FSD return an MDL pointing to thedata in the cache).

- Non-cached user I/O. This includes all non-cached I/O operations thatoriginate outside the virtual memory system.

- Paging I/O. These are I/O operations initiated by the virtual memory system inorder to satisfy the needs of the demand paging system.

Cached I/O is any I/O that can be satisfied by the file system data cache. Insuch a case, the operation is normally to copy the data from the virtual cachebuffer into the user buffer. If the virtual cache buffer contents are resident inmemory, the copy is fast and the results returned to the application quickly. If the virtual cache buffer contents are not all resident in memory, then the copyprocess will trigger a page fault, which generates a second re-entrant I/Ooperation via the paging mechanism.

Non-cached user I/O is I/O that must bypass the cache - even if the data ispresent in the cache. For read operations, the FSD can retrieve the datadirectly from the storage device without making any changes to the cache. For write operations, however, an FSD must ensure that the cached data is

properly invalidated (if this is even possible, which it will not be if the file is alsomemory mapped).

Paging I/O is I/O that must be satisfied from the storage device (whether localto the system or located on some "other" computer system) and it is beingrequested by the virtual memory system as part of the paging mechanism (andhence has special rules that apply to its behavior as well as its serialization).

Q26 How do I map devices back to drive letters in WindowsNT/2000/XP? Can I use the mount manager to do this? If so, how do I dothis?

In Windows NT there is no mechanism for mapping from a device back to a


16/45



drive letter. Further, because the mapping of driver letters to devices is not one-to-one and it can change, any mechanism for mapping drive letters must beable to handle the various issues involved. The simplest mechanism for physical media volumes is to open each possible drive letter ("\DosDevices\A:",then "\DosDevices\B:", etc.) and determine if that drive letter yields the proper device object. For a network drive, however, the drive letter is embedded withinthe name, and the FSD can extract the drive letter from that name; the namemust also be adjusted to account for the server/share name that is typicallypart of the embedded name.

In Windows 2000 and Windows XP, volume mount points (the generalization of drive letters on Windows 2000 and Windows XP) are managed by the mountmanager, which is a kernel mode driver that is responsible for managing volumemount points. The IOCTL operations to directly access the mount manager areincluded in the DDK (in the mountmgr.h header file) but the meanings of thesevolume mount point operations are not directly documented in the DDK.

The volume mount point API is documented in the Platform SDK in great detail,including various calls for creating, deleting, and locating volume mount points.This API in turn relies upon the mount manager IOCTL operations to retrievethe current list of mount points within the system. Again, we note that thisincludes drive letters, but also accommodates other volume mount points (andwe note that drive letters are not required as mount points!).

Q27 An application opened the file using the short name. How do Iretrieve the long name?

The one mechanism that works on all versions of Windows NT, 2000, and XP isto look into the directory that contains the entry. In Windows XP, this can bedone using NtQueryDirectoryFile, which is part of the Windows XP IFS Kit.However, in earlier versions this call is not exposed by the IFS Kit and thus itrequires that a filter driver implement equivalent functionality by using theIRP_MN_QUERY_DIRECTORY operations.

Using NtQueryDirectoryFile, the filter driver can specify a search of thedirectory using the short file name. The FSD will then return one (or more) file

that has a matching short file name as well as its corresponding long file name.For the standard file systems, this would only return a single matching entry.From this, both the short name and long name of the file can be retrieved.

Q28 I see I/O requests with the IRP_MN_MDL minor function code? Whatdoes this mean? How should I ha ndle it in my file system or filter driver?

Kernel mode callers of the read and write interface (IRP_MJ_READ andIRP_MJ_WRITE) can utilize an interface that allows retrieval of a pointer to thedata as it is located in the file system data cache. This allows the kernel modecaller to retrieve the data for the file without an additional data copy.

For example, the AFD file system driver has an API function it exports thattakes a socket handle and a file handle. The file contents are "copied" directlyto the corresponding communications socket. The AFD driver accomplishesthis task by sending an IRP_MJ_READ with the IRP_MN_MDL minor operation. The FSD then retrieves an MDL describing the cached data (at Irp->MdlAddress) and completes the request. When AFD has completedprocessing the operation it must return the MDL to the FSD by sending anIRP_MJ_READ with the IRP_MN_MDL_COMPLETE minor operation specified.

For a file system filter driver, the write operation may be a bit more confusing.When a caller specifies IRP_MJ_WRITE/IRP_MN_MDL the returned MDL maypoint to uninitialized data regions within the cache. That is because the cachemanager will refrain from reading the current data in from disk (unless


17/45



necessary) in anticipation of the caller replacing the data. When the caller hasupdated the data, it releases the buffer by callingIRP_MJ_WRITE/IRP_MN_MDL_COMPLETE. At that point the data has beenwritten back to the cache.

An FSD that is integrated with the cache manager can implement these minor functions by calling CcMdlRead and CcPrepareMdlWrite. The correspondingfunctions for completing these are CcMdlReadComplete andCcMdlWriteComplete. An FSD that is not integrated with the cache manager can either indicate these operations are not supported (in which case the caller

must send a buffer and support the "standard" read/write mechanism) or it canimplement them in some other manner that is appropriate.

Q29 Who is responsible for maintaining the 'file pointer'(CurrentByteOffset fie ld)? When an application does an append to theend of the file, how is this presented to the file system?

The "file pointer" is a field (the CurrentByteOffset field) within the file object thatis maintained by the FSD and used by the I/O Manager. Each time asynchronous I/O operation is performed, the file system updates this field toreflect the current byte offset. For example, here is code from CDFS thataccomplishes this task:

//// Update the current file position in the user file object.//

if (SynchronousIo && !PagingIo && NT_SUCCESS( Status )) {

????IrpSp->FileObject->CurrentByteOffset.QuadPart = ByteRange;}

Note that this field is not updated for asynchronous I/O or for paging I/O. TheSynchronousIo value used here was computed based upon the bits of the fileobject:

SynchronousIo = FlagOn( IrpSp->FileObject->Flags, FO_SYNCHRONOUS_IO);

The I/O Manager then uses this CurrentByteOffset field when specifying theoffset to use for a read or write operation.

In addition, an application can modify the CurrentByteOffset field by making anappropriate IRP_MJ_SET_INFORMATION call (setting theFilePositionInformation for a given open file instance). This is also handled bythe FSD.

An application can append to the end of a file by utilizing the manifest constantFILE_WRITE_TO_END_OF_FILE. This 64 bit numeric value indicates that the

data is being appended to the file. An FSD should perform the I/O operationrelative to the current end of the file. Thus, two applications can interleave I/Oand it will always be appended to the end of the file.

Q30 I never see any cal ls to several fast I/O opera tions. Does this mea n Idon't need to filter them? What happens if I do need to filter them?

There are a number of fast I/O operations that are currently not used by theOS. For example, the compressed MDL operations have been reserved for future use. However, the risk a file system filter driver runs in not implementingthese is that a subsequent release or update (even a service pack) might beginusing these APIs. It is also possible that a third party product might utilizethese APIs as well. Thus, by not filtering them, it is possible the file system


18/45



filter driver will not handle these operations and as a result its function may becompromised.

If a file system filter driver does not implement a particular fast I/O routine thedefault behavior for the OS is to call the underlying file system directly. Thisensures that even if an old file system filter driver is used with a newer system,the behavior of the file sys tem will be correct, although the file system filter driver may not function properly.

There are six fast I/O routines that are not invoked in a file system filter driver,

even if they are implemented by the file system filter driver. They are:

? AcquireFileForNtCreateSection? ReleaseFileForNtCreateSection? AcquireForModWrite? ReleaseForModWrite? AcquireForCcFlush? ReleaseForCcFlush

Each of these six fast I/O operations deals with serialization between the virtualmemory system and the file system. As such, most file system filter drivers donot need to filter these operations in any case. In Windows XP, a file systemfilter driver can register to filter these six fast I/O operations by utilizing theFsRtlRegisterFileSystemFilterCallbacks API (note that this API is not availableprior to Windows XP). In this case, the filter registers a set of twelve functionsthat will be called before, and after, the FSD's fast I/O entry point is called.

Q31 Issues calling an FSD from a completion routine

When calling back into a file system from a completion routine it is possiblethat per-thread state information used by the file systems can cause problemsfor the new operation.

Frequently, a file system filter driver attempts to perform operations in itscompletion routine that may involve calling down to the underlying file system.However, this can cause problems because the underlying file system utilizes

thread-local storage for its own operations. That thread local storage mayindicate incorrect state to the underlying file system.

Logically, this is because a file system may perform a series of operationswhen completing an I/O operation, such as the following example:

?????IoCompleteRequest(Irp, IO_NO_INCREMENT);?????IoSetTopLevelIrp(NULL);?????return STATUS_SUCCESS;

The "problem" here is that the call to IoCompleteRequest will, in turn, generatea call to your filter driver's completion routine. Of course, if your completionroutine then calls the file system again, it will be using the same "top level IRP"

information. The FSD may then (in turn) use this information, believing it isrelated to the operation you are performing. Unfortunately, this can lead toerroneous behavior of the underlying file system.

Further, while the Windows file systems have systematically been changed tofree their state before calling IoCompleteRequest (precisely because of thisproblem) this issue arises when you implement "layered" or s tacked filesystem functionality. In such a case, your driver might receive a request fromone file system and then redirect it to another file system. To implement thisfunctionality, your driver must save the upper file system's state before callinginto the lower file system. This would be done using the following mechanism:

?????PVOID oldTopLevelIrp;?????oldTopLevelIrp = IoGetTopLevelIrp();


19/45



?????IoSetTopLevelIrp(NULL);?????status = CallLowerFsd(Device, NewIrp);?????/* omit status handling */?????IoSetTopLevelIrp(oldTopLevelIrp);

This ensures that the state of the "upper file system" does not interfere with thestate of the "lower file system".

Q32 Obtaining a drive le tter a ssignment

A file system that is not associated with a physical media volume may wish tohave a drive letter assignment for its volume so that Win32 applications areable to access it.

The correct way to achieve this in Windows is to utilize the services of themount manager. There is a documented, exported API for achieving this -SetVolumeMountPoint.

Of course, this mechanism is implemented in terms of file system controloperations against the kernel level mount manager (\Device\MountMgr) and theIOCTL values are exported in the DDK header file mountmgr.h. The relevantIOCTL for creating a new mount point isIOCTL_MOUNTMGR_CREATE_POINT, which is documented in the DDK.

The mount manager is not responsible for creating NTFS mount points (whichare implemented using reparse points), although the SDK routine will useNTFS mount points as necessary. NTFS mount points can be created fromkernel level software by using the FSCTL_SET_REPARSE_POINT operation.For this call, you must describe the mount point information using theREPARSE_DATA_BUFFER structure. This will then store the relevant reparsepoint with the corresponding file or directory on the NTFS volume.

When creating mount points within a file system, keep in mind the restrictions:

? Of the Windows file systems, only NTFS supports reparse points;? NTFS requires that a directory be empty if you are going to create a mount

point for the directory;? A volume mounted using a reparse point need not have any drive letters.

Of course, the simplest solution is to create your mount points within the user mode environment with the SDK API, rather than within the kernel modeenvironment.

For those file systems that must function in a pre-Windows 2000 environment(prior to the existence of the mount manager API) either the driver, or anapplication program, must create the drive letter. A driver does so by creating asymbolic link using IoCreateSymbolicLink and a Win32 application does thisusing DefineDosDevice. Similarly, the drive letter may be removed by usingIoDeleteSymbolicLink (for the driver environment) or DefineDosDevice.

Q33 Handling FILE_COMPLETE_IF_OPLOCKED in a filter driver.

A filter driver may be called by an application that has indicated theFILE_COMPLETE_IF_OPLOCKED. If the filter in turn calls ZwCreateFile it maycause the thread to deadlock.

A common problem for file systems is that of reentrancy. The Windowsoperating system supports numerous reentrant operations. For example, anasynchronous procedure call (APC) can be invoked within a given threadcontext as needed. However, suppose an APC is delivered to a thread while itis processing a file system request. Imagine that this APC in turn issuesanother call into the file system. Recall that file systems utilize resource locks


20/45



internally to ensure correct operation. The file systems must also ensure thecorrect order of lock acquisition in order to eliminate the possibility of deadlocks arising. However it is not possible for the file system to define alocking order in the face of arbitrary reentrancy!

To resolve this problem, the file systems disable certain types of reentrancythat would not be safe. They do this by calling FsRtlEnterFileSystem whenthey enter a region of code that is not reentrant. When they leave that region of code, they call FsRtlExitFileSystem to enable reentrancy.

This is important to an understanding of this problem because the CIFS fileserver uses oplocks as part of its cache consistency mechanism betweenremote clients and local clients. This is done using a "callback" mechanism,which is implemented using APCs.

Normally, the FSD will block waiting for the completion of the APC that breaksan oplock. Under certain circumstances, however, the CIFS server thread thatissued the operation requiring an oplock break is also the thread that mustprocess the APC. Since the file system has blocked APC delivery, and nowthe thread is blocked awaiting completion of the APC, this approach leads todeadlock. Because of this, the Windows file system developers introduced anadditional option that advises the file system that if an oplock break is requiredto process the IRP_MJ_CREATE operation, it should not block, but insteadshould return a special status codeSTATUS_OPLOCK_BREAK_IN_PROGRESS. This return value then tells thecaller that the file is not completely opened. Instead, a subsequent call to thefile system, using the FSCTL_OPLOCK_BREAK_NOTIFY, must be made toensure that the oplock break has been completed.

Of course, this works because by returning this status code the APC can bedelivered, once the thread exits the file system driver.

Note that FSCTL_OPLOCK_BREAK_NOTIFY, and the other calls for theoplock protocol, are documented in the Windows Platform SDK.

Q34 Opening files during IRP_MJ_CREATE processing

While processing an IRP_MJ_CREATE a filter may need to open the file withdifferent attributes/rights, etc. This is often done by using a second call toZwCreatefile. This then will generate a call back into the FSD filter. Thus, acommon filter issue is being able to detect this reentrancy.

There are several ways of dealing with reentrancy during an IRP_MJ_CREATEoperation, and the appropriate solution for your particular driver will dependupon the circumstances. In addition, there are a number of techniques thatmight work for a single file system filter driver, but that fail when used in amulti-filter environment.

For Windows XP and newer versions of Windows, the best mechanism for

opening a file within the filter is to use IoCreateFileSpecifyDeviceObjectHint. Afilter driver can call this function and specify a given device object. TheIRP_MJ_CREATE that is built will be passed to the specified device object.This technique avoids reentrancy issues and is the best mechanism availablefor a filter to open a file.

For versions of Windows prior to Windows XP, this mechanism is not available.The best mechanism in this environment is to implement your own functionalequivalent of IoCreateFileSpecifyDeviceObjectHint. This can be done bycreating a second device object for each device you are filtering.

For example, suppose you decide to filter some given file system deviceobject, FSDVolumeDeviceObject. You then create a device objectMyFilterDeviceObject and attach it using IoAttachDeviceToDeviceStack (of


21/45



course, in Windows XP you would use IoAttachDeviceToDeviceStackSafeinstead). In addition, you create a second device objectMyFilterShadowDeviceObject. This device object must be assigned a name("\Device\MyFilterDevice27", for example). The name can be anything, but itmust obviously be unique. In your device extension for your two device objects,you need to track this name, and you need to maintain pointers to therespective device objects (that is, the device extension for MyFilterShadowDeviceObject should point to MyFilterDeviceObject and thedevice object extension for MyFilterDeviceObject should point toMyFilterShadowDeviceObject). Don't forget to set the StackSize field of the

device object correctly!)

Now, an IRP_MJ_CREATE request arrives in your filter, specifyingMyFilterDeviceObject. To open the file without experiencing reentrancyproblems, you call IoCreateFile (or ZwCreateFile). Since you must pass thename of the file being opened, you construct that by using both the name yougave MyFilterShadowDeviceObject and the name that is in the FileObject of theI/O stack Location (IoGetCurrentIrpStackLocation(Irp)->FileObject).

Since you are passing a name in that points to your second device object, theI/O Manager will build the IRP_MJ_CREATE and pass the resulting I/O requestpacket to your driver, but specifying MyFilterShadowDeviceObject.

In your IRP_MJ_CREATE dispatch handler you must detect that this is a"shadow" device object, rather than a typical filter device object. In this case,you should send the IRP_MJ_CREATE operation down to the device beingfiltered by MyFilterDeviceObject. Indeed, since you should not need to do anyfurther processing, you can use IoSkipCurrentIrpStackLocation (rather thanIoCopyCurrentIrpStackLocationToNext).

The original filter (where the IoCreateFile call was made) will receive back a filehandle that can then be used for subsequent operations (using the Zw APIroutines).

Typically, filter drivers that attempt to use IoCreateFile or ZwCreateFile with thesame file/device name as the original request experience reentrancy into their driver. A number of techniques for dealing with this have been tried in the past,but they exhibit various problems when combined with other filters. Theseinclude:

? Appending a "special string" to the end of the file name. This will not workwhen two filters using this technique are loaded onto the same system (sinceeach one appends its "special s tring" onto the previous filter's "special string").? Track thread identifiers to detect reentrancy. This technique fails whencombined with a filter that utilizes a separate service for opening the file; filterssometimes must switch to a different thread context in order to eliminate stackoverflow conditions.? Building create IRPs within a filter. This technique does work properly, but iscompletely unsupported and quite difficult to implement correctly . Because of this, it is a fragile solution that should not be used given the availability of

alternative mechanisms.? Re-using the file object from the current IRP_MJ_CREATE. In this sequence,the filter allows the create operation to complete and then uses the file objectsubsequently. When done, the filter then sends a cleanup and close operationto the underlying file system. It then sends the original IRP_MJ_CREATEoperation to the underlying FSD. There are several potential issues with thisapproach. First, in this technique the filter does not have a file handle for the fileobject and thus cannot use the Zw API calls. Second, the file object must berestored to its original state - otherwise, there are fields within it that are notset up properly. Third, because the file object has not yet been "properlyreferenced" the filter may find that the OS deletes the object because itsreference count drops to zero during its use. Used carefully, this technique hasbeen successful in the past.


22/45



Regardless of the technique used, the filter driver must be cognizant of theissues involving oplocks (and the FILE_COMPLETE_IF_OPLOCKED option). If an oplock break occurs during an IRP_MJ_CREATE, whether from an originalapplication, or from a filter driver attempting to open the file, the filter must beable to handle it gracefully.

Finally, the filter must be aware of the fact that even though it is calling fromkernel mode, the underlying FSD will continue to enforce sharing accessagainst the file.

Q35 How do I retrieve the "user name" for the user performing a givenoperation?

User names, per se, are not a concept of the core OS. Rather, users aretracked internally as "security identifiers" or SIDs. It is possible to extract theSID of the current thread. If a "user name" is needed, a user mode service canbe used to convert from the SID to the corresponding text user name. This isdone using the Win32 function LookupAccountSid, which is documented in thePlatform SDK.

The SID of the calling thread can be extracted from its token. This is done byfirst attempting to open the thread token (ZwOpenThreadTokenEx or NtOpenThreadToken or NtOpenThreadTokenEx). If this fails because the threadhas no token, the filter should open the process token(ZwOpenProcessTokenEx or NtOpenProcessToken or NtOpenProcessTokenEx). In either case, the filter will have a handle for atoken.

The SID can be retrieved from the given token using NtQueryInformationTokenor ZwQueryInformationToken. The filter should specify TokenUser as theTOKEN_INFORMATION_CLASS value. The call will return a buffer thatcontains the TOKEN_USER structure. This structure contains the SID of thetoken.

Note, however, that obtaining the SID of the current caller is often not precisely

what a filter is trying to accomplish. Instead, often the filter wishes to know theSID of the requesting thread. For local calls, this will typically be the same. For remote calls, however, the CIFS server routinely utilizes impersonation duringIRP_MJ_CREATE and for some IRP_MJ_SET_INFORMATION operations.Otherwise, the CIFS server uses the local system's credentials. To handle thiscase, a filter must store away the credential information of the original caller. Inthe case of IRP_MJ_CREATE the original caller's token is specified as part of the IO_SECURITY_CONTEXT parameter. The ACCESS_STATE structure inturn contains the SECURITY_SUBJECT_CONTEXT and the filter can retrieve apointer to the token using SeQuerySubjectContextToken. The SID can then beretrieved from the token using SeQueryInformationToken.

Q36 How do I de tect reentrancy back into my filter driver?

In addition to the standard problem of detecting re-entrancy during anIRP_MJ_CREATE operation (discussed in Question 34), some filter driversmust also be able to detect re-entrancy for other I/O operations, such asIRP_MJ_READ or IRP_MJ_WRITE. The technique described in the earlier section can also be applied for other I/O operations, but may not be necessary.

Certain operations tend to experience considerable re-entrancy. These include:

? I/O operations related to paging. This is because a normal "user read"operation may trigger subsequent paging I/O operations, which are satisfied inthe same thread context.? Cache Memory functions. These can cause the reference count on a file


23/45



object to drop to zero which, in turn, will cause an IRP_MJ_CLOSE to be sentto the file system.? Closing related file objects. These can cause Cache and Memory Manager releases of memory that in turn trigger IRP_MJ_CLOSE operations to be sentto the file system.

The typical solution to detecting this type of reentrancy is to maintain some"per thread" information. For Windows XP, this can be the filter contextinformation. For Windows 2000 (and earlier) the filter driver should maintain itsown lookaside table for tracking that state.

Q37 How do I cal l a user-mode function from my kerne l driver?

In general, user mode functions are not available from the kernel modeenvironment. This is because routines developed to work in the user mode partof the OS rely upon the specifics of that environment - for example, the abilityto call Win32 functions. Of course, since those functions are not available inthe kernel environment, routines that rely upon them are not, in turn, availablein the kernel environment.

However, there are techniques that allow a function, available in user mode, tobe exploited by most kernel mode components. The best of these (in that it isstraight-forward to use and fits within the Windows architecture) is to couple auser mode service with the kernel mode driver.

The service is typically written to use the Win32 API. This service then createsone or more threads for handling kernel-level requests. Each of these threadsthen opens the kernel mode driver (via a device object created by the kernelmode driver) and issues a custom IOCTL operation using the Win32 functionDeviceIoControl.

The DeviceIoControl call is presented to the kernel mode driver as anIRP_MJ_DEVICE_CONTROL call, and the custom IOCTL is the control code of that call. The kernel mode driver then:

? Marks the IRP as pending (using IoMarkIrpPending)

? Places the IRP on a driver-owned queue specifically dedicated to trackingthese requests? Returns STATUS_PENDING to the caller.

The I/O operation has not been completed at this point and the service's threadblocks waiting for the I/O operation to be processed.

At some time in the future, a kernel mode operation requires invoking the user mode function. Rather than invoking the operation directly, the kernel modethread removes an IRP from the driver-owned queue. If there are no IRPs on thequeue, the kernel mode thread can either block and wait for one to becomeavailable or fail the operation - or even combine the two, so that it blocks andwaits for some finite period of time and after that point it fails the operation.

If there was an IRP on the queue, the kernel mode thread takes the IRP (thatwas removed from the queue) and creates the "output" information for thisrequest by describing the desired operation. Typically, the developer willspecify some command structure format, so that the format contains an "opcode" to indicate the operation being performed, as well as a "request ID" toindicate which kernel level request has constructed this command structure(this is needed when the answer is being provided). Of course, any additionalparameter information is also provided at this point. The kernel mode driver thenperforms normal completion processing on this request by setting theStatus.Information and Status.Status fields, indicating a successfulcompletion. Then the IRP is completed using IoCompleteRequest.

Since the service thread was blocked awaiting completion of this request, it is


24/45



scheduled and runs at some point following the call to IoCompleteRequest. Theservice thread then examines the command described in the "output" datareturned from the driver. That data indicates the specific operation to beperformed - in other words, the user mode function that is to be invoked. Theparameter data should also be present in the output data returned by the driver.

Once the function call returns the necessary information, the thread constructsa response block. While the specific contents of the response block aredependent on the information being returned, it will normally indicate the"request ID" from the original command block. With the response block

constructed, the service thread calls DeviceIoControl again, indicating that it issending a response message.

The driver receives the response message as a newIRP_MJ_DEVICE_CONTROL operation. Examining the message, the driver uses the "request ID" to match up this response information with the originalrequest. Since the original requesting thread is probably blocking, awaiting thisresponse, once the driver has stored the response data in a suitable location, itshould signal the other thread (typically, this is done using a notification event).

The original thread is presumably blocked awaiting this response (e.g., theoriginal thread has called KeWaitForSingleObject using a notification event).Once it unblocks, the necessary data is available and it can continueprocessing.

A warning is in order with respect to using this t echnique, because it ispossible to create deadlocks within the system. If the "original thread" ownsany resources (e.g., locks) at user level that must also be acquired by theservice thread, this technique will deadlock. This becomes an issue becausesome Win32 operations serialize their activities. Typically, this serialization isinternal to the Win32 subsystem and not under the control of the driver or service developer. This becomes an issue, for instance, if the first thread callssome Win32 service (typically this is done by calling a Win32 API). Thisservice then acquires a mutex and calls into the OS to perform someoperation. This ultimately resolves into a call to the driver that is blocked. If thedriver's service then invokes a Win32 function that attempts to acquire thesame mutex the threads will be deadlocked. Thus, this technique must beused cautiously in order to avoid these potential deadlock situations. Since theinternal serialization model of the Win32 subsystem is not documented and issubject to change over time, developers must be vigilant for this type of deadlock.

Q38 What is structured exception handling? How should I use it? Whydo I get STOP code 0x1E (KMODE_EXCEPTION_NOT_HANDLED)? Howdo I deal with this?

Typically a function indicates any errors using a return value. However, analternative mechanism for dealing with errors is provided by the Windowsoperating system and is known as an exception. A driver can raise an

exception by calling ExRaiseStatus, which is described in the Windows DDKdocumentation. The advantage of the exception mechanism is that it onlyinvokes registered exception handler's to deal with the exception. In doing this,it may be necessary to unwind the calling function stack so that control can bereturned to a much earlier point.

Structured exception handling is the mechanism used by a driver to register anexception handler and - when that exception occurs - to verify that theexception is handled by the given exception handler, and if it is handled, totransfer control to that exception handler. Thus, an exception handler consistsof three distinct pieces:

? A protected block of code. This is the region of the program where, if theexception occurs, the exception handler may be invoked.


25/45


26/45



operating system searches for a different exception handler to process thisexception.? EXCEPTION_CONTINUE_EXCECUTION - this value (-1) indicates that theexception has been corrected and execution should continue at the pointfollowing the exception.

For a driver, structured exception handling must be used whenever calling aroutine that might generate an exception. A failure to do so will cause thedefault OS-provided exception handler to be invoked. This exception handler calls KeBugCheckEx with the bug code

KMODE_EXCEPTION_NOT_HANDLED.

In addition, a driver should use structured exception handling whenever performing an operation on a user virtual address. If the address is invalid, theexception handler will be invoked and the driver can process the accessviolation as necessary.

Another interesting use is to trap operations that work by raising exceptions.For example, the breakpoint mechanism works by raising a breakpointexception (STATUS_BREAKPOINT). A driver that has embedded breakpointsinvokes the debugger because it raises this exception. If a debugger is notattached, however, th

Documents

File Systems_IFS FAQ.pdf