This article appeared in the September 1997 edition of NT Developer and no longer in publication.

Download the Code

John's Publication Index

A Cheap Mirror

A treatment on using new-style directory change notifications in order to automatically back up files immediately as they are saved.

by John M. Dlugosz

Happiness is having a current backup. Iím sure most of us feel the same way. So why do most of us go for too long without making spare copies of current work?

If itís going to work at all, it has to be automatic. Thatís the honest truth of it. Especially in a home system where I may work in disconnected sessions throughout an evening or weekend, itís difficult to even know that Iím done for the day. At the office, at least one can establish a shut-down ritual at the end of the day that includes copying changed files.

Even having been burned myself by hardware failures or accidental loss of data, I still donít back up every single day the way I know I should. Iím paranoid about the safety of my files, but to lazy to do anything about it. And, Iím not alone in this respect. I have yet to meet someone who does back up current work daily. Those who do have functioning plans get their secretary or office assistant to actually do it! Programmers as a group are even less likely to stick to a daily backup routine then they are to label floppy disks.

But it doesnít have to be that way. After all, Iím a programmer, right? I should be able to create an automated agent that does this for me (since I donít have a human office assistant). Then I would have my cake (daily backups) and eat it (no routine chores) tooómachines are good at following routine tasks just as programmers are not.

So I thought about using the AT command or some other means to trigger a program automatically. That program might be a simple batch file, or perhaps a MAKE file or a Perl script that analyses what needs to be done.

But once itís automatic, I mused, the big question hit me. Why lose a whole dayís work? If the program were smart enough to only copy changed files, and ran efficiently, why not trigger it every hour? Trigger it as part of a screen saver? Trigger it every minute? The first two seem reasonable, but every minute seems a bit intrusive. With short cycle times, a totally different technique is called for. That is what I present here.

I want a program that will immediately make a copy of my file after I save it. Not hourly or every few minutes, but right away. With this kind of mirroring, I would be truly happy because I would always have a current backup!

The possibilities of such a program are many. Personally, Iíd like to save several revisions of a file. Perhaps the last five changes to a source file or document would be retained for safe keeping. It could have time minimums between backups so that hitting Save several times in rapid succession wonít displace useful versions from the queue. For example, The first copy is always immediate. The old #1 is only itself saved (as #2) if itís more than 30 seconds newer than the current #2. The displaced #2 is then only saved (as the new #3) if itís five minutes newer than the current #3, and so on. This way I would always have immediate work saved, a few minutes ago saved (before a set of major edits), earlier in the day, and so on.

Also, it could know that .CPP files get the version treatment, but .OBJ files are not of interest. They can be re-created if needed.

And, just what does "saving" a file mean? Making a spare copy, anywhere, is good if you accidentally delete or corrupt it. Personally, Iím much more afraid that Microsoft Word will hose my document then I am about hard disk failure.

But for control of more widespread damage, copies should be kept far away from the original. On a different drive, for example, or better yet on a different machine across the network. These can be combinedóthe first copy is quickly made on the same disk, and remote copies are later created in the background.

These are just a few ideas; Iím sure you will have your own scheme that fits the way you work and your personal fears. But they all have something in common: how does the process get triggered in the first place?

Keep your Eye on the Directory

When Win32 was designed, it included a feature often asked for in Windows. That is, directory change notifications. This allows a program to know when a directory changed, so file-manager style list boxes could be refreshed automatically. However, thatís about all the feature is good for. The notification just tells you that something happened. There is no clue as to just what changed. It could be any file in (or under) the directory being watched! The only thing to do was re-scan the directory and compare the data with what was seen before, to look for changes. This is hardly better than the periodic script. It still needs to check through everything to see what changed.

A newer feature is far better suited to our purposes. The Win32 API function ReadDirectoryChangesW will read a stream containing log information concerning a directory. That is, upon using this function you might get something that indicates that "subdir\foo.bar has been modified." This function appears to read from a stream that contains records reporting every event that occurs in the directory. This is what we needódetailed information about exactly what happened, immediately when it happens.

Exploring the Feature

Before using a little-known and poorly documented feature, I wanted to get a more complete understanding of it. Itís good to know exactly what this thing does before investing in a large coding effort.

The program Watch1.cpp, in listing 1, is the result. I will guide you step-by-step through this proof-of-concept code.

First, you need to open the target directory as if it were a file. Actually, directories are special files, so this is not too shocking.

HANDLE h= CreateFile (
name,
FILE_LIST_DIRECTORY, //instead of GENERIC_READ
FILE_SHARE_WIDEOPEN, //don't restrict other uses
OPEN_EXISTING,
FILE_FLAG_BACKUP_SEMANTICS | FILE_FLAG_OVERLAPPED
);

RatWin (My RATionalized WINdows headers) has an overloaded signature of CreateFile that leaves out the security and template-file arguments, and makes the rest of the arguments type-safe through enumerations.

The permissions argument, FILE_LIST_DIRECTORY, can be considered a special flag that you use to open the directory as a file. This argument is normally where GENERIC_READ and GENERIC_WRITE would go. This makes a lot more sense if you have an understanding of what this parameter is actually for, so let me digressÖ

The NT Executive provides access to internal objects through HANDLEs. Security is checked when a handle is created, so the overhead of the elaborate checking need only be done once. Then, you can use that handle to access the object, but only certain types of access are allowed.

Each type kernel object has a set of permission flags representing the various things that can be done with that object. For example, a normal disk file has separate flags for reading data, writing data, reading extended attributes, appending to the file, and others. When the HANDLE is created, you specify exactly what you plan on doing with the object. The kernel then validates that you have all the necessary access rights to that object, and gives you back a HANDLE that can only perform those actions, and no others. If you create a HANDLE that has read permissions only, an attempt to write using that handle is quickly checked. The kernel needs only look at a few bits, rather than performing the detailed security check.

Now using all these detailed permission flags would be a royal pain. There are macros that combine a whole set of related flags together, so you could say FILE_GENERIC_READ and get five separate flags having to do with reading. But, thatís still not quite handy enough. Different types of kernel objects have different permission flags, depending on what that object is for. So do you need different flags for an ordinary disk file, a socket, a named pipe, and a raw device? I really donít know if the bits and permissions are the same for all file-like things. However, I donít have to know. The GENERIC_READ flag is a polymorphic form. This one flag bit means "all the read-related stuff" no matter what kind of object is being used. There is a set of tables somewhere that translate GENERIC_READ into a set of permission flags for each type of object that the kernel knows about.

So, except in special circumstances, you will typically use GENERIC_READ and GENERIC_WRITE with any function that obtains, opens, or creates a HANDLE to a kernel object.

But you could use the detailed flags if you needed to. And thatís what I do here. For a directory, the bit 0x0000001 happens to be for reading data. So thatís what I use here. So what happens if you use GENERIC_READ instead of FILE_LIST_DIRECTORY? I tried it, and it worked just the same. Perhaps it doesnít matter. Perhaps on a secure system there are cases where a user might not have all the rights implied in GENERIC_READ, while the single flag for FILE_LIST_DIRECTORY would be available. So in general, it makes sense to not ask for more than you actually need. Like I said, the ReadDirectoryChanges feature is not well documented by Microsoft.

A Wild and Crazy API Function

Now back to the regular program. Once a HANDLE to the directory is available, the program can read changes logged by the kernel through a call to ReadDirectoryChangesW.

The symbol name ends in W because many of the functions in the Win32 API have ĖW and ĖA suffixed forms. Normally, macros in <windows.h> will redefine one or the other as a plain, non-suffixed, form of the name. But ReadDirectoryChangesW does not have such a macro, and there is no corresponding ReadDirectoryChangesA form.

This function is similar to ReadFile and ReadFileEx. Mentally, I think of reading a stream that contains a log of disk activity, so itís not surprising that the function bears a resemblence to more familiar functions. But, both ReadFile-like behavior and ReadFileEx-like behavior have been combined into one monster function signature!

For normal reading, there are separate functions ReadFile and ReadFileEx that take different parameters. You can read from a file synchronously so that the call doesnít return until the read has completed, use an event flag in the OVERLAPPED structure argument which will be set when the operation has completed, or, provide a callback function that will be invoked when the operation has completed.

For directory changes, there is one single function that takes arguments for all possible uses. I think itís silly to use some arguments or use different arguments depending on what is to be done, so I provided overloaded forms that just take the arguments needed for each use. While at it, I dropped the W, too.

ulong filter= FILE_NOTIFY_CHANGE_LAST_WRITE | FILE_NOTIFY_CHANGE_FILE_NAME;
ReadDirectoryChanges (h, dest, sizeof dest, subtree, filter, overlapped, callback);

The parameters of interest are the open directory handle, the destination where the data is to go, the size of the destination, all of which should be old-hat. Next is a Boolean flag that specifies whether just this directory should be watched, or if subdirectories are to be included. Next is a set of flags that indicate the kind of events we are interested in. In the sample program, you can experiment with different flags to see what kinds of things they cause to be reported. I found that these two flags are sufficient for my needs: LAST_WRITE should cover anything that updates the timestamp of the file, and no matter what a "save" operation does, it certainly will freshen the time stamp. The CHANGE_FILE_NAME is needed for programs that save by creating a temp file, then when that is successful, deleting the old file and renaming the temp file.

Next is the overlapped structure, which the system uses to keep track of various background operations. Just including this argument indicates that I want the operation completed in the background without blocking. The overlapped structure contains an event flag, but Iím not using it here.

Instead, a callback function is supplied. This function will be invoked when the operation completes.

In this program, Iím using the form called alertable I/O. Itís more flexible than simply waiting for the event flag in the overlapped structure, and scales better into what I will need for a full-blown program. Itís not as complicated as using I/O completion ports.

Alertable I/O Completion

Basically, we know that the callback function is supposed to be called. The question is, when? Itís not like a signal or interrupt. Itís more like the Windows message queue.

In a nutshell, when the I/O operation completes, the callback function is placed in a queue. When the thread is in an alertable state, the queue is serviced.

Simple enough, but what the heck is an alertable state and how does my thread get into one? When a thread is waiting, such as with Sleep, WaitForSingleObject, etc. it clearly has nothing better to do. In this case, you can optionally put your thread on alert for callbacks. To this end, many of the functions have ĖEx forms that take another argument, namely a Boolean that specifies whether you want the thread to be on alert or not while it waits.

As usual, I got rid of the separate normal and ĖEx signatures and just overload the single signature to take an optional bool argument. So, to get to the punchline, the "message pump" for this thread looks like this:


for (;;) { const ulong sleep_forever= 0xffffffff; Sleep (sleep_forever, true/*alertable*/); }

Twenty years ago, we were told on TV about the possibility of cleaning our ovens while we slept. Now, we see that a thread can service I/O completion callbacks while it sleeps.

Actually, servicing a callback will cause Sleep to return, even though it hasnít timed out. Thatís why the Sleep is in an infinite loop.

You Called?

When the callback is triggered, it is given three parameters: First, an error code. This is the same error that GetLastError would produce after a synchronous call. In the test program, I print out this value so Iíll know what codes to expect. In the past, Iíve seen differences from what the documentation states in which error codes are reported.

The second parameter is the number of bytes actually read. This program doesnít care, since the data itself is delimited. But the test program prints the value, to see if there are any surprises. In general, Iíve found that when handles are closed, any remaining queued I/O requests have their callbacks triggered with a size of zero.

The last parameter to the callback function is of particular interest. This is the same overlapped structure that was originally given in the function that initiated the background I/O request. There is no information in this structure thatís of interest to me. But, by seeing which instance it is, I can correlate this callback with the original request.

Iíll want to provide my own information to the callback. But there is no user-defined argument anywhere! This mystified me the first time I encountered it. The only way to figure out whatís going on in the callback is via the overlapped structure. My favorite mechanism is to derive my own structure from the OVERLAPPED structure, adding the information I need. Then in the callback, cast the OVERLAPPED* back to the derived class.

Here, the whole mechanism is encapsulated in a watcher class. I derived watcher from OVERLAPPED. Class ratwin::io::overlapped is itself derived from OVERLAPPED, adding a constructor that initializes all the members properly. Class watcher is then derived from overlapped.

When the callback is triggered, it reflects that into a member function of the watcher class.


void watcher::callback (ulong error, ulong bytes_transferred, OVERLAPPED* ov)
 {
 watcher* self= static_cast<watcher*>(ov);  //get back the full object
 self->callback();  //convert to member function call.
 }

The Results

Finally, the callback() member is invoked when the background read operation completes. Here is where I can deal with the results.

The read destination buffer holds a stream of data indicating what happened. The data has the following format:

Offset to start of next record: 4 bytes

Action code: 4 bytes

Name length in bytes: 4 bytes

Name in Unicode: length bytes (length/2 characters)

The buffer is stuffed with multiple records before the callback is triggered. The last record in the stream has an Offset value of zero.

An odd thing is that the destination buffer must be aligned on a 4-byte boundary. The Intel processor doesnít care about alignment when writing words, so this is very mysterious. If the buffer is mis-aligned, there is no error or exception of any kind. Rather, ReadDirectoryChanges just doesnít work. I could have forced proper alignment by re-ordering my declarations, but I left it as-is to illustrate the problem in the code.

The name is relative to the directory that is being watched. It is not nul-terminated, so some care needs to be taken in processing it (the FWATCH example code in the Win32 SDK is wrong). Also note that the length is given in bytes, not characters.

The program displays the results, and then issues another ReadDirectoryChanges request. So what happens to directory changes that occur after the first request has completed but before the second is issued?

The sample code includes a delay before reusing the buffer, to find out. It turns out that no change records are lost. The logging ability seems to be related to the open HANDLE, and ReadDirectoryChanges just reads from the log. This makes things a lot simpler than I originally feared.

Reflections

Now I know how to get detailed directory change information. The test code explored all the issues I had identified, and taught me the details of how this feature performs. But, this information still needs a bit of analysis before it can work in my mirroring code.

When a file is being written, multiple change events are reported. How does the program know when Iím done writing? The simplest way I can think of is to wait for a few seconds to pass without more changes.

Then, the mirroring program would try to open the file for exclusive access. If another program is still using the file, then this will fail. If successful, then proceed to copy the file.

The sample program watch2.cpp applies the post-processing logic. The relevant parts are shown here:


void mirror::process_record (action_codes action, wstring name) { char buf[1024]; const int len= name.elcount(); CharToOemBuff (name.get_buffer(), buf, len); buf[len]= '\0'; switch (action) { case FILE_NOTIFY_INFORMATION::MODIFIED: add_modified (buf); //defer the action break; case FILE_NOTIFY_INFORMATION::RENAMED_NEW_NAME: arrived_file (buf); //process immediatly break; case FILE_NOTIFY_INFORMATION::RENAMED_OLD_NAME: remove_from_modified (buf); break; default: if (verbose) cout << "ignoring action " << action << " on file " << buf << endl; break; } }

If a MODIFIED event is received, donít actually do anything yet, just add the file name to a list. Within the add_modified function, duplicates are eliminated.

If the RENAMED_NEW_NAME event is received, go ahead and process the file immediately. Within arrived_file, the name is removed from the modified list if it happens to be there too.

I also found after testing this that I need to watch for RENAMED_OLD_NAME events too. By removing the name from the modified list here, it wonít try to back up the TMP file (which no longer exists) from the save operation.

So MODIFIED events just add file names to a list. Later, after five seconds of inactivity, this list will be processed. The "message pump" is modified to manage this:


HANDLE dir= watcher::opendir ("test_directory");

mirror ww; ww.watchit (dir); for (;;) { const ulong cycle_time= 5000; //five seconds int retval= Sleep (cycle_time, true/*alertable*/); if (retval == 0) ww.do_deferred(); }

If Sleep waits Ďtill the time is up, it returns zero. That triggers processing of the modified-file list. If an I/O completion routine was serviced, Sleep returns something else.

So Close, and yet so far away

Eureka! The "brains" of the operation work! Now all thatís left is to copy the files whose names this routine produces. Thatís actually a more difficult problem; or rather, it needs more code to do well. It needs to map the file name to its destination name, create directories if necessary, and finally call the primitive CopyFile function. Did I say "finally?" The CopyFile function doesnít match the file timestamp, so thatís another step.

That doesnít illustrate any groundbreaking new features, so Iíll just show you the finished code. Further developing it into a fancy program that archives multiple revisions, knows the difference between interesting and uninteresting (such as .OBJ) files, and provides customization is left to you the reader, to incorporate this file-watching component into what you envision in your own Cheap Mirror program.