Wednesday, January 24, 2007

Privacy and Flash Memory Cards

If you use a camera with flash card memory, or a USB thumbdrive, or any number of other popular miniature electronic storage devices on the market today, then this alert is for you.

Almost all modern flash storage devices use NAND memory. NAND cells have some very cool features, chief among them the ability to retain stored information when you turn off the power. Most digital camera's use a form of NAND memory incarnated as Compact Flash I or II, Secure Memory, XD Memory, etc. The majority of USB storage thumb drives on the market also use NAND memory.

One of the weaknesses of NAND memory is that each bit can only be used reliably so many times. In a perfect world, each cell can be cycled over a million times. However - since cells sometimes have tiny defects and fail randomly, and because we write data in blocks, each block can realistically be written or erased only 100,000 times (or less) before failure. To alleviate the problem and extend the useful lifetime of a storage card, there is a special circuit that forces consecutive writes to be balanced across all available blocks on the device. Each block also has a counter, and the load balancing attempts to keep all counters for all blocks roughly the same.

So what does this have to do with security?

Since "true" erasing is the same as writing, most flash storage NAND devices try not to erase anything to save on wear. They do allow overwriting - but not near as often as you might think. Here is why (by example):

Each time you write a file to that card, it's saved across one or more blocks. Let's use a hypothetical (smallish) file that occupies blocks A through C. Now let's erase that file and write a new file to the device of the same size - a common occurrence with camera's for example. The new file does NOT overwrite the old, it's written to blocks D through F. The erased file is not actually erased -- it's only marked as "cleared." Meanwhile the actual bits within the block are still set to the same state ("0" or "1") as they were when that file was written.

Repeat for the next several files, until you reach the end of the available space on the card and the underlying load manager cycles back to the beginning block for use. Until the original blocks (A through C) are actually overwritten - which on large cards may take a while - that first deleted file can be recovered VERY easily by almost any file recovery tool on the market today.

Format the drive? No dice - since these storage cards do not allow you to do a low level format. When you format, each block is simply marked as cleared, but - again - the bits are not reset to zero. All the files are still recoverable with those same file recovery applications.

Some manufactures provide tools to force a genuine block by block bit reset to zero - a true erasure - but good luck finding those tools in the retail world. (If I find some, I will post links to them here.) Most do not. Some work around the problem by using encryption whenever you write to the storage device, so that even if the file is not truly erased, the only way anyone can read it is by placing the device into the same hardware where the file was created, or by providing a crypto key via a utility. Such solutions have their own problems though, including lower performance and the loss of your data should you lose your key.

The crux of the matter is thus: if you have an older device that you think has failed that may contain information that you don't want revealed or released, then don't throw it away. Destroy it if possible. Cutting them in pieces is good. Hammers also work very well, as does fire. Just please practice safe card destruction yadda yadda etc etc when you shred/smash/burn your old flash memory cards.

For more information on the problems of undesired data retention on memory devices, check out this excellent resource.