Many cases hinge entirely around the contents of e-mail and attachments.  Because of this it's important to have a basic understanding of the structure of the most common enterprise e-mail applications.

Types of E-Mail Servers
     Microsoft Exchange
     IBM Lotus Notes
     Novell GroupWise
E-Mail is Boxes within Boxes
Where is the E-Mail Located?
     Server-Based
          Microsoft Exchange with Outlook
          IBM Lotus Notes
          Novell GroupWise
     Webmail
     Webmail through Outlook
     Outlook Express
     Windows Mail
Other Strange Sources of E-Mail
     E-Mail Backups May Be Your Friend
     E-Mail Appliances
 

Types of E-Mail Servers

In most environments, there are three types of e-mail or messaging server commonly in use.  Typically these are Microsoft Exchange, IBM Lotus Notes or Novell GroupWise.  It is not unusual in large enterprise environments to have any combination of these e-mail servers in use.  This is particularly common in situations where a company has acquired many smaller companies over a period of years.

• Microsoft Exchange - Microsoft Exchange is the most commonly encountered mail server in most corporate environments. Extraction of individual user mailboxes, referred to as PST files, from a Microsoft Exchange server is easily accomplished with a Microsoft provided tool called EXmerge.
• IBM Lotus Notes - Lotus Notes is the second most commonly encountered mail server. Extraction of individual user mailboxes, referred to as NSF files, is easily accomplished.
• Novell GroupWise - Novell GroupWise is an older mail server package no longer found is commonly in most environments. Novell GroupWise is known for its reliability and ease of maintenance, which keeps it from being replaced by newer systems. Extraction of individual user mailboxes, referred to as simply mailboxes, is notoriously difficult. Extraction of GroupWise mailboxes requires network access to the GroupWise server, GroupWise administrator permissions and expensive third-party software.

The bottom line on mail servers is, Microsoft Exchange and Lotus Notes are easy, Novell GroupWise is hard.

E-Mail Is Boxes within Boxes 

Modern e-mail servers and e-mail clients such as Microsoft Outlook and Exchange do not store e-mail on hard drives and in simple formats like text.  Modern e-mail systems store e-mail in a proprietary database within other proprietary databases, the box within a box.  For example: Microsoft Outlook stores e-mail in a PST or OST file, which are actually containers for the e-mail messages and attachments.

In short, without the proper viewer (in this case, Microsoft Outlook) we cannot actually view the contents of a PST or OST file.  If we were to attempt to open an individual's PST file without the proper viewer it would simply look like machine garble.

To make matters more complicated, individual user mailboxes are stored in yet another database on the server.  In the case of Microsoft Exchange, user mailboxes are stored in an Exchange Database file called an EDB file.  To complete our box within a box analogy; a users e-mail is stored in a Microsoft Outlook PST file that is the first box, which is then stored in a Microsoft Exchange Database file, which is the second box.

There are a number of E-Discovery ramifications to the storage of e-mail databases within databases:

• The server e-mail database may actually be several databases, none of which can be copied or forensically acquired while the mail server is running.
• Some mail servers do not store all of the users e-mail on either the server or on a user's computer; rather some of the users e-mail is stored in both locations.
• Some e-mail systems utilize complicated security structures to protect user's mailboxes. This security can make the acquisition of e-mail for authorized E-Discovery purposes very difficult.
• In some environments, System Administrators make extensive use of mailbox encryption and compression features. Encrypted mailboxes require additional processing time, while the mailboxes are decrypted or passwords recovered. Compressed mailboxes may result in unrealistic collection expectations as the amount of e-mail, a compressed mailbox may be up to 10 times the size of the compressed mailbox.

Where Is the E-Mail Located?

Depending on which e-mail solution is used, and individual users e-mail can reside only on their local computer, only on the e-mail server or a combination of both.  By default, each e-mail solution has its own way of handling individual user e-mail storage; however, it is important to keep in mind that an e-mail administrator can configure user e-mail storage however he likes.  E-mail is usually stored as follows:

Server-Based: 

• Microsoft Exchange with Outlook - Current Microsoft Exchange e-mail can always be acquired directly from Microsoft Exchange server. In most environments users also have an Outlook PST file on their computer that may contain more information than what is stored on the Microsoft Exchange server. Microsoft Outlook users also frequently have OST files on a computer. OST files, also known as Off-Line Folder files. The OST file makes it possible for the user to work within Outlook while disconnected from a Microsoft exchange server and then synchronize next time their online. Because PST files and OST files may contain differing information is important to acquire both. It is also quite common for users to archive old e-mail by creating additional PST files. This means on a user's computer there could be several PST files and OST files that may all contain different relevant e-mail information. The default location of a PST file in Windows XP is C:\Documents and Settings\user\Local Settings\Application Data\Microsoft\Outlook. For Windows Vista, it is C:\Users\user\AppData\Local\Microsoft\Outlook. (Replace user with the user name specific to the computer)
• IBM Lotus Notes - Lotus Notes e-mail is stored in NSF files on the user's local system, as well as on the Lotus Notes or Domino server. Processing of Lotus Notes e-mail may also require the names.NSF and user.ID files from the user's computer. These two additional files contain security information that may be required to properly process the e-mail. Frequently, Lotus Notes and NSF files are simply converted to Outlook PST files to make them easier to process. The default location of a NSF file in Windows XP is C:\Documents and Settings\user\Local Settings\Application Data\Lotus\Notes\Data. For Windows Vista, it is C:\Users\user\AppData\Local\Lotus\Notes\Data. (Replace user with the user name specific to the computer)
• Novell GroupWise - Novell GroupWise e-mail is stored in a series, typically five, database files on the Novell GroupWise server. Expensive third-party software must be used with administrator rights to extract user mailboxes from Novell GroupWise servers. Once acquired, Novell GroupWise mailboxes are also typically converted to Outlook PST files to make them easier to process. GroupWise does not create any storage on a users machine unless the user initiates it. In this case the user specifies where the file is saved and it receives and MLM extension.

Webmail:

Webmail is a type of e-mail that is hosted on an outside company's website and accessed through an internet connection.  The most common webmail providers are Yahoo, Google (gmail), AOL, and MSN (hotmail).  Although webmail is more commonly used as a personal e-mail account, it is not uncommon for employees to use it for business as well, especially for smaller companies that don't have the need for an e-mail server.

In its rawest form, all e-mail is stored on the provider's server (i.e. Yahoo mail is stored on a Yahoo server).  The only trace of the e-mail that will be found on the user's computer will be in their temporary internet files.  While it is possible to sometimes see full messages in these temporary files, they are typically only crumbs compared to the full content of their mailbox.  If discovery requires access to all of the mail from a webmail account, the webmail provider will usually release it with a proper subpoena.

Webmail through Outlook:

It is possible for a user to setup Outlook to download webmail so that it can be accessed without using the web interface and can be viewed while offline.  By default, Outlook will only download the titles of the e-mail.  Once the user clicks an e-mail to view it, it will download the content.  Any downloaded information is stored in a local PST file.  This PST file and its contents are easily accessible for discovery off of the user's machine but it will only contain the mail that has been accessed through Outlook.

Outlook Express:

Outlook Express is similar to Outlook but it has less features.  It is usually more common for personal use as it comes preinstalled on most computers.  Outlook Express stores the e-mails in separate, folder-named DBX files such as Inbox.dbx.  Usually DBX files need to be converted to PST in order to be processed.  The default location of DBX files on Windows XP is C:\Documents and Settings\user\Local Settings\Application Data\Identities\{###}\Microsoft\Outlook Express. (Replace user with the user name specific to the computer and ### will be a long string of random letters and numbers.)

Windows Mail:

Windows Mail is the Vista replacement for Outlook Express.  It has now been replaced by Windows Live Mail.  Unlike Outlook Express, both versions of Windows Mail use individual files to save your e-mail messages instead of container files.  Mail items are saved as EML files.  The default location for EML files in Windows Vista is C:\Users\user\Local Settings\Microsoft\Windows Mail\Local Folders.  (Replace user with the user name specific to the computer)

Other Strange Sources of E-Mail

E-Mail Backups May Be Your Friend:

The availability of e-mail is considered to be business-critical in most environments.  Because of this most IT departments make regular backups of user mailboxes and mail server databases.  It is important to find out how e-mail is backed up, and how often e-mail is backed up.  Some IT departments will regularly backup individual user mailboxes, while others simply pack up the entire mail server database.  In either case, it may be preferable to acquire a copy of a recent e-mail backup rather than interrupt a business-critical system.

Another advantage of e-mail databases and user mailboxes recovered from backups such as magnetic tape is that the backups may contain older e-mails that are no longer available on either the mail server or the individual user's computer.  This can be a very important distinction as many IT departments enforce strict mailbox size and age limits on "live" e-mail (email still on servers) that would not apply to backups. 

E-Mail Appliances:

In most large enterprise environments local delivery of e-mail to users is handled by an e-mail server such as Microsoft Outlook or IBM Lotus Notes.  Incoming and outgoing organizational e-mail will often pass through a high-performance e-mail appliance and commonly a spam/antivirus filtering appliance.  These appliances are sometimes known as mail gateways or Mail Transfer Agents (MTA).  E-mail appliances are purpose built computers designed to handle e-mail at speeds and volumes that a normal computer could not handle.  E-mail appliances do not usually keep copies of incoming or outgoing e-mails, but they do often log the sender and destination e-mail addresses.  This may be helpful in cases where the knowledge of the communication is more important than the content.

It looks like the people at JD&A aren't the only ones just stumbling upon personally identifying information.

CNN reports  that about a year ago, Chris Ogle of New Zealand bought a used MP3 player from a thrift store in Oklahoma, and a few weeks ago finally plugged it in to his computer.  Instead of a blank player or some leftover MP3s, however, Ogle discovered confidential US military documents on the MP3 player, including "mission briefings and lists of equipment deployed to hot spots in Afghanistan and Iraq," along with "home addresses, social security numbers, and cell phone numbers of U.S. soldiers."

According to the Associated Press, the MP3 player was collected by US officials on the 28^th of January.

As much as we'd like to think so, this is not an uncommon story; the Privacy Rights Clearinghouse's "Chronology of Data Breaches" shows that incidents like this happen all the time.

For firsthand evidence of unsecured personal info, we need only take a look at JD&A's current PII Quest statistics.  In just a few months, we've saved potential victims over $70K in non-recoverable damages and from wasting almost 9,000 hours trying to rectify their situations; that translates to a lot of PII just floating around out there.  And all the information that we've stumbled upon and destroyed we have come across just as easily and innocently as in Ogle's story.

The PII Quest's high numbers along with the multitude of stories similar to Ogle's illustrate just how many people don't take basic security precautions regarding their data.  Data security is not just a corporate or military "thing."  Everyone should be taking steps to ensure that their PII is safe and sound, since a criminal getting just a few key items, such as a name and social, can lead the average victim to spend nearly $1,000 and 133 hours of unrecoverable time and money trying to recover their identity.

So what steps can you take?  Well, for example, every time you save or copy a file, ask yourself if you'd be fine with a stranger getting their hands on it.  If not, the file (or the device to which you're saving it) should be encrypted.  If whatever device that file is saved on is stolen or lost, there's a good chance that whoever gets their hands on that device will be able to access whatever you've got on there.  (Quick note: the simple "logon" password protection used in most versions of Windows is NOT encryption; files "protected" using only this method are still very much accessible.)

Contrary to the sound of the word, though, "encryption" doesn't have to be complicated or expensive; there are free, easy-to-use programs out there like TrueCrypt.  Your thumb drive, for example, is a thin piece of plastic away from breaking off your key chain.  Make sure that if it gets lost, you can write it off instead of sweating it out.

Using encryption is just one step you should be taking to safeguard your information; check out the "Protect Yourself" section of JD&A's PII Quest for free tools, inexpensive tricks, and info on federally-mandated programs (all of which many ID theft prevention companies will try to charge you for) that can assist you in securing your PII.

On the twelfth day of dd, we use losetup to view the image files we’ve made with dd.

Let’s say you want to take a look at one of those dd image files you made. By using the command losetup you can turn an image file into a device so that you can do just that. First go to the directory that your image is stored. Next, use the basic losetup command which looks like this:

# losetup /dev/loop0 output.img

This will make the contents of “output.img” mountable like any other disk partition. Multiple loops can be setup by changing the number at the end (loop0, loop1, etc.) Use the standard mount command with /dev/loop0 as your device.

# mount /dev/loop0 /mnt

If you want to go directly to a mounted image you can use the following command:

# mount -o loop output.img /mnt

This command will do the same thing as the first two commands listed and you can immediately access the mounted image at /mnt. The number at the end of the loop will automatically be assigned. When you are finished with the image you can also unmount with the standard “umount” command. Then to remove the image from the loop, use the following command:

# losetup -d /dev/loop0

In order to see loops that are active, use this command:

# losetup –a
/dev/loop0: [fd00]:4694055 (output.iso)
/dev/loop2: [fd00]:4694077 (output2.iso)

This technique will only be able to show one partition at a time. If you have an image that contains multiple partitions, there are several extra steps to take in order to view the contents. Start by creating a loop for the whole image as per the first command. Next, input the following command:

# fdisk /dev/loop0

Next enter u to change diplay/entry units to sectors. Then enter p to see your device information. You will see something like this:

Disk /dev/loop0: 500.1 GB, 500106780160 bytes
255 heads, 63 sectors/track, 60801 cylinders, total 976771055 sectors
Units = sectors of 1 * 512 = 512 bytes
Disk identifier: 0xfa12aa49

Device Boot   Start   End   Blocks   Id   System
/dev/sda1 *  63  401624  200781  83  Linux
/dev/sda2  401625  976768064  488183220  8e  Linux LVM

Choose which partition you want to view and multiply the “Start” sector number by 512. (Example: For the first partition starting at 63: 512*63=32256). Enter q to exit back to the command prompt.Now we can set our first partition to a new loop. If you want to view all of the partitions, you will need a separate loop and mount point for each. Create your first partitions loop use this command:

# losetup –o32256 /dev/loop1 output.img

In this command the “-o” should be followed by the number derived by multiplying your starting sector by 512. From here you can mount as normal and view your partitioned image. Changes can be made to your mounted image if the file system type allows for it.

Well, that wraps things up for The Twelve Days of dd; we hope you've enjoyed them. We’ll continue blogging about other useful e-discovery and computer forensics topics, so remember to check back often, or add our blog as an RSS feed. Happy holidays!

On the eleventh day of dd, we dd a tape (part two of two).

On the Tenth Day of dd we learned how to get our tape drive prepared for dd. Now we can begin the actual command line process.

With your tape in the drive the first thing we want to do is rewind the tape and move it to the first file or file 0. Some tape drives like my new Quantum SDLT 600 drive do this automatically when you put a new tape it but others don't. Type the following:

# mt asf 0

The asf tells the drive to position the tape at the beginning of the count file. Positioning is done by first rewinding the tape and then spacing forward over count filemarks. But let’s check just to make sure. Type the following:

# mt status

This should verify for you that you are at the beginning of the tape and on file 0. There are a bunch of codes that the mt status command will show that are of use to you:

BOT - Beginning Of Tape
EOT - End Of Tape
EOD - End Of Data

Next we start the process of copying data and more importantly trying to guess what block size the tape backup software that created the tape used to write the tape. There is no real science to this part just some logical guessing. First mount up a target drive that you are going to dd the tape data out to; in our example we will assume that our target drive is /mnt/storage. Then start by trying the smallest block size first. Type the following:

# dcfldd if=/dev/nst0 of=/mnt/storage/file0.img bs=512b

If everything is good you will get a number of blocks copied from the tape to /mnt/storage directory and a positive dcfldd output that looks like:

20044080+0 records in
20044080+0 records out
20525137920 bytes transferred in 5665.925325 secs (3622557 bytes/sec)

Remember from my previous postings on dd that +0 is good and +1 or +2 is bad. You will also probably get some kind of error message from the tape drive if you didn't read any data. If the read failed you have to repeat the whole process and increment the block size. Type the following:

# mt asf 0 <---to move the tape back to file 0
# mt status <---just to make sure we are back on file 0 and at BOT
# dcfldd if=/dev/nst0 of=/mnt/storage/file0.img bs=1k <---notice that we have incremented the block size to 1k

Here is the bad news. You don't know how many files are going to be on the tape and they may have different block sizes. I usually find that the blocks are 512 bytes, 1k, 4k, 16k, 32k and 64k. Different tape backup software from different manufactures can create files in a variety of different block sizes on a tape. Some tape backup software creates tape lead-ins and lead-outs that are small like two blocks of 512 bytes and then writes all other files in 16k blocks.

Once you have successfully found the right block size and copied file0.img the tape will now be on file 1. It is important that you keep track of what file number you are working on so that when you do the “mt asf #” command you don't mess up and keep copying the same file over and over. For example after having successfully copied file 0 you would do the following:

# dcfldd if=/dev/nst0 of=/mnt/storage/file1.img bs=1k <---notice we changed the file name

If everything went well you just keep copying away but if it errors you have to go back to file 1 by typing:

# mt asf 1 <---rewinds the tape back to file 1
# dcfldd if=/dev/nst0 of=/mnt/storage/file1.img bs=2k <---notice we changed the block size

You want to just keep repeating this process until you get an EOD or EOT notice from mt status. When you are done copying all of the data off the tape you can rewind and eject the tape by typing:

# mt offline

Once you have finished copying all the data I recommend generating MD5 sums of all of the files. You could do this while doing the dcfldd but I like to write all of the hash values to a single file when I'm finished. The easiest way to do this is to use MD5deep. MD5deep allows you to recursively examine a directory and make MD5 hashes of every file in the directory and every subdirectory. Type the following:

# md5deep -r –e -l /mnt/storage/* > md5sums.txt

The “-r” tells md5deep to recurse, the “-e” provides an estimate of how long the md5 hash will take to compute for each file and the “-l” tells md5deep to only output the file name not the absolute path into the log file. Be sure to read the man page for md5deep for more options. Generating MD5 hashes of large files can take several hours, for example a 100GB file takes about 3 1/2 hours to compute on a 3Ghz P4. It can take 10-15 hours to compute all of the MD5 hash values for an entire tape.

Tomorrow, for the twelfth and final day of dd, we’ll be examining how dd’s “losetup” can allow you to view your image files. Stay tuned!

On the tenth day of dd, we dd a tape (part one of two).

When it comes to using tapes, there is a lot more involved than just a few command line prompts. There is typically more prep work involved in making sure your tape drive is running properly. This is because most people don’t already have a tape drive up and running like the rest of the drives we have looked at. We will start by looking at our equipment setup before we get into the dd commands.

The first thing you need to do is make sure that you've got a SCSI card with a 68-pin connector in your Linux system. I recommend using a solid Adaptec SCSI card like the AHA-2940. You really can't go wrong with Adaptec SCSI cards; there is driver support for Adaptec cards in almost every OS. I used Fedora Core 3 for all the duplication, you could probably also use a bootable CD-ROM like Knoppix but I like an installed OS for really long processes like this. Knoppix does have automatic detection and driver support for Adaptec SCSI controllers.

Next you’re going to need a tape drive. I recommend checking The Linux Tape Device Certification Program web site before purchasing your tape drive. Tape drives are expensive and it's best to be sure that you are going to get hardware that someone else has actually had some luck with before dropping big bucks. I always check the certification list before purchasing a tape drive and I've never had a drive that wouldn't work. When it comes to getting a tape drive right away I recommend CDW, they aren't always the cheapest but when you need the equipment right and right away you can count on them. I've been using the same account manager at CDW at four different companies for the past ten years and they have never disappointed me. It's also nice to be able to call an account manager and find out critical information prior to making your purchase, such as, “Does a SCSI cable come included with this tape drive?”

I've used Sony, Quantum, Overland and HP tape drives over the past 4-5 years and have found all but the HPs to be very reliable. I've experienced a number of drive failures and robot arm failures on HP systems.

Once you have all the equipment together and connected it is time to get started. The first thing you should do is write-protect your source tape. There is no magic to write protecting the tape; it is usually just a slider on the back or side of the tape. The magic is in figuring out which way the slider has to be positioned to be write-protected. The only good way to figure this out if you don't have the instructions that are packaged with the tape is to go to the manufactures web site and dig the information up. Beware that write-protection on some tapes is not perfect you can still erase some tapes even when the write-protect is activated.

SCSI tape devices on a Linux system are recognized as /dev/st0 and /dev/nst0. /dev/nst0 is the same tape device as /dev/st0 but the /dev/nst0 tells the tape drive not to automatically rewind the tape. For everything we do here we want to use /dev/nst0. Obviously if you have more than one tape drive attached they will be /dev/st0, /dev/st1 and so on. Be sure to check the dmesg if you are having problems (dmesg | less). Good dmesg output should look like this:

SCSI subsystem initialized
scsi0 : Adaptec AIC7XXX EISA/VLB/PCI SCSI HBA DRIVER, Rev 6.2.36
Adaptec2940 Ultra SCSI adapter
aic7880: Ultra Wide Channel A, SCSI Id=7, 16/253 SCBs
(scsi0:A:5): 20.000MB/s transfers (10.000MHz, offset 8, 16bit)
Vendor: QUANTUM Model: SDLT600 Rev: 1E1E
Type: Sequential-Access ANSI SCSI revision: 04
st: Version 20041025, fixed bufsize 32768, s/g segs 256
Attached scsi tape st0 at scsi0, channel 0, id 5, lun 0
st0: try direct i/o: yes (alignment 512 B), max page reachable by HBA 1048575

We’ll finish up dding a tape tomorrow, on the eleventh day.  Click here to continue to part two.