Keith and I were recently featured in the March 2008 edition of Entrepreneur magazine discussing the challenges of data retention and data destruction for small businesses.  The article "Storage Smarts" by Amanda C. Kooser, Entrepreneur's Assistant Technology Editor on page 28 of the magazine is kind of short but has a great picture of us.  I wanted to take this opportunity to expand a little bit on what was covered in the article.

Amanda's article focused on the difficulties and expense that small businesses face when trying to determine how long they have to retain various types of data and how to go about proper destruction of the data when it is no longer needed.  When we get questions like this from our clients I like to point to the 1999 Gramm-Leach-Bliley Act, commonly known as GLB or GLBA, as a model for what to protect and how to properly dispose of it later.  Better yet the GLBA charges the Federal Trade Commission with developing standards for protection and destruction of "personal information".  The FTC defines personal data as:

• Names
• Addresses
• Phone numbers
• Bank and credit card account numbers
• Income and credit histories
• Social Security numbers

While the GBLA and the FTC's Safeguard and Disposal Rules are really intended to address the responsibilities of "financial institutions", I believe they are excellent policies for almost every organization.  The FTC has created clear and concise documentation that is designed with enough flexibility that even the smallest company could implement a safeguard and disposal program without great difficulty or expense.

In paragraph four of the article Amanda asked me how we stored sensitive client data that was no longer being used.  We were specifically discussing the data storage problems associated with long term storage of large quantities of client data from our e-discovery and computer forensic services.  I know that I come off sounding kind of old school stating "Hard drives fail. If we have to hold data for more than six months, we transfer it to tape."  I know that kind of thinking gets a lot of SAN, NAS and other RAID storage vendors sputtering but I live in the real world where hardware just inexplicably dies.  The average expected life time of a hard disk is 5 years while the expected life time of a tape is 30 years.  I don't expect you to take my word for this; I direct your attention to the really smart guys at Google Research and their award winning paper "Failure Trends in a Large Disk Drive Population", where they go into great detail on how and why hard disks fail.  They also debunk the myth that hard drives fail due to heat or overuse.  One of my other references for hard disk failure is a highly technical paper from the Computer Science Department at Carnegie Mellon University, with the provocative tile "Disk failures in the real world: What does an MTTF of 1,000,000 hours mean to you?" (Beware: there is math with Greek characters and Poisson assumptions) where researchers are able to show that the Mean Time To Failure (MTTF) of a hard drive is actually much lower than what drive manufactures suggest.   

On the other hand, tape drives have actually been around since 1951, although they didn't start to see common use until the IBM System/360 "9 track" tapes arrived in 1964.  I was personally still using the IBM System/360 tapes on military systems as late as 1995 and I'm sure there are still some in use today.  To summarize my point, magnetic data tapes are very survivable, highly portable and with proper care will store large amounts of data for decades.  I know that some people also dismiss tape as being too slow.  For data that needs to be accessed on a regular basis I completely agree that tape is not the ideal storage media.  When we are talking about long term data retention of large amounts of data, the current generation of LTO-4 tapes store 1.2-1.4TB at a transfer rate of 120MB/sec.  In real world daily use we can typically transfer a little over 100GB of data from SATA II disks to LTO-4 tape in one hour.  Things like hashing, encryption and Write Once Read Many (WORM) can all slow down the tape speed.  The ability to easily do WORM on LTO-4 tape is a big bonus if you are required to be Health Information Portability and Accountability Act (HIPAA) compliant.  By simply using WORM tapes in a most LTO-4 drives you get a tape that can be read but not altered.  We have found the Exabyte/Tandberg Data external SCSI LTO-4 drives to be easy-to-use and very reliable.

As for destruction of large amounts of magnetic media, check back for our upcoming paper on the disposal process of over 600 (60TB) pieces of media.  We'll discuss everything from the initial planning through physical shredding of hard disks.

I want to express our thanks to Amanda C. Kooser, Matt Samarin and everyone else at Entrepreneur magazine.

A recent CBS News article highlights a case where an individual, Sebastien Boucher, was stopped at the Canadian/U.S. border.  At that time, Boucher helped border agents initially inspect his laptop computer.  The purpose of the search seems to be unclear.  It was reported that the agent discovered file names consistent with child pornography.  The agency sized the laptop and upon further review, the agents ran into encryption on Mr. Boucher's hard drive.  Here too, the details are a bit unclear as the article reports the following information:

It is unclear as to how the agent saw the file names of suspect images during his initial search.  If PGP drive encryption was used, the file names should have been encrypted (if it was a separate drive or partition) and they would have never been seen in the initial search.  The only situation that would make sense was if the initial search occurred when the laptop was turned on and the system drive was protected with PGP.  Later, when the agents examined the hard drive while it was turned off, the file names and content would have been encrypted.  

The surprising outcome of this case was that a federal magistrate ruled that forcing Mr. Boucher to reveal his password would be unconstitutional.  It has been argued that a password is something a person knows, which is different than something physical such as a brass key.  Forcing a person to divulge their password forces them to give information that incriminates himself, which ends up being unconstitutional. 

I would assume that if an agent had a search warrant to view the contents of a person's physical safe that could only be opened via a code typed into the digital panel, the code would be protected by the fifth amendment as well.  The difference with a physical safe is that the agent has the luxury of using a safe cracker which could physically open the safe without the use of the code. 

The issue of access to PGP volumes on Mr. Boucher's laptop is a little more difficult than my physical safe analogy above.  PGP volumes are one of the toughest encryption mechanisms to crack.  Unlike other encryption mechanisms, it is not as simple as guessing a small eight character password to get into the volume.  Instead, PGP can be encrypted using long phrases.  Analysis of the payoff of getting into a PGP volume versus the cost/time associated with it usually dictates if someone will even attempt to crack the volume.  Even then, it cannot be guaranteed that the volume will be cracked before it is needed in court or the statute of limitations runs out.

JRD recently performed work for a company who had terminated an employee for under performance of his assigned duties as well as gross abuse of company provided Internet access for personal use. The employer had admonished the employee on two previous occasions for poor performance and Internet misuse. The ex-employee continued to spend most of his day misusing his time on his company issued laptop, looking up sushi recipes, visiting MySpace or talking to his mom about dinner. During the meeting in which the employee was terminated, it was made clear to him why he was being terminated. The employer collected all materials and items that belonged to the company such as his laptop and hard drives, and then they cleaned out his desk and he was escorted out.

Approximately two weeks after the employee was terminated the company received a letter from the State of Maryland, Department of Labor, Licensing and Regulations (DLLR), stating that their ex-employee had filed for unemployment on the grounds that his employers did not provide sufficient reason for his termination. Needless to say this was a surprise to everyone at the firm and on the whole appeared to be an open and close case. The ex-employee spent a large part of his tenure with the company surfing the net, neglecting his daily tasks, and chatting with his friends over instant messenger. This employee was now claiming he was unfairly terminated and he was seeking to collect unemployment payments.

Unemployment is a safety net to help out those who are unexpectedly laid off or are truly terminated for the wrong reasons. However, his former employers felt that in this instance the former employee was taking advantage of the system, thus imposing unfair costs to the tax payers of Maryland, as well as their company. The employers appealed his unemployment request and a hearing date was set to argue the case.

JRD used our extensive computer forensic and electronic data discovery experience to help the company collect and present the computer evidence against the former employee for the appeals hearing. Almost everything the company presented during the appeals hearing was information pulled from the former employee's work laptop. The following paragraphs outline the process we went through to help the company prepare for their unemployment hearing.

JRD was fortunate enough to have worked with the company previously to establish solid Acceptable Use Policies (AUP) and technical employee termination procedures. When the employee was terminated, his access to company files, records, and equipment was removed. During the morning of his termination, all items that belonged to the company were collected while the employee was in his exit interview. This included all work related files, manuals, company laptop, and all external hard drives in his possession. By following this procedure the company protected themselves and the employee by denying him an opportunity to alter any data upon discovering the fact that he was being fired. This also prevented the employee from being able to delete or alter anything on his company computer that would be considered evidence, or could point to his misconduct. Another important step that the company executed was to lock the employee out of all network services prior to termination. By taking these basic steps the company assured themselves that the materials on the former employee's computer were accurate up to his last work day.

While the company did not expect to be involved in litigation with the terminated employee they followed the prudent policy of making a forensic duplication of all terminated employees work computers. Once they collected his laptop, the company's HR department contacted JRD and requested that a forensic duplication of the hard drive be made per their termination procedures.

As a general procedure, JRD typically creates forensic images in either the DD or Guidance Software EnCase (.EO1) formats. Typically the forensic image format selected is based on the client company's preference. In this case we used a bootable Linux CD that contained the Guidance Software LinEn forensic duplication utility to collect the forensic image of the former employee's laptop hard drive. The final forensic image was placed in an evidence bag and sealed for safe keeping.

The company's policy to make forensic images of former employee's computers when he/she leaves or is terminated because it may be needed later in a legal or administrative dispute paid off tenfold in this instance. The cost of making a forensic duplication of a terminated employee's computer hard disk is insignificant when compared to the cost of a potential Employment Practices Liability (EPL) law suit. Based on the 2006 D&O Survey prepared by Tower Perrin, 61.3% of the number of claims brought against surveyed companies was employment related. Following is a highlight of the major types of employment claims:

The 2006 Jury Verdict Research Report provides the following median settlements for employment practices claims:

When you consider that these figures do not include legal defense costs, making a forensic duplication of a hard disk seems like an awfully good idea. When the company contacted JRD about their upcoming unemployment hearing, we simply pulled the forensic duplication of the former employee's hard drive from our evidence storage and made a working copy for forensic analysis. We created a working copy of the forensic duplication so as not to tamper with the original evidence. We could then search the former employee's computer for documents, Internet habits and emails exactly as they were on the day of his termination.

The first set of files we extracted from the former employee's hard drive were his old emails. We then proceeded to extract all of the email from both his Microsoft Outlook "Inbox" as well as his "Sent Items" folders from the last week of his employment. To no one's surprise, most of the emails we reviewed had nothing to do with business. We then produced two print copies of the relevant emails and marked them as evidence for the upcoming hearing. Needless to say we found some very convincing evidence that showed that the company did not terminate the employee without cause.

Our next step was to pull the former employee's Internet history from multiple web browsers that had been installed on his computer. There are many tools out there that make viewing a web browser's history very easy. In this case JRD used two such tools. The first tool is called Web Historian and it is freely available on the internet, portions were based on the Pasco tool written by JRD's own Keith Jones. We used this tool to pull the history from the former employee's Opera web browser. The tool extracts data from the browser's "history.dat" file and outputs it into a Microsoft Excel spreadsheet which is very easy to read. The second tool we used was called "Dork 0.0" to examine Mozilla Firefox web browser history. This tool also outputs its results into a nice spreadsheet.

Collecting the ex-employee's emails and browser history was a simple task that took less than two hours and armed the company with all the evidence they needed for their unemployment hearing. Both tools we used are available to anyone on the Internet, and are easy-to-use. Since all evidence at the unemployment hearing was required to be in print form, we generated a giant pile of emails and spreadsheets that amounted to an evidence folder the size of a small phone book. We had more than enough evidence to support the company's claims for terminating the employee but now we had to prepare this huge pile of paper documents in a coherent and presentable manner.

Working with the companies HR department, we developed a plan to walk the examiner, citing specific evidence, through the company's reasons for termination. We created an evidence summary outline which gave a general explanation of each piece of evidence and why it was relevant. All of the evidence was placed into a large binder with each piece of evidence being separated with a tab, for ease of reference. In the end, we had helped the company and the HR department create a narrative of events that lead up to the termination of their former employee with the supporting evidence arranged in a way that was very easy to follow. A secondary benefit of putting the evidence together in this manner was that it allowed the company's HR representative to easily explain everything to the examiner even though she was not involved with the technical evidence collection or forensic analysis process.

The day of the unemployment hearing the HR representative was able to explain everything very clearly to the examiner. Along with a few witnesses, the evidence JRD collected for the company made for an open-and-shut case. It usually takes four to six weeks to receive a judgment from the State of Maryland DLLR. The company received a judgment in their favor within 3 business days.

In conclusion the company did several things that led to a successful outcome of a potentially damaging unemployment case:

1. The company had established, written Internet Acceptable Use Policies
2. The company's HR department had fully documented each meeting with the former employee where his poor performance was discussed
3. Personnel in the company's HR department fully understood and followed procedures designed to protect company data and assets during the former employee's termination
4. The company's HR department had established policies and procedures for automatically initiating a forensic duplication of any terminated employee's company owned computer

Article in PDF Format 39.53 Kb

During the last several incident responses that JRD has been involved in we've noticed a rather disturbing trend. The intrusions have typically started with some very good quality spear phishing targeted at corporate executives and key personnel. While I've been thinking about this and warning our clients about the threat I came across several articles that confirmed what we have been seeing.

The first article I found was the most recent Phishing Attack Trends Report (1/25/08) from the Anti-Phishing Working Group (APWG). The report covers a lot of phishing topics but the APWG notes that a drop of general phishing attacks in November 2007, was mirrored by an equal increase in phishing attacks targeted at corporate executives nd key personnel.

The second article was from Internet Security Systems' X-Force team at IBM as reported by Matt Hines at InfoWorld. The X-Force was reporting that they had seen an increase web sites hosting sophisticated "personalized" attacks designed to take advantage of the unfortunate user's particular browser and operating system. They also reported that some of the more advanced malware groups were collecting the IP addresses of the web site visitors so they didn't repeatedly attack the same host.

Both of these reports closely mirror several of the intrusions that we've recently responded to. Unfortunately in most cases the intruder has experienced some level of success with these personalized phishing attacks. We've also noted that there is usually a series of these attacks not just a singular phishing attempt.

We are proponents of multiple security systems as part of a strong IT securityinfrastructure. In these cases even the best anti-phishing, anti-spam filtering would have failed due to the personal nature of the phishing emails which included correct email addresses and current corporate information from the victim companies. I'm not saying that content filtering email doesn't work, it does, and everyone should use it as one method of protecting users. The problem is that even the most up-to-date filtering usually can't protect you from a email that appears to have legitimate recipients, legitimate content and normal looking attachments.

In cases where these attacks were successful we saw a breakdown in end user security awareness training and web proxy filtering. This is question of multiple layers of security; if the incoming email filtering fails and the end user fails a properly functioning web proxy filter has a good chance of blocking the malicious site (contained in the email or attachment) or preventing the malware download.

Along that line, it is important that the web proxy be able to review both HTTP and HTTPS content. It doesn't hurt to monitor text messaging traffic if that is common in your environment; there are plenty of phishers working the instant messaging route too.

About two weeks ago Keith Jones forwarded me an email from CPSI, a Baltimore IT services company(1), that I immediately thought was spam. The email stated that in November's Special Session, the Legislators of the State of Maryland passed into law a 6% sales tax on computer services, to take effect July 1, 2008. Again my reaction was "this must be spam; no one could be that ignorant of economic reality". The email contained a link to the Maryland Computer Services Association, Inc., website marylandneedsit.org. Not only did I discover that the tax had already been passed into law but that it was written in such a broad manner that it covered almost every IT related function(2).

Article – Tax – General (11–101.)

(1) “Computer service” includes:

(i) computer facilities management and operation;

(ii) custom computer programming;

(iii) computer system planning and design that integrate computer hardware,software, and communication technologies;

(iv) computer disaster recovery;

(v) data processing, storage, and recovery;

(vi) hardware or software installation, maintenance, and repair.

I was stunned. I've been in working in Maryland since I left the US Army at Fort George G. Meade in 1997. Since that time all of my jobs have been IT related. I've worked in defense contracting, computer security and biotechnology in that past ten years. All of those IT fields are critical to the economy of Maryland and will suffer under this new IT Services Tax. The more I researched the issue the stranger it got.

It found that the Maryland Chamber of Commerce was listing repealing the IT Services Tax as a priority for the 2008 legislative session. The Chamber had some interesting statistics:

• Maryland spends millions of dollars annually to attract and expand technology companies. Computer service companies employ 68,000 Marylanders, with an annual payroll of $5.2 billion, paying wages nearly twice the statewide average. Imposing a sales tax on these activities will jeopardize these jobs.
• Maryland businesses will pay the majority of this $200 million computer services sales tax. The recent special session resulted in more than $800 million in new business taxes.
• This new tax on computer services could jeopardize BRAC jobs that would have relocated to Maryland. Maryland-based subcontractors will face a 6 percent price disadvantage when bidding to participate on federal contracts.

All of these things sounded bad to me. The chamber also provided a link to information on the 10 other states with IT services taxes. My reaction to the list was that none of the states with IT services taxes where "high-tech" states and three of the ten states, Pennsylvania, Florida and Massachusetts, have already repealed their IT services taxes.

Things got really strange when I found that the Comptroller of Maryland, Peter Franchot, was calling on the legislature to repeal the IT Services Tax. Franchot's assessment was "This technology tax, if allowed to stand, will erode Maryland's competitive advantage in the Knowledge-based economy. The computer services tax will take a disproportionate toll on those small and
independently-owned businesses that are the backbone of strong communities," said Comptroller Franchot. "The last thing we need is another tax increase, especially one that will undermine our Knowledge-based economy and damage our long-term economic success."

At this point I decided to do something I'd never done in my life, I called a politician. Specifically I phoned Delegate Gail H. Bates, Republican, District 9A, Howard County, MD. Delegate Bates is also a long standing member of the Appropriations Committee. My hope was that with her background in the state and specifically on tax issues she would understand my concerns. As was extremely pleased to find out from Delegate Bates's office that not only were they taking many calls on the IT Services Tax, but that Delegate Bates had already sponsored a bill (HB 187) to completely repeal the tax. Better yet, Delegate Bates and the Maryland Republican Caucus held a press conference that we attended on January 30, 2008, where she carefully outlined how the projected $200M revenue increase from the IT Services Tax could be replace by simply cutting
some existing expenses.

Keith and I are encouraging all our fellow Maryland small business owners to call Delegate Bates
at (410) 841-3556, (301) 858-3556 or 1-800-492-7122, ext. 3556 (toll free), e-mail: This e-mail address is being protected from spambots. You need JavaScript enabled to view it and support the full repeal of the IT Services Tax.

(1) JRD is not associated or affiliated with CPSI but we thank them for bringing this issue to our attention.

(2) Full text of the State of Maryland Senate Bill 2 - Tax Reform Act of 2007 is available here (pdf).