Lois Lerner's Emails Lost

[sunmaster14]

<snip>

You really don't understand IT do you?

I find this kind of unhelpful comment to be extremely annoying. If you have something you want to say, put it out there for scrutiny. Don't hide behind a sarcastic quip.

And, no, I'm not an IT professional, but for a layman, I think I have a pretty good understanding of computer technology. The computer I'm using is DIY by the way.

 

[The Don]

Well, I think it's complicated enough that you need IT professionals to do the job properly. But I also don't think it's too much to expect a large organization to do a proper job of archiving data.

In your organization, what are the limits for each user on the email server? I remember in the late '90s that I was running into my storage limits all the time at my firm. A few years later, the problem completely disappeared and for all intents and purposes there was no limit. I figured this meant that storage had become so cheap that IT people just decided to let everybody store whatever they wanted.

Good storage is still flipping expensive. Backing up good storage is also flipping expensive. Depending on backup strategy and how available the backup needs to be the storage requirements for a comparatively modest project can run into millions of pounds. Scale that up to a huge organisation and the costs are in the hundreds of millions.

 

[elbe]

Well, I think it's complicated enough that you need IT professionals to do the job properly. But I also don't think it's too much to expect a large organization to do a proper job of archiving data.

I haven't seen evidence that they aren't. Again, 6 months isn't blowing me away, especially for internal data (even if it's about "customers", it's still internal). I haven't seen anything about their tax data solutions.

In your organization, what are the limits for each user on the email server?

Internal or customers? Internally we limited it, I believe, down to 2 gigs and 1 or 2 years back (anything older than the date is automatically purged by the exchange server), and even that long a time is probably because we have a lot of users who "save" information by just moving old emails to outlook folders. For emails we host for customers is unlimited (beyond the 6 month backup retention period), which has created a bit of a data growth problem for us.

I remember in the late '90s that I was running into my storage limits all the time at my firm. A few years later, the problem completely disappeared and for all intents and purposes there was no limit. I figured this meant that storage had become so cheap that IT people just decided to let everybody store whatever they wanted.

Oh, nooooo. Storage has become cheaper, yes, but users have compensated by trying to use more of it. When I started in this company 8 years ago a client was large if it had more than a gig of data on our servers, now double digits are common, triple digits are annoying, and terabyte clients make us hate them. We've had to go through multiple storage solutions alone just to keep up with the growth rate. Our current solution seems to be working (which allows potential growth into the petabyte range) but there are concerns because it isn't as fast as other servers so customers will complain about lag when accessing their data stored there. It's basically a never ending battle that's never as simple as throwing another TB drive in a server. And every time we allow for expanded customer data we have to raise our backup capacity to store the daily and weekly backups going back all six months.

As an example, we have an internal server which as 4 primary "drives" on it. They are used to work with customer data (converting it between competitors systems and ours mostly). The primary drive is currently ~18tb and spends most of it's time being nearly entirely full, the other 3 are 3tb drives that act as spillover but as far as I know those three are also not backed up so they shouldn't be used for data that can't be lost. And that's just internal.

 

[Unabogie]

I know, right? How on earth can a government industry keep up with best IT practices of private businesses when it only gets four times the IT budget that the average corporation spends?

I agree with you that government agencies should follow best practices, and six months isn't long enough, but that's not what this phony Fox scandal is alleging. They are alleging that Lerner made up a story about lost emails because she's hiding something, when the actual record clearly shows that there is documentation backing her up completely on this. Ziggurat and Sunmaster refuse to concede defeat on this point. Do you join then on that liferaft? Or do you agree that the email chain I posted proves conclusively that her IT department failed to recover any files off of a corrupted hard drive and told her so in 2011?

 

[The Don]

I find this kind of unhelpful comment to be extremely annoying. If you have something you want to say, put it out there for scrutiny. Don't hide behind a sarcastic quip.

And, no, I'm not an IT professional, but for a layman, I think I have a pretty good understanding of computer technology. The computer I'm using is DIY by the way.

Speaking from personal experience I found the transition from understanding the costs and specifications from a personal or even a workgroup solution to a full corporate solution dizzying.

I simply could not understand that although I could go out and buy a 10 MB disk for a few hundred pounds, getting that much storage assigned in a corporate solution ran into many hundreds of thousands of pounds.

More than 20 years on the same applies, but the numbers are larger. I can go out and buy a 16TB drive for under a thousand pounds. Securing that much storage in corporate storage with the requisite backup and restore capability (and the managed services around it) can easily run into millions over the lifetime of the solution.

 

[stevea]

The IRS last Friday (natch) finally told Congress that it couldn't recover over two years worth of emails from Lois Lerner, from early 2009 to mid-2011. In its explanation of the way archiving works at the IRS, it's clear that we can't really be sure that a complete record of emails exists for any IRS employee prior to December 2012. Backup tapes for the email server are reused after six months, and the only copies which exist are those the user stored locally.

There is a massive technical error in that report. It claims the "the IRS had a policy of backing up the data on its email server (which runs Microsoft Outlook) every day". But MS Outlook is an email CLIENT and doesn;t run on servers. Either the witness or reporter is incompetent.

Backups are destroyed after six months? Only 500MB allocated for each user's account? This archival procedure is nothing short of willfully negligent. In the finance industry, every email, chat, and text must be archived for 7 years, and not by the user on his local hard drive (where the user usually has the power to selectively delete any file).

Yes, Sorbanes-Oxley requires biz to retail these records, and Federal law requires it's agencies to retain such record throug the Nataional Archives (NARA) record retention policy.

The explanation for the missing emails of course is that Lois Lerner's hard drive crashed in mid-2011, and that the data was unrecoverable

That's the claim, but it's utter nonsense. A copy on her local drive is not a backup. No IT department, even in the 3rd world, depends on local drives.

Color me skeptical (hey, it's a skeptic's forum). Not that the hard drive crashed, but that this is the primary reason for the failure to produce the subpoenaed emails.

It's less plausible than "The dog ate my homework".

Coupled with the obvious footdragging on completing an examination, and the very delayed revelation to congrss that the emails are missing - I smell a coverup.


Anyway, this is a remarkable development. Just to show you how on top of this story the mainstream news media is, I did a google search: "lois lerner emails site:nytimes.com" and came up with two brief articles, one a paragraph from a blog that nobody reads, and the other a brief article which appeared today on page A17 of the print version (it was online yesterday).

I'm not really sure which is more important: the likelihood that the IRS is engaged in a cover-up of damaging emails involving Lois Lerner; or that the IRS had an archival policy which was clearly designed to reduce transparency and impede internal investigations.[/QUOTE]

Yes, you hardly need the concept of a private coverup when the press colludes.

Who? What? An agency on government-funding didn't have the resources available to a commercial company?

Ridiculous !

The IRS manages and collates and records financial records for ~100million workers, and co-ordinates w/ the records of another 200mill dependents and all US corporations, LLCs partnerships and hundreds of millions, perhaps a billion 1099s.

Your suggestion that they don't have the IT resources to manage biz-records for their 90k employees is perfectly ridiculous. So you are an apologist for the 'dog ate my homework' position ?

Or, that the agency lacked funding to develop and maintain an email system that would be standard in the private sector.

Fail - it's required by federal law that the IRS retain these records, and the IRS has a $1.8B budget (2013) for information systems. There is no lack of money to accomplish required record-keeping in their $11.9B total budget.

=======


I smell special prosecutor, tho' I expect Dems to continue to foot-drag and block the efforts.

I'm willing to entertain the idea that IRS IT is massively incompetent, however this claim requires real independently verified evidence, and many heads should roll. THere is no excuse for keeping the loss of evidence fo rcongress - someone should go to jail for perjury.

It seems the IRS "amnesia virus" has spread from Lerner's drive to destroy the email evidence of several other IRS employee involved in this "targetting" issue. Just amazing. I keep better personal email records than the IRS.

======

The National Archives (NARA) is responsible for long term retention. Agencies are responsible for transferring emails to then at 90 days according to the CAPSTONE policy. There are various requirement for email retention but currently.... top decision makers emails require permanent storage; all other employees for 7 years.

Email Management
Federal agencies are required to manage their email records in accordance with the Federal Records Act and 36 CFR Chapter XII Sub-chapter B.

http://www.archives.gov/records-mgmt/email-mgmt.html#bulletin2013-02
http://www.archives.gov/records-mgm...mple-agency-user-guide-for-managing-email.pdf


I expect someone is guilty of destruction of evidence.

 

[Ziggurat]

Wow, you really don't understand these things, do you? Let's go step by step, and please read slowly. I know this is complicated.

Lois Lerner contacted her IT people, who put their best person on it. She failed to recover any data, and so they sent the drive to "CI Forensics". Do you understand what those people do? They run data recovery software against a hard drive to retrieve any files at all that are not corrupted. Do you know what form those files come back in? You don't get back a complete operating system. First, the software attempts to preserve folder names, but if that doesn't work, it merely dumps out any file it can find that isn't corrupted. The files are then transferred to another storage format and that format is returned to the user. Then it's up to the user to wade through the files to try and find the really important ones.

They reported back that NONE of the files, and NONE of the disk was recoverable. They disposed of the hard drive and gave nothing to Lerner. That is as dispositive as you can get here. There were no files -- email or otherwise -- recoverable, so Lerner got nothing back, just an email telling her that her entire drive was gone.

I understand perfectly. You, however, do not. You have a severe reading comprehension problem. The issue is not whether Lerner was able to recover any files. The issue is that we have no record of what files were actually lost, because we have no record of what was on that hard drive. You claimed that we did, but we do not. The only contemporaneous description that we have of what was on that drive is "personal files".

So here we are again and in the face of obvious and completely dispositive evidence (THE ENTIRE HARD DRIVE WAS UNRECOVERABLE) you insist that just because it didn't reference emails (which were backed up on tape, remember?) that a completely corrupted hard drive that was sent to the "hard drive cemetery", and most importantly, WAS NEVER RETURNED TO LERNER, is somehow still suspicious in regards to a phony scandal that was trumped up a year later?

I said nothing about whether this was suspicious or not. That is entirely a fiction of your own imagination. What I said was that your description was false. And despite your bloviating, it was false. This has nothing to do with how unrecoverable the hard drive was. This has everything to do with you falsely claiming that we have contemporanous records that she lost email. We do not. We only have contemporaneous records that she lost her hard drive with "personal files". I was explicit from the start that this doesn't prove that there were not emails on the hard drive, only that we don't have proof that there were.

 

[DavidJames]

I find this kind of unhelpful comment to be extremely annoying. If you have something you want to say, put it out there for scrutiny. Don't hide behind a sarcastic quip.

And, no, I'm not an IT professional, but for a layman, I think I have a pretty good understanding of computer technology. The computer I'm using is DIY by the way.

I am an IT professional. I've managed teams who's job it is to manage backups for customers in large enterprises. In fact, peers of my teams have done exactly that for the federal government.

Failed backups and unrecoverable data is not as uncommon as one might think. It's no different whether it's private or public company.

I've been doing this for decades and it would be absolutely wrong to make any assumptions about what happened or the environment in which it happened, without knowing it's intimate details. Frequency of backups, reliability, retention details, etc. etc. vary. It's 100% wrong to assume government IT systems are best in class. Making such assumptions, including the infallibility of the U.S. government is exactly the technique used by conspiracy theorists to concoct their crazy theories.

 

[sunmaster14]

Good storage is still flipping expensive. Backing up good storage is also flipping expensive. Depending on backup strategy and how available the backup needs to be the storage requirements for a comparatively modest project can run into millions of pounds. Scale that up to a huge organisation and the costs are in the hundreds of millions.

By good storage, I imagine you mean extremely robust, quick to access storage. The kind of stuff that a financial institution would need. I don't think throwing stuff on digital tape and keeping the tapes in a warehouse is terribly expensive. Of course you need a staff of IT people, etc. But the marginal cost of extra storage is pretty small. It sounds like the IRS was actually archiving everything on tape, but then recycling tapes to save money? Sounds penny wise and pound foolish to me. This paper (warning: big file) discusses trends in digital storage. From the graph on p. 8, it looks like since 2010 the cost of digital tape has been less than $50/TB. That's $50 per 2,000 people for email, assuming everybody was at their email allocation limits.

 

[elbe]

I understand perfectly. You, however, do not. You have a severe reading comprehension problem. The issue is not whether Lerner was able to recover any files. The issue is that we have no record of what files were actually lost, because we have no record of what was on that hard drive. You claimed that we did, but we do not. The only contemporaneous description that we have of what was on that drive is "personal files".

Interestingly because it's a individual workstation's drive there is likely no guarantee that any data could be provided if the drive hadn't died. IT is responsible for backing up server data, which is why important files should be stored on file servers and emails on email servers, not workstation drives. While an organization can have policies on what users can and cannot store and/or delete on their drive (usually using security policies and such), data those drives should be considered ephemeral: useful data might be on them if audited, but if it's important it shouldn't have to be audited.

If you want to complain that 6 months is too short, be my guest (though it's still not wildly short), but that a workstation drive didn't survive is a pretty stupid complaint to make. Heck, they could have just upgraded or reimaged her machine at some point which could have also lost the relevant data.

 

[elbe]

By good storage, I imagine you mean extremely robust, quick to access storage. The kind of stuff that a financial institution would need. I don't think throwing stuff on digital tape and keeping the tapes in a warehouse is terribly expensive. Of course you need a staff of IT people, etc. But the marginal cost of extra storage is pretty small. It sounds like the IRS was actually archiving everything on tape, but then recycling tapes to save money? Sounds penny wise and pound foolish to me. This paper (warning: big file) discusses trends in digital storage. From the graph on p. 8, it looks like since 2010 the cost of digital tape has been less than $50/TB. That's $50 per 2,000 people for email, assuming everybody was at their email allocation limits.

Good storage can depend on the needs of the organization and the data.

Tape storage isn't exactly cheap either. They need to be stored offsite in a controlled environment. We used to use Iron Mountain. You then have to arrange transport for your regular backups between the data centers and the storage, inventory your tapes obviously, and make sure you have enough tapes going forward (generally pretty straightforward but still has costs). You also can't tell when a tape will just go bad, especially if you are reusing expired tapes to save costs. Remember, you can't just look at the front cost of a solution and ignore all the extra costs just to get it working the way you want.

 

[Unabogie]

I understand perfectly. You, however, do not. You have a severe reading comprehension problem. The issue is not whether Lerner was able to recover any files. The issue is that we have no record of what files were actually lost, because we have no record of what was on that hard drive. You claimed that we did, but we do not. The only contemporaneous description that we have of what was on that drive is "personal files".



I said nothing about whether this was suspicious or not. That is entirely a fiction of your own imagination. What I said was that your description was false. And despite your bloviating, it was false. This has nothing to do with how unrecoverable the hard drive was. This has everything to do with you falsely claiming that we have contemporanous records that she lost email. We do not. We only have contemporaneous records that she lost her hard drive with "personal files". I was explicit from the start that this doesn't prove that there were not emails on the hard drive, only that we don't have proof that there were.

For the love of god, the entire hard drive was unrecoverable. She lost the entire drive, and it wasn't returned to her, and no files were recovered, at all. It's like talking to a brick wall.

ETA: And this is such a simple concept that I sincerely hope the ignorance is willful and not congenital.

The emails that the witch hunters are asking for are from 2011. IRS retention policy mandates emails on the server side be kept for six months, and then the tapes are re-used for newer emails. This is obviously to keep costs down, which Republicans normally claim is something they think is important. However, this means that there are no tape-backed up emails from 2011, something the Republicans obviously knew, so they demanded emails from her personal computer, to which she replied that her hard drive crashed and she has no files at all from 2011. In support of this, she provided iron clad proof that her entire hard drive was corrupted and subsequently disposed of. To which the Republicans responded "COVER UP"!!!! In that context, how in the world is that email exchange not a 100% refutation of the accusation against her that she's hiding emails from 2011? You know, emails that would only be found on a hard drive that was thrown away after being found 100% unrecoverable?

 

[sunmaster14]

For the love of god, the entire hard drive was unrecoverable. She lost the entire drive, and it wasn't returned to her, and no files were recovered, at all. It's like talking to a brick wall.

I don't think you've responded to my point that the IRS is using her hard drive crash from 2011 as an excuse to not provide emails which may still exist somewhere.

Her emails shouldn't have only been stored locally, but even if that is the case, we don't know that they were stored only (or at all) on the drive that crashed. She could have had other local storage which did not crash or has not been erased.

 

[sunmaster14]

Interestingly because it's a individual workstation's drive there is likely no guarantee that any data could be provided if the drive hadn't died. IT is responsible for backing up server data, which is why important files should be stored on file servers and emails on email servers, not workstation drives. While an organization can have policies on what users can and cannot store and/or delete on their drive (usually using security policies and such), data those drives should be considered ephemeral: useful data might be on them if audited, but if it's important it shouldn't have to be audited.

Agreed. catsmate1 seems to disagree. I would be interested to have him weigh in.

 

[elbe]

Agreed. catsmate1 seems to disagree. I would be interested to have him weigh in.

While I can't speak for catsmate1, I think the point is that they may have gone beyond their expected duty looking for copies that exist outside the official backups. One such avenue, locally cached emails, were obviously lost.

Earlier this year we were apparently involved in some legal dispute (again, I'm not in legal so I only get the snippets that may involve my team) about something we did for a client in 2011. The emails about the work were lost to the ravages of time/exchange server policy so I tried digging through as many possible sources as I could to see if we had anything about the job. The best I could do was find a record that we did do the work for them in a general sense but could not provide them any further details. The most I really needed to tell them was that it was too long ago for us to have that information anymore but I tried to find what I could to be helpful.

 

[Unabogie]

I don't think you've responded to my point that the IRS is using her hard drive crash from 2011 as an excuse to not provide emails which may still exist somewhere.

Her emails shouldn't have only been stored locally, but even if that is the case, we don't know that they were stored only (or at all) on the drive that crashed. She could have had other local storage which did not crash or has not been erased.

They had backups. The server backed up emails and stored them on tape drives. But the retention policy mandated they re-use tape after six months. While you can object to that policy, that's not what's at play here. The accusation is that something nefarious happened, and that's been conclusively shown to be false, since the only other place where those emails exist are to the people she sent them to, is THOSE people kept the emails. And since this is just a fishing expedition, they don't even know who to ask to see if they have any emails from Lerner. They don't even have a specific email in mind. They just want all the emails from 2009-2011 so they can troll them for something, anything, to show that Obama is bad. And when they heard back that emails from 2011 aren't around, they cynically trumped up another scandal to pretend like this was something unusual.

 

[The Don]

By good storage, I imagine you mean extremely robust, quick to access storage. The kind of stuff that a financial institution would need. I don't think throwing stuff on digital tape and keeping the tapes in a warehouse is terribly expensive. Of course you need a staff of IT people, etc. But the marginal cost of extra storage is pretty small. It sounds like the IRS was actually archiving everything on tape, but then recycling tapes to save money? Sounds penny wise and pound foolish to me. This paper (warning: big file) discusses trends in digital storage. From the graph on p. 8, it looks like since 2010 the cost of digital tape has been less than $50/TB. That's $50 per 2,000 people for email, assuming everybody was at their email allocation limits.

There's a huge difference between a $50 tape and what you have to pay for the managed service which looks after and stores these tapes. Even with a six month re-use policy there will be 200 sets of tapes (each set containing perhaps tens or hundreds of tapes) and then these sets of tapes will have to be stored (and periodically verified). So although there may only be $1m worth of tapes, the service will cost a heck of a lot.

I'm surprised that they didn't have a scheme where weekly backups are retained for longer and monthly backups are retained indefinitely but as Catsmate1 points out, after a while the backups may be incompatible with the restore mechanism.

 

[Ziggurat]

For the love of god, the entire hard drive was unrecoverable. She lost the entire drive, and it wasn't returned to her, and no files were recovered, at all. It's like talking to a brick wall.

For the love of god, the issue has never been about how much was recoverable. I know the entire hard drive was lost. I never suggested otherwise. That has absolutely nothing to do with what I'm saying.

You claimed that there were contemporaneous records that email was lost. There are no such records. There are no records at all of what was on that hard drive, except "personal files". If there are no records of email being on the drive, then there can be no records of email being lost when the hard drive crashed. I never once suggested that emails had been recovered.

Your reading comprehension failures continue apace.

 

[Newtons Bit]

Good storage is still flipping expensive. Backing up good storage is also flipping expensive. Depending on backup strategy and how available the backup needs to be the storage requirements for a comparatively modest project can run into millions of pounds. Scale that up to a huge organisation and the costs are in the hundreds of millions.

Uh, what? Storage is ridiculously cheap. Cloud storage options are in the 0.02$/GB/Month. I talked to a buddy of mine who does Llarge IT projects for universities, and he estimated the ce cost upgrading the IRS email system to a 2GB 7 year storage is in the ~$1million range.

 

[ravdin]

Larger issue: we're talking about this in the first place because we've forgotten what the purpose of taxes should be. Taxes should be for raising revenue for the government, nothing more. There's no reason it should be a complicated, burdensome process to pay taxes. Using taxes as a carrot and stick for political purposes (in the form of penalties or subsidies for industries or companies in favor with politicians) is corruption.

It shouldn't surprise anyone if the IRS is overtly using audits to punish political groups that aren't in favor with the administration. If you're OK with targeting the Tea Party, think about what happens when the other party is in power and sets the ACLU or SPLC or even the JREF in its sights.

The larger point to me is that if the IRS simply collected revenue and it was easy for us to figure out what we owe, no one would give a crap about what was in Lois Lerner's emails.