AA77 FDR Data, Explained

[Dave Rogers]

The variables do not change. The code does not change. The operators associated with the variables do not change. The only dynamic data is provided by the raw file.

If that's true, then how can the compass deviation be included in the animation?

Dave

 

[beachnut]

By the way, I've only seen the CSV file, which starts just before takeoff on 9/11. Has anyone seen the rest of the 24 hours recorded by the FDR for this aircraft? If not, has anyone asked for it?

p4t have all the data and it does show all the previous flight for the airframe, and they fail to explain this data is proof this is the one and only Flight 77 aircraft.

 

[Worm]

I'm baffled by this whole 'software doesn't mess with CSV' argument.

(FWIW - I'm not a professional programmer, but I do have a degree in Computer Science and nearly thirty years of IT experience, if it matters...)

Software can, as you would expect, do pretty much anything with CSV data, or any other data. It depends on the software, how it is coded, set up, tested, and used. Either intentionally, or through error, all sorts of things can happen to the data from input to output. None of it has to be static, it can be as dynamic as you like, that's kind of the whole point in many cases.

Data is just data, you can do what you like with it. As has been stated here, if you want to talk about the FDR data, then you need to look at the FDR data, looking at anything else is allowing for changes to have been made, changes which may be unpredictable, inconsistent or erroneous. CSV is not some holy format that isn't subject to the vagaries of data manipulation, it's just data - no better or worse than any other data, and any attempt to say otherwise is simply wrong.

 

[Reheat]

Data is just data, you can do what you like with it. As has been stated here, if you want to talk about the FDR data, then you need to look at the FDR data, looking at anything else is allowing for changes to have been made, changes which may be unpredictable, inconsistent or erroneous. CSV is not some holy format that isn't subject to the vagaries of data manipulation, it's just data - no better or worse than any other data, and any attempt to say otherwise is simply wrong.

Yep, but you see there are some people that like to look up an elephant's butt with one eyeball and not notice that they can't see the entire elephant!

 

[grmcdorman]

I'm baffled by this whole 'software doesn't mess with CSV' argument.

(FWIW - I'm not a professional programmer, but I do have a degree in Computer Science and nearly thirty years of IT experience, if it matters...)

Software can, as you would expect, do pretty much anything with CSV data, or any other data. It depends on the software, how it is coded, set up, tested, and used. Either intentionally, or through error, all sorts of things can happen to the data from input to output. None of it has to be static, it can be as dynamic as you like, that's kind of the whole point in many cases.

Data is just data, you can do what you like with it. As has been stated here, if you want to talk about the FDR data, then you need to look at the FDR data, looking at anything else is allowing for changes to have been made, changes which may be unpredictable, inconsistent or erroneous. CSV is not some holy format that isn't subject to the vagaries of data manipulation, it's just data - no better or worse than any other data, and any attempt to say otherwise is simply wrong.

I am a professional programmer (also with a CS degree), and I have to say I agree with you.

In particular, there is of necessity some sort of transform going on here: the FDR data is not the same as the CSV data (at the very least, it's binary). Some of the CSV values may even be derived from multiple values in the FDR, meaning - if it's floating point - the whole issue of FP behaviour gets involved.

And then when it gets loaded into the animation software, there would almost certainly be adjustment settings that can be applied - specifically for instrument error in addition to things like mag heading v.s. true and local pressure altitude.

Not to mention the intrinsic issues with the FDR data; for example, Beachnut has mentioned that some values are only accurate to something like 0.2 or 0.5 nautical miles (and PFT tries to read it as if it's accurate to 0.1).

 

[DGM]

Correct me if I'm wrong but, wouldn't the data be more focused (and the analysis) on what the plane was doing. The FDR is used to determine what went wrong with the plane not where the plane actually was. Sure altitude would be important but, where the plane ended up is a no brainier, It's where they found the box they got the data from.

Call me simple but, if they could fake everything the CT's think they can, why would they release an animation that's obviously (to idiots) wrong.

The logic of a "truther" is baffling to me.

 

[jaydeehess]

Correct me if I'm wrong but, wouldn't the data be more focused (and the analysis) on what the plane was doing. The FDR is used to determine what went wrong with the plane not where the plane actually was. Sure altitude would be important but, where the plane ended up is a no brainier, It's where they found the box they got the data from.

Call me simple but, if they could fake everything the CT's think they can, why would they release an animation that's obviously (to idiots) wrong.

The logic of a "truther" is baffling to me.

In post 3494 I wrote;

the NTSB has no reason to come up with an accurate depiction of the last moments of Flt 77. It hit a big building killing all aboard, the FDR shows no problem with any system, there is ample evidence to show it was hijacked, the hijackers deliberatly flew it into the building.

The NTSB is tasked with determining the cause of aircraft mishaps. This one was caused by the person at the controls. It was not caused by weather conditions, mechanical or electrical/electronic failure, or pilot error.

Pray tell, what reason would the NTSB then have to produce an accurate animation showing that the plane hit a building that dozens of eyewitnesses say hit the building?

I have yet to see a reply to the question posed in the bolded part although I admit to having been reluctant to read this thread recently with all the off topic bickering. Perhaps trib or Scott did post something that would be an answer to my question and I have not seen it.

 

[wstutt]

Earlier on this thread there was debate as to whether the data recorded by the flash memory of the FDR may have become corrupted during the crash.

I have written a program to check the Hamming codes and page parities in the AAL77 FDR file. There are only two pages where the Hamming code and page parity shows as being incorrect. It appears to me that this is because these pages were not completely recorded. You can read about it at my web site at warrenstutt dot com. Follow the AAL77 Hamming code and page parity checker link.

I see no evidence that any of the data was corrupted after being recorded.

Warren.

 

[Myriad]

Here is the link: http://www.warrenstutt.com/
AAK77 Hanning Code page: http://www.warrenstutt.com/AAL77CheckHammingCodesAndPageParities/index.html

From the web link:

Incorrect Hamming codes and page parities indicate data that has either not been recorded correctly or not recorded completely or has been corrupted after it has been recorded.

From the OP:

There are only two pages where the Hamming code and page parity shows as being incorrect. It appears to me that this is because these pages were not completely recorded.

How do you determine whether an incorrect code or parity indicates an incompletely recorded page rather than data corrupted after being recorded?

Why would there be two incompletely recorded pages, if no data were corrupted? Nothing I've read about this FDR system suggests that, in normal operation, the recorder would begin recording a page before the previous page is completely recorded.

Welcome to the forum, Warren!

Respectfully,
Myriad

 

[T.A.M.]

Earlier on this thread there was debate as to whether the data recorded by the flash memory of the FDR may have become corrupted during the crash.

I have written a program to check the Hamming codes and page parities in the AAL77 FDR file. There are only two pages where the Hamming code and page parity shows as being incorrect. It appears to me that this is because these pages were not completely recorded. You can read about it at my web site at warrenstutt dot com. Follow the AAL77 Hamming code and page parity checker link.

I see no evidence that any of the data was corrupted after being recorded.

Warren.

Totally out of my realm of expertise.

However, thank you for taking the time to put this in the appropriate thread, rather than starting a new one.

Welcome to the forum.

TAM

 

[beachnut]

Earlier on this thread there was debate as to whether the data recorded by the flash memory of the FDR may have become corrupted during the crash.

...
I see no evidence that any of the data was corrupted after being recorded.

Warren.

What does this mean relative to the fact Flight 77 impacted the Pentagon?

There are only two pages where the Hamming code and page parity shows as being incorrect. It appears to me that this is because these pages were not completely recorded.

Where in the DATA? How many seconds are on a page of data?

 

[EvilBiker]

What does this mean relative to the fact Flight 77 impacted the Pentagon?

Where in the DATA? How many seconds are on a page of data?

I would add to that: how many pages of data are we talking about here? What % of data loss do you envisage?

 

[R.Mackey]

Earlier on this thread there was debate as to whether the data recorded by the flash memory of the FDR may have become corrupted during the crash.

I have written a program to check the Hamming codes and page parities in the AAL77 FDR file. There are only two pages where the Hamming code and page parity shows as being incorrect. It appears to me that this is because these pages were not completely recorded. You can read about it at my web site at warrenstutt dot com. Follow the AAL77 Hamming code and page parity checker link.

I see no evidence that any of the data was corrupted after being recorded.

Warren.

Welcome to the Forum. Never thought I'd see this thread resurface again...

One thing to keep in mind, as I've mentioned on and off in the bulk of this discussion, is that we do not have the raw data. What we have is a file extracted by NTSB (if I recall correctly), and the process of generating that file may fix or ignore a lot of data errors. In fact, the NTSB report talks about a very large number of measurements that didn't conform to the standard, so if you had a bit-perfect data dump, you should expect to see all kinds of data errors, not just two.

The FDR does not have a file system. To do this properly, we would need access to the FDR flash itself, or to a copy of the image. As far as I know, neither has ever been made available.

If this was available, I expect we would find all kinds of data errors at the end of the record -- errors that are simply ignored and dumped in the process that created our file. But I have no way be certain. Could be there was some other interruption, could even be there's a flaw in the FDR. Whatever the reason, we know the last few seconds are simply not there from correlation to the radar data.

 

[apathoid]

Earlier on this thread there was debate as to whether the data recorded by the flash memory of the FDR may have become corrupted during the crash.

I have written a program to check the Hamming codes and page parities in the AAL77 FDR file. There are only two pages where the Hamming code and page parity shows as being incorrect. It appears to me that this is because these pages were not completely recorded. You can read about it at my web site at warrenstutt dot com. Follow the AAL77 Hamming code and page parity checker link.

I see no evidence that any of the data was corrupted after being recorded.

Warren.

Hi Warren, very interesting find. Have you considered going after the raw FDR data and analyzing it?

 

[wstutt]

How do you determine whether an incorrect code or parity indicates an incompletely recorded page rather than data corrupted after being recorded?

The reason why I believe these two pages are incompletely recorded is that the Hamming code and page parity which are the last bits in these pages along with more than 100 bits prior to these are set to 1 which is what all bits are set to when pages in the EEPROM are erased. In each of these two pages it appears to me that recording of the data was stopped part way through each page and the Hamming code and page parity were not recorded at the end of each page, leaving those bits as 1.

Also, if you look at the line that starts with CURINDEX in the text header of the FDR file, the numbers can be used to calculate file offsets of the next pages after the two pages showing incorrect Hamming code and page parities. It would appear that the program that produced the .FDR file was able to determine the next page to be written for each of the two flight data streams. I can give details of the calculations if you like.

Why would there be two incompletely recorded pages, if no data were corrupted? Nothing I've read about this FDR system suggests that, in normal operation, the recorder would begin recording a page before the previous page is completely recorded.

This is actually an interesting question. If you look at figure 4.2-2 of the PDF document linked to on my AAL77 Hamming code page that you provided a link to in your post, you will see that it shows two flight data streams numbered 0 and 1. Some data is recorded in each one. I don't know why there are two data streams instead of one. One of the two pages is in one flight data stream and the other page is in the other flight data stream.

I believe the Hamming code and page parity only gets written when the page is full, so there can be a page in each flight data stream that has had some data written to it but is not yet full.

Also note that every following page in each 64 kilobyte block containing each of these two pages as well as the entire next 64 kilobyte block for each of their respective data streams are erased.

Welcome to the forum, Warren!

Respectfully,
Myriad

Thanks Myriad, T.A.M. and R.Mackey and anyone else I may have missed for your welcomes.

I will endeavour to answer your questions although it may take me a day or two.

Warren.

 

[wstutt]

What does this mean relative to the fact Flight 77 impacted the Pentagon?

On the surface it means nothing. However, it has been suggested on this thread that the reason why the NTSB CSV file shows a high altitude at the end of the data is that there is data missing and it was further suggested that the missing data is due to the final few seconds of data in the FDR being corrupted during the crash. I do not see any evidence of the FDR data having been corrupted.

Where in the DATA?

The two 128 byte pages of data that show incorrect Hamming codes and page parities start at hex file offsets 38972A and 39952A respectively in the .FDR file

How many seconds are on a page of data?

There is no exact answer to this question since the frames which each contain four seconds of data are compressed using Huffman coding so the number of bits used to store each frame varies from frame to frame. Averaging from the last ten complete looking frames from each flight data stream, I get a figure of approximately 0.5 seconds of data per full page.

Warren.

 

[wstutt]

I would add to that: how many pages of data are we talking about here?

There are 187092 flight data pages that are filled (not erased) and that have the correct Hamming code and page parity and 2 such pages that have an incorrect Hamming code and page parity.

What % of data loss do you envisage?

I see no evidence of data being corrupted. Does this answer your question? I am not sure I have correctly understood what you meant.

Warren.

 

[lapman]

There are 187092 flight data pages that are filled (not erased) and that have the correct Hamming code and page parity and 2 such pages that have an incorrect Hamming code and page parity.

I see no evidence of data being corrupted. Does this answer your question? I am not sure I have correctly understood what you meant.

Warren.

Would the software skip over the corrupted bytes and use a default value which caused the Hamming and parity errors?

 

[Myriad]

This is actually an interesting question. If you look at figure 4.2-2 of the PDF document linked to on my AAL77 Hamming code page that you provided a link to in your post, you will see that it shows two flight data streams numbered 0 and 1. Some data is recorded in each one. I don't know why there are two data streams instead of one. One of the two pages is in one flight data stream and the other page is in the other flight data stream.

Well, one obvious possibility is if the overall data acquisition rate were about twice what a single flash ram bank would be able to keep up with. But that's just speculation on my part.

Thanks for the answers. I hate to stack up questions while you're still working on others, but it appears you're saying that the NTSB file really does resolve into a complete bit image of the contents of the FDR's memory, not just a decoding or translation of it. Am I reading that correctly?

Respectfully,
Myriad

 

[wstutt]

Welcome to the Forum. ...

Thank you.

One thing to keep in mind, as I've mentioned on and off in the bulk of this discussion, is that we do not have the raw data. What we have is a file extracted by NTSB (if I recall correctly), and the process of generating that file may fix or ignore a lot of data errors. In fact, the NTSB report talks about a very large number of measurements that didn't conform to the standard, so if you had a bit-perfect data dump, you should expect to see all kinds of data errors, not just two.

I believe the .FDR file basically is a data dump when you remove the text header from the beginning of it. The 64 kilobyte pages are laid out as in figure 4.2-2 of the PDF document on my web page that I referred to in my reply to Myriad. There are two flight data streams in the .FDR file as in the same figure.

When you refer to the file extracted by the NTSB do you mean the .CSV file?

The NTSB report says that "The remaining parameters either were not recorded properly or were not confirmed to have been recorded properly". I do not interpret this as definitely saying that some parameters were found to not be recorded properly, it could be that the NTSB did not try to confirm that any of the parameters that they did not validate were recorded properly.

In any case, the Hamming code and page parity only gives an indication of data being corrupted after it has been recorded. If something happens to parameter before it is recorded for example a faulty sensor sending invalid data, the Hamming code and page parity checks will not pick that up.

At this time, I am unable to check the parameters in the .FDR file for validity since I am unable to decompress the frames in it.

The FDR does not have a file system. To do this properly, we would need access to the FDR flash itself, or to a copy of the image. As far as I know, neither has ever been made available.

I believe the .FDR file is close to being a copy of the image. It is available through the links on my web page. I wouldn't call the structure of the FDR memory storage a file system as such, but there is a structure. Refer to the figure referenced above.

If this was available, I expect we would find all kinds of data errors at the end of the record -- errors that are simply ignored and dumped in the process that created our file. But I have no way be certain. Could be there was some other interruption, could even be there's a flaw in the FDR. Whatever the reason, we know the last few seconds are simply not there from correlation to the radar data.

The only data errors in the FDR that the Hamming code and page parity is designed to detect is corruption of the data after it has been recorded.

Warren.