[Guest post by DRJ]

NASA released today its controversial National Aviation Operational Monitoring Service (NAOMS) report, a study of commercial airline safety that was previously deemed too disturbing to release because it would upset travelers and hurt airline profits. For several reasons, the Houston Chronicle was not impressed:

“NASA begrudgingly released some results today from an $11.3 million federal air safety study it previously withheld from the public over concerns it would upset travelers and hurt airline profits. It published the findings in a format that made it cumbersome for any thorough analysis by outsiders. Released on New Year’s Eve, the unprecedented research conducted over nearly four years relates to safety problems identified by some 29,000 pilots interviewed by telephone.

Earlier characterizations from people who have seen the results said they would show that events like near collisions and runway interference occur far more frequently than previously recognized. Such information could not be gleaned from the 16,208 pages posted by NASA on its Web site, however, because of information that was edited out. The data was based on interviews with about 8,000 pilots per year from 2001 until the end of 2004.

The NASA Web site shows formatted, printed reports that the space agency scrubbed to ensure none of the pilots who were interviewed and promised anonymity could potentially be identified. The data was posted as NASA officials began a telephone news conference, allowing no time to look at the material and ask them questions about it.

NASA did not provide documentation on how to use its data, nor did it provide keys to unlock the cryptic codes used in the dataset.

NASA Administrator Michael Griffin told reporters the agency typically releases information in Adobe System’s portable document format, known as pdf, which presents the information on formatted, printed pages. But there are dozens of reports available from NASA’s Web site about other subjects in Microsoft’s Excel data format, which would permit researchers to conduct a meaningful analysis more easily.

Griffin said NASA wanted to ensure that no one modified the survey results and circulated false data as NASA’s research product. He said even inexpensive optical character recognition software could convert the formatted reports. Such software can risk introducing errors in the data as it performs these conversions. “We’ve gone the extra mile with this data, and well beyond our original intentions,” Griffin said.

He dismissed suggestions that NASA chose to release the data late on New Year’s Eve, when the public is distracted by holidays and news organizations are thinly staffed. “We didn’t deliberately choose to release on the slowest news day of the year,” Griffin said.

NASA drew harsh criticism from Congress and news organizations for keeping the information secret. Rejecting an Associated Press request under the Freedom of Information Act, NASA explained that it did not want to undermine public confidence in the airlines or hurt airline fortunes. Griffin later overruled his staff and promised Congress that he would release at least some data by the end of the year.

NASA’s survey, the National Aviation Operations Monitoring System, was launched to see if a massive pilot survey would help pinpoint problems and prevent accidents. Survey planners said it was unique because it was a random survey, with an 80 percent response rate, that did not rely on pilots to take the initiative to report problems but rather reached out and interviewed them.

Griffin said NASA never intended to analyze the data it collected, but rather they planned on passing on its methodology to the aviation community. He said he had only looked at a few results, but that, “It’s hard for me… to see any data here that the traveling public would care about or ought to care about.” That would be up to others who chose to analyze the data, he said.

Pilots were asked how many times they encountered safety incidents in flight and on the ground, such as near-collisions, equipment failure, runway interference, trouble communicating with the tower and unruly passengers.”

Based on this article, it’s hard not to share the Chronicle’s disdain for NASA’s attitude. On what planet is it logical to believe the traveling public would not be interested in “near-collisions, equipment failure, runway interference, trouble communicating with the tower and unruly passengers”?

It seems others are concerned, too:

“Griffin outraged some NASA employees by saying the project had been poorly managed and its methodology not properly vetted. Survey experts who worked on it, however, said they used state-of-the-art industry techniques and carefully validated the results.

NASA’s handling of the matter prompted a congressional investigation and separate investigations by its inspector general and by a union representing NASA workers.

Jon Krosnick, a Stanford University professor who helped design the project for NASA, said the release of information was inadequate. “The data they released are intentionally designed to prevent people from analyzing the rates properly and are designed to entrap analysts into computing rates that are much higher than the survey really shows,” he said Monday.”

Here is the NASA NAOMS website that contains links to the study the Chronicle described as too “cumbersome for any thorough analysis by outsiders.”

This sounds like a job for the internet!

— DRJ