BLOGSCAN: Forensic Statistics

Several interesting points are raised in the newsletter of the American Association of Physicians and Surgeons (AAPS) in a post entitled "Forensic Statistics" in their July 2011 newsletter headlined "Numbers." Healthcare Renewal is cited:

Forensic Statistics

While claims from RCTs fail to replicate about 20% of the time, the problem with epidemiology is so bad as to constitute a crisis, writes S. Stanley Young (“Everything Is Dangerous: a Controversy,” National Institute of Statistical Sciences, June 2008, www.niss.org). Fewer than 20% of nonrandomized trials [e.g., observational studies - ed.] replicate; i.e. 80%-90% of epidemiologists’ claims are false.

More than $1 billion in grant/tax money flows to institutions with reproducibility problems, Young states. A fundamental flaw in their methodology is to ask multiple, often hundreds to thousands of questions, of the same data set. It’s like playing “maverick solitaire”: given 25 randomly selected cards from a deck of 52 playing cards, the probability of being able to arrange them into 5 “pat hands” (e.g. a full house) is 98%.

Since data miners are good at concealing their footsteps, critics need full access to the raw data and the code used for the statistical analysis—often not forthcoming.

The EHR software that is supposed to support all this “research” and to guide medical treatment also needs a forensic evaluation, writes Scot Silverstein, M.D., of Drexel University (see http://hcrenewal.blogspot.com). He cites such an evaluation of the Cerner FirstNet system used in New South Wales, Australia, done by Prof. Jon Patrick. The authoritarian implementation processes of the governmental HIT “support” staff were familiar to Silverstein, such as disenfranchising the clinical staff and failing to acknowledge the validity of complaints.

“Healthcare reform” demands acceptance because it claims to be based on science. But then, so did Communism.

The "maverick solitaire" data mining issues (to be used, no doubt, in future "comparative effectiveness research" based on EHR data) are a additional concern to those I raised in an essay "The Syndrome of Inappropriate Overconfidence in Computing: An Invasion of Medicine by the Information Technology Industry?" in the AAPS journal several years ago (PDF).

I can also add: where are the RCT's of EHR/CPOE systems?

-- SS