On Negative Results

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

On Negative Results

Tom Johnson (via Google Docs)
Originally found on TechnologyReview.com
See http://archives.trblogs.com/2005/08/on_negative_res.trml?trk=nl 

On Negative Results
Posted by David Appell at August 30, 2005 08:48 AM in Biotechnology
and Health Care.

"There's a very interesting article by John Ioannidis in PLoS
Medicine, the free online journal. Most current published research
findings might well be false, he says. There are several factors, and
I think it's worth presenting them in detail:

    1. Many research studies are small, with only a few dozen participants.

    2. In many scientific fields, the "effect sizes" (a measure of how
much a risk factor such as smoking increases a person's risk of
disease, or how much a treatment is likely to improve a disease) are
small. Research findings are more likely true in scientific fields
with large effects, such as the impact of smoking on cancer, than in
scientific fields where postulated effects are small, such as genetic
risk factors for diseases where many different genes are involved in
causation. If the effect sizes are very small in a particular field,
says Ioannidis, it is "likely to be plagued by almost ubiquitous false
positive claims.

    3. Financial and other interests and prejudices can also lead to
untrue results.

    4. "The hotter a scientific field (with more scientific teams
involved), the less likely the research findings are to be true,"
which may explain why we sometimes see "major excitement followed
rapidly by severe disappointments in fields that draw wide attention."

"This ought to be an eye-opener.... The solution? More publication of
preliminary findings, negative studies (which often suffer that fate
of the file-drawer effect), confirmations, and refutations. PLoS says,
"the editors encourage authors to discuss biases, study limitations,
and potential confounding factors. We acknowledge that most studies
published should be viewed as hypothesis-generating, rather than
conclusive." And maybe this will temper journalists' tendency to offer
every new study as the Next Big Thing."

--tj
--
==============================================
J. T. Johnson
Institute for Analytic Journalism
www.analyticjournalism.com
505.577.6482(c)                          415.775.2530(h)
http://www.jtjohnson.com               [hidden email]

"He who refuses to do arithmetic is doomed to talk nonsense."
            -John McCarthy, Stanford University mathematician
==============================================