Friam

The "decline effect"

Classic

List

Threaded

13 messages Options

Nick Thompson

The "decline effect"

All—

Have others seen the article in the New Yorker on the “decline effect”, the alleged tendency for the effect sizes of well documented phenomena to decline with successive years of replication. I kept turning back to the front of the article to reassure myself that it was not one of the “Shouts and Murmurs” series. It is not. The passage that particularly caught my eye:

Many scientific theories continue to be considered true even after failing numerous experimental tests. … [This] holds for any number of phenomena, from the disappearing benefits of second-generation antipsychotics to the weak coupling ration exhibited by decaying neutrons, which appears to have fallen by more than ten standard deviations between 1969 and 2001. [NY mag, 15 december 2010, p57]

At least until recently, when the NY-er writes about science, they try very hard not to write anything stupid.

What gives?

Nicholas S. Thompson

Emeritus Professor of Psychology and Biology

Clark University

http://home.earthlink.net/~nickthompson/naturaldesigns/

http://www.cusf.org

============================================================
FRIAM Applied Complexity Group listserv
Meets Fridays 9a-11:30 at cafe at St. John's College
lectures, archives, unsubscribe, maps at http://www.friam.org

lrudolph

Re: The "decline effect"

On 12 Dec 2010 at 0:46, Nicholas Thompson wrote:

> At least until recently, when the NY-er writes about science, they try very
> hard not to write anything stupid.

What??? Have you forgotten the whole disgraceful
Paul Brodeur episode? Refresh your memory by reading
<http://bobpark.physics.umd.edu/WN10/wn121010.html>.
Worse than stupid, verging on criminal, given the
amount of money it's caused to be thrown away and
the amount of anxiety it's generated or caused to
be misplaced.

I haven't yet read the "decline effect" article, and
am not commenting on it, just on your quoted sentence
above.

============================================================
FRIAM Applied Complexity Group listserv
Meets Fridays 9a-11:30 at cafe at St. John's College
lectures, archives, unsubscribe, maps at http://www.friam.org

Pamela McCorduck

Re: The "decline effect"

The NYer can be obtuse about scientific topics, but this article intrigued me. Is the decline effect real?

It's certainly the case that many medical practitioners follow outdated advice. And the use of statistics in medicine (to be sure, a special subset of science) can be awkward. I keep asking people: if I've lowered my chances by twenty percent of contracting a certain cancer by doing thus-and-so, and I find four other thus-and-so's to also do, does that mean I'll never get that cancer? No one can answer.

On Dec 12, 2010, at 5:58 AM, [hidden email] wrote:

On 12 Dec 2010 at 0:46, Nicholas Thompson wrote:

At least until recently, when the NY-er writes about science, they try very
hard not to write anything stupid.

What??? Have you forgotten the whole disgraceful
Paul Brodeur episode? Refresh your memory by reading
<http://bobpark.physics.umd.edu/WN10/wn121010.html>.
Worse than stupid, verging on criminal, given the
amount of money it's caused to be thrown away and
the amount of anxiety it's generated or caused to
be misplaced.

I haven't yet read the "decline effect" article, and
am not commenting on it, just on your quoted sentence
above.

============================================================
FRIAM Applied Complexity Group listserv
Meets Fridays 9a-11:30 at cafe at St. John's College
lectures, archives, unsubscribe, maps at http://www.friam.org

"If you're away from Broadway, you're only camping out."

				Thomas E. Dewey

Eric Charles

Re: The "decline effect" and the RTQ method

In reply to this post by Nick Thompson

Nick, et al.,
I kept flirting with the idea of writing an article for the Journal of Irreproducible Results about the RTQ method of improving test scores for underachieving students. RTQ stands for Re-Test Quickly. The method is well proven. To try it: 1) Create a set of matched multiple choice tests on general relativity. 2) Give it to a group of 2nd graders. 2) Identify the lowest quartile of the class. 3) Immediately retest them with another version. 4) You will find that the students knowledge of general relativity has significantly increased. *Warning, do not try to apply this method with already well performing students.*

The New Yorker might merely be noticing that several scientists fail to heed the warning.

On a more serious note (and the previous part was fairly serious already): Given that half the "major discoveries" promoted in psychology are assuredly garbage, how does this surprise you? Are you a "hard-science" snob, and only surprised because this is happening to physicists? There are a million reasons why an initial report of a phenomenon might overestimate the effect size. Some reasons are malicious (i.e., drug company funded studies as to the effectiveness of new drugs), others are benign (i.e. sampling error, unforeseen methodological shortcomings in initial tests, biased acceptance and promotion of "sexy" results).

Whole academic industries arise over non-existent effects: Piaget's "A-non-B error", menstrual synchrony, and infant's "innate mathematical abilities." Once the discipline is formed, it is very hard to unform.

So, if the NYer is being stupid, it is being stupid either for 1) not understanding what is going on, 2) not recognizing the legitimacy of what is going on, or 3) being selective in reporting by not noticing that some effects raise over time. I suspect a combination of all of those, with #3 being the most damning from a journalistic perspective.

Eric

P.S. It is also possible that the effect sizes are legitimately changing over time. Lets be honest, doesn't almost everything seem a little less important now than it used to? I mean, just a month or two ago backscatter technology and forced groping seemed like a big deal... and how many people's lives are currently being endangered this week by WikiLeakes... what about Obama's Hope and Change effect... or the way the Republicans would fix Washington... talk about a pervasive drop in effect size!

On Sun, Dec 12, 2010 02:46 AM, "Nicholas Thompson" <[hidden email]> wrote:

All—

Have others seen the article in the New Yorker on the “decline effect”, the alleged tendency for the effect sizes of well documented phenomena to decline with successive years of replication. I kept turning back to the front of the article to reassure myself that it was not one of the “Shouts and Murmurs” series. It is not. The passage that particularly caught my eye:

Many scientific theories continue to be considered true even after failing numerous experimental tests. … [This] holds for any number of phenomena, from the disappearing benefits of second-generation antipsychotics to the weak coupling ration exhibited by decaying neutrons, which appears to have fallen by more than ten standard deviations between 1969 and 2001. [NY mag, 15 december 2010, p57]

At least until recently, when the NY-er writes about science, they try very hard not to write anything stupid.

What gives?

Nicholas S. Thompson

Emeritus Professor of Psychology and Biology

Clark University

<a href="http://home.earthlink.net/%7Enickthompson/naturaldesigns/" onclick="window.open('http://home.earthlink.net/~nickthompson/naturaldesigns/');return false;">http://home.earthlink.net/~nickthompson/naturaldesigns/

<a href="http://www.cusf.org/" onclick="window.open('http://www.cusf.org/');return false;">http://www.cusf.org

Roger Critchlow-2

Re: The "decline effect" and the RTQ method

On Sun, Dec 12, 2010 at 7:24 AM, ERIC P. CHARLES <[hidden email]> wrote:

[ ... ]

On a more serious note (and the previous part was fairly serious already): Given that half the "major discoveries" promoted in psychology are assuredly garbage, how does this surprise you? Are you a "hard-science" snob, and only surprised because this is happening to physicists? There are a million reasons why an initial report of a phenomenon might overestimate the effect size. Some reasons are malicious (i.e., drug company funded studies as to the effectiveness of new drugs), others are benign (i.e. sampling error, unforeseen methodological shortcomings in initial tests, biased acceptance and promotion of "sexy" results).

The Neutral Model of Inquiry (or, What Is the Scientific Literature, Chopped Liver?)

Whole academic industries arise over non-existent effects: Piaget's "A-non-B error", menstrual synchrony, and infant's "innate mathematical abilities." Once the discipline is formed, it is very hard to unform.

-- rec --

Robert Holmes

Re: The "decline effect"

In reply to this post by Pamela McCorduck

... and this is why probability and statistics should be a compulsory class for everyone who goes through our education system :-)

-- R

On Sun, Dec 12, 2010 at 5:38 AM, Pamela McCorduck <[hidden email]> wrote:

The NYer can be obtuse about scientific topics, but this article intrigued me. Is the decline effect real?
It's certainly the case that many medical practitioners follow outdated advice. And the use of statistics in medicine (to be sure, a special subset of science) can be awkward. I keep asking people: if I've lowered my chances by twenty percent of contracting a certain cancer by doing thus-and-so, and I find four other thus-and-so's to also do, does that mean I'll never get that cancer? No one can answer.
On Dec 12, 2010, at 5:58 AM, [hidden email] wrote:

On 12 Dec 2010 at 0:46, Nicholas Thompson wrote:

At least until recently, when the NY-er writes about science, they try very
hard not to write anything stupid.

What??? Have you forgotten the whole disgraceful
Paul Brodeur episode? Refresh your memory by reading
<http://bobpark.physics.umd.edu/WN10/wn121010.html>.
Worse than stupid, verging on criminal, given the
amount of money it's caused to be thrown away and
the amount of anxiety it's generated or caused to
be misplaced.

I haven't yet read the "decline effect" article, and
am not commenting on it, just on your quoted sentence
above.

============================================================
FRIAM Applied Complexity Group listserv
Meets Fridays 9a-11:30 at cafe at St. John's College
lectures, archives, unsubscribe, maps at http://www.friam.org
"If you're away from Broadway, you're only camping out."

Thomas E. Dewey
============================================================
FRIAM Applied Complexity Group listserv
Meets Fridays 9a-11:30 at cafe at St. John's College
lectures, archives, unsubscribe, maps at http://www.friam.org

Eric Charles

Re: The "decline effect"

In reply to this post by lrudolph

Pamela,

What an odd question...

Don't you know that your initial chances of getting THAT type of cancer are less than 20% from the start?!? If you can find just one thing to lower your changes by twenty percent, that puts you into the negative probability range, and you can worry about other things.

Eric

On Sun, Dec 12, 2010 07:38 AM, Pamela McCorduck <[hidden email]> wrote:

The NYer can be obtuse about scientific topics, but this article intrigued me. Is the decline effect real?

It's certainly the case that many medical practitioners follow outdated advice. And the use of statistics in medicine (to be sure, a special subset of science) can be awkward. I keep asking people: if I've lowered my chances by twenty percent of contracting a certain cancer by doing thus-and-so, and I find four other thus-and-so's to also do, does that mean I'll never get that cancer? No one can answer.

On Dec 12, 2010, at 5:58 AM, lrudolph@... wrote:

On 12 Dec 2010 at 0:46, Nicholas Thompson wrote:

At least until recently, when the NY-er writes about science, they try very

hard not to write anything stupid.

What??? Have you forgotten the whole disgraceful
Paul Brodeur episode? Refresh your memory by reading
<<a href="http://bobpark.physics.umd.edu/WN10/wn121010.html" onclick="window.open('http://bobpark.physics.umd.edu/WN10/wn121010.html');return false;">http://bobpark.physics.umd.edu/WN10/wn121010.html>.
Worse than stupid, verging on criminal, given the
amount of money it's caused to be thrown away and
the amount of anxiety it's generated or caused to
be misplaced.

I haven't yet read the "decline effect" article, and
am not commenting on it, just on your quoted sentence
above.

Eric Charles

Professional Student and
Assistant Professor of Psychology
Penn State University
Altoona, PA 16601

Nick Thompson

Re: The "decline effect"

Funny how everybody criticizes the question but nobody answers it?

From: [hidden email] [mailto:[hidden email]] On Behalf Of ERIC P. CHARLES
Sent: Sunday, December 12, 2010 7:57 PM
To: friam
Subject: Re: [FRIAM] The "decline effect"

Pamela,

The NYer can be obtuse about scientific topics, but this article intrigued me. Is the decline effect real?

On Dec 12, 2010, at 5:58 AM, [hidden email] wrote:

On 12 Dec 2010 at 0:46, Nicholas Thompson wrote:

At least until recently, when the NY-er writes about science, they try very

hard not to write anything stupid.

What??? Have you forgotten the whole disgraceful
Paul Brodeur episode? Refresh your memory by reading
<http://bobpark.physics.umd.edu/WN10/wn121010.html>.
Worse than stupid, verging on criminal, given the
amount of money it's caused to be thrown away and
the amount of anxiety it's generated or caused to
be misplaced.

I haven't yet read the "decline effect" article, and
am not commenting on it, just on your quoted sentence
above.

Eric Charles

Professional Student and
Assistant Professor of Psychology
Penn State University
Altoona, PA 16601

Frank Wimberly

Re: The "decline effect"

Doesn’t saying that you will reduce your risk of getting cancer of the X by 20% by doing thus-and-so mean that you reduce the risk, for example, from 1% to 0.8% ?

And to answer Pamela’s question, I think the only way to be sure you won’t get cancer is to die of something else first.

Frank

From: [hidden email] [mailto:[hidden email]] On Behalf Of Nicholas Thompson
Sent: Sunday, December 12, 2010 8:34 PM
To: 'The Friday Morning Applied Complexity Coffee Group'
Subject: Re: [FRIAM] The "decline effect"

Funny how everybody criticizes the question but nobody answers it?

From: [hidden email] [mailto:[hidden email]] On Behalf Of ERIC P. CHARLES
Sent: Sunday, December 12, 2010 7:57 PM
To: friam
Subject: Re: [FRIAM] The "decline effect"

Pamela,

The NYer can be obtuse about scientific topics, but this article intrigued me. Is the decline effect real?

On Dec 12, 2010, at 5:58 AM, [hidden email] wrote:

On 12 Dec 2010 at 0:46, Nicholas Thompson wrote:

At least until recently, when the NY-er writes about science, they try very

hard not to write anything stupid.

Eric Charles

Professional Student and
Assistant Professor of Psychology
Penn State University
Altoona, PA 16601

Tom Carter

Re: The "decline effect"

In reply to this post by Nick Thompson

<base href="x-msg://355/">Nick -

There seem to me to be some good parts, and some not so good parts, to the article.

Back when I used to teach "Science, Technology, and Human Values" I had my students read this article from Science (about salt and diet, and science, and public policy):

http://www.junkscience.com/news3/taubes.html

(If you have Science access, the article with pictures is here: http://www.sciencemag.org/content/281/5379/898.short )

There is more to the story, but there are some good points in there . . .

About neutron coupling ratios . . . here's something to contemplate:

http://pdg.lbl.gov/2010/reviews/rpp2010-rev-history-plots.pdf

Clearly the saddest part is the declining life expectancy of the neutron (down to 886 seconds, from over 1000 in 1960). Maybe someone should do a correlation study with global warning . . . it would be a shame to see neutrons go extinct :-(

WRT Paul Brodeur -- obviously his biggest mistake was not having Julia Roberts (or Cher?) play him in a biopic :-)

tom

On Dec 11, 2010, at 11:46 PM, Nicholas Thompson wrote:

All—

Have others seen the article in the New Yorker on the “decline effect”, the alleged tendency for the effect sizes of well documented phenomena to decline with successive years of replication. I kept turning back to the front of the article to reassure myself that it was not one of the “Shouts and Murmurs” series. It is not. The passage that particularly caught my eye:

Many scientific theories continue to be considered true even after failing numerous experimental tests. … [This] holds for any number of phenomena, from the disappearing benefits of second-generation antipsychotics to the weak coupling ration exhibited by decaying neutrons, which appears to have fallen by more than ten standard deviations between 1969 and 2001. [NY mag, 15 december 2010, p57]

At least until recently, when the NY-er writes about science, they try very hard not to write anything stupid.

What gives?

Nicholas S. Thompson
Emeritus Professor of Psychology and Biology
Clark University
http://home.earthlink.net/~nickthompson/naturaldesigns/
http://www.cusf.org

============================================================
FRIAM Applied Complexity Group listserv
Meets Fridays 9a-11:30 at cafe at St. John's College
lectures, archives, unsubscribe, maps at http://www.friam.org

Robert J. Cordingley

Re: The "decline effect"

In reply to this post by Frank Wimberly

... and (building on Frank's case) if you to something else to reduce it by another 20% your overall risk goes to 0.64% and so on. It can never go to 0% how ever hard you try, but as Frank says your chance of dying of something else first might be much greater (e.g. in a car wreck).
Robert C

On 12/12/10 9:02 PM, Frank Wimberly wrote:

Doesn’t saying that you will reduce your risk of getting cancer of the X by 20% by doing thus-and-so mean that you reduce the risk, for example, from 1% to 0.8% ?

And to answer Pamela’s question, I think the only way to be sure you won’t get cancer is to die of something else first.

Frank

From: [hidden email] [[hidden email]] On Behalf Of Nicholas Thompson
Sent: Sunday, December 12, 2010 8:34 PM
To: 'The Friday Morning Applied Complexity Coffee Group'
Subject: Re: [FRIAM] The "decline effect"

Funny how everybody criticizes the question but nobody answers it?

From: [hidden email] [[hidden email]] On Behalf Of ERIC P. CHARLES
Sent: Sunday, December 12, 2010 7:57 PM
To: friam
Subject: Re: [FRIAM] The "decline effect"

Pamela,

What an odd question...

Don't you know that your initial chances of getting THAT type of cancer are less than 20% from the start?!? If you can find just one thing to lower your changes by twenty percent, that puts you into the negative probability range, and you can worry about other things.

Eric

On Sun, Dec 12, 2010 07:38 AM, Pamela McCorduck <[hidden email]> wrote:

The NYer can be obtuse about scientific topics, but this article intrigued me. Is the decline effect real?

It's certainly the case that many medical practitioners follow outdated advice. And the use of statistics in medicine (to be sure, a special subset of science) can be awkward. I keep asking people: if I've lowered my chances by twenty percent of contracting a certain cancer by doing thus-and-so, and I find four other thus-and-so's to also do, does that mean I'll never get that cancer? No one can answer.

On Dec 12, 2010, at 5:58 AM, [hidden email] wrote:

On 12 Dec 2010 at 0:46, Nicholas Thompson wrote:

At least until recently, when the NY-er writes about science, they try very

hard not to write anything stupid.

What??? Have you forgotten the whole disgraceful
Paul Brodeur episode? Refresh your memory by reading
<http://bobpark.physics.umd.edu/WN10/wn121010.html>.
Worse than stupid, verging on criminal, given the
amount of money it's caused to be thrown away and
the amount of anxiety it's generated or caused to
be misplaced.

I haven't yet read the "decline effect" article, and
am not commenting on it, just on your quoted sentence
above.

Eric Charles

Professional Student and
Assistant Professor of Psychology
Penn State University
Altoona, PA 16601
============================================================
FRIAM Applied Complexity Group listserv
Meets Fridays 9a-11:30 at cafe at St. John's College
lectures, archives, unsubscribe, maps at http://www.friam.org

Frank Wimberly

Re: The "decline effect"

Yes, assuming independence. I suspect there would likely be synergies, interferences or redundancies in the effects which would change the calculation.

Frank

From: [hidden email] [mailto:[hidden email]] On Behalf Of Robert J. Cordingley
Sent: Sunday, December 12, 2010 11:24 PM
To: The Friday Morning Applied Complexity Coffee Group
Subject: Re: [FRIAM] The "decline effect"

Doesn’t saying that you will reduce your risk of getting cancer of the X by 20% by doing thus-and-so mean that you reduce the risk, for example, from 1% to 0.8% ?

And to answer Pamela’s question, I think the only way to be sure you won’t get cancer is to die of something else first.

Frank

From: [hidden email] [[hidden email]] On Behalf Of Nicholas Thompson
Sent: Sunday, December 12, 2010 8:34 PM
To: 'The Friday Morning Applied Complexity Coffee Group'
Subject: Re: [FRIAM] The "decline effect"

Funny how everybody criticizes the question but nobody answers it?

From: [hidden email] [[hidden email]] On Behalf Of ERIC P. CHARLES
Sent: Sunday, December 12, 2010 7:57 PM
To: friam
Subject: Re: [FRIAM] The "decline effect"

Pamela,

The NYer can be obtuse about scientific topics, but this article intrigued me. Is the decline effect real?

On Dec 12, 2010, at 5:58 AM, [hidden email] wrote:

On 12 Dec 2010 at 0:46, Nicholas Thompson wrote:

At least until recently, when the NY-er writes about science, they try very

hard not to write anything stupid.

Eric Charles

Professional Student and
Assistant Professor of Psychology
Penn State University
Altoona, PA 16601

============================================================

FRIAM Applied Complexity Group listserv

Meets Fridays 9a-11:30 at cafe at St. John's College

lectures, archives, unsubscribe, maps at http://www.friam.org

Pamela McCorduck

Re: The "decline effect"

An interesting example (amost) of Zeno's Paradox.

On Dec 13, 2010, at 10:30 AM, Frank Wimberly wrote:

Yes, assuming independence. I suspect there would likely be synergies, interferences or redundancies in the effects which would change the calculation.

Frank

From: [hidden email] [[hidden email]] On Behalf Of Robert J. Cordingley
Sent: Sunday, December 12, 2010 11:24 PM
To: The Friday Morning Applied Complexity Coffee Group
Subject: Re: [FRIAM] The "decline effect"

... and (building on Frank's case) if you to something else to reduce it by another 20% your overall risk goes to 0.64% and so on. It can never go to 0% how ever hard you try, but as Frank says your chance of dying of something else first might be much greater (e.g. in a car wreck).
Robert C

On 12/12/10 9:02 PM, Frank Wimberly wrote:
Doesn’t saying that you will reduce your risk of getting cancer of the X by 20% by doing thus-and-so mean that you reduce the risk, for example, from 1% to 0.8% ?

And to answer Pamela’s question, I think the only way to be sure you won’t get cancer is to die of something else first.

Frank

From: [hidden email] [[hidden email]] On Behalf Of Nicholas Thompson
Sent: Sunday, December 12, 2010 8:34 PM
To: 'The Friday Morning Applied Complexity Coffee Group'
Subject: Re: [FRIAM] The "decline effect"

Funny how everybody criticizes the question but nobody answers it?

From: [hidden email] [[hidden email]] On Behalf Of ERIC P. CHARLES
Sent: Sunday, December 12, 2010 7:57 PM
To: friam
Subject: Re: [FRIAM] The "decline effect"

Pamela,
What an odd question...

Don't you know that your initial chances of getting THAT type of cancer are less than 20% from the start?!? If you can find just one thing to lower your changes by twenty percent, that puts you into the negative probability range, and you can worry about other things.

Eric

On Sun, Dec 12, 2010 07:38 AM, Pamela McCorduck <[hidden email]> wrote:
The NYer can be obtuse about scientific topics, but this article intrigued me. Is the decline effect real?

It's certainly the case that many medical practitioners follow outdated advice. And the use of statistics in medicine (to be sure, a special subset of science) can be awkward. I keep asking people: if I've lowered my chances by twenty percent of contracting a certain cancer by doing thus-and-so, and I find four other thus-and-so's to also do, does that mean I'll never get that cancer? No one can answer.

On Dec 12, 2010, at 5:58 AM, [hidden email] wrote:

On 12 Dec 2010 at 0:46, Nicholas Thompson wrote:

At least until recently, when the NY-er writes about science, they try very
hard not to write anything stupid.

What??? Have you forgotten the whole disgraceful
Paul Brodeur episode? Refresh your memory by reading
<http://bobpark.physics.umd.edu/WN10/wn121010.html>.
Worse than stupid, verging on criminal, given the
amount of money it's caused to be thrown away and
the amount of anxiety it's generated or caused to
be misplaced.

I haven't yet read the "decline effect" article, and
am not commenting on it, just on your quoted sentence
above.
Eric Charles

Professional Student and
Assistant Professor of Psychology
Penn State University
Altoona, PA 16601

============================================================
FRIAM Applied Complexity Group listserv
Meets Fridays 9a-11:30 at cafe at St. John's College
lectures, archives, unsubscribe, maps at http://www.friam.org
============================================================
FRIAM Applied Complexity Group listserv
Meets Fridays 9a-11:30 at cafe at St. John's College
lectures, archives, unsubscribe, maps at http://www.friam.org

"If you're away from Broadway, you're only camping out."

				Thomas E. Dewey