http://friam.383.s1.nabble.com/The-decline-effect-tp5827610p5828149.html
Nick, et al.,
I kept flirting with the idea of writing an article for
the Journal of Irreproducible Results about the RTQ method of improving test
scores for underachieving students. RTQ stands for Re-Test Quickly. The method
is well proven. To try it: 1) Create a set of matched multiple choice tests on
general relativity. 2) Give it to a group of 2nd graders. 2) Identify the
lowest quartile of the class. 3) Immediately retest them with another version.
4) You will find that the students knowledge of general relativity has
significantly increased. *Warning, do not try to apply this method with already
well performing students.*
The New Yorker might merely be noticing that
several scientists fail to heed the warning.
On a more serious note
(and the previous part was fairly serious already): Given that half the "major
discoveries" promoted in psychology are assuredly garbage, how does this
surprise you? Are you a "hard-science" snob, and only surprised because this is
happening to physicists? There are a million reasons why an initial report of a
phenomenon might overestimate the effect size. Some reasons are malicious
(i.e., drug company funded studies as to the effectiveness of new drugs),
others are benign (i.e. sampling error, unforeseen methodological shortcomings
in initial tests, biased acceptance and promotion of "sexy"
results).
Whole academic industries arise over non-existent effects:
Piaget's "A-non-B error", menstrual synchrony, and infant's "innate
mathematical abilities." Once the discipline is formed, it is very hard to
unform.
So, if the NYer is being stupid, it is being stupid either for
1) not understanding what is going on, 2) not recognizing the legitimacy of
what is going on, or 3) being selective in reporting by not noticing that some
effects raise over time. I suspect a combination of all of those, with #3 being
the most damning from a journalistic perspective.
Eric
P.S. It is
also possible that the effect sizes are legitimately changing over time. Lets
be honest, doesn't almost everything seem a little less important now than it
used to? I mean, just a month or two ago backscatter technology and forced
groping seemed like a big deal... and how many people's lives are currently
being endangered this week by WikiLeakes... what about Obama's Hope and Change
effect... or the way the Republicans would fix Washington... talk about a
pervasive drop in effect size!
On Sun, Dec 12, 2010 02:46 AM,
"Nicholas Thompson" <[hidden email]>
wrote:
All—
Have others seen the article in the New Yorker on the
“decline effect”, the alleged tendency for the effect sizes of well
documented phenomena to decline with successive years of replication.
I kept turning back to the front of the article to reassure myself that
it was not one of the “Shouts and Murmurs” series. It is not.
The passage that particularly caught my eye:
Many scientific theories
continue to be considered true even after failing numerous experimental tests.
… [This] holds for any number of phenomena, from the disappearing
benefits of second-generation antipsychotics to the weak coupling ration
exhibited by decaying neutrons, which appears to have fallen by more than ten
standard deviations between 1969 and 2001. [NY mag, 15 december 2010,
p57]
At least until recently, when the NY-er writes about
science, they try very hard not to write anything stupid.
What gives?
Nicholas S. Thompson
Emeritus Professor of Psychology and
Biology
Clark University
<a
href="http://home.earthlink.net/%7Enickthompson/naturaldesigns/"
onclick="window.open('http://home.earthlink.net/~nickthompson/naturaldesigns/');return false;">http://home.earthlink.net/~nickthompson/naturaldesigns/
<a href="http://www.cusf.org/" onclick="window.open('http://www.cusf.org/');return false;">http://www.cusf.org
Meets Fridays 9a-11:30 at cafe at St. John's College