Is There a Reproducibility Crisis in Writing Center Research?
/
Arne Hjorth Johansen, 'Egg' 2009
Today a public/private partnership between New America, Arizona State University and the online magazine Slate will hold an event called “Trust But Verify: The Crisis in Biomedicine.” The ‘crisis’ the title refers to arises from a recent, widespread realization that important research in the field may be fundamentally flawed, a majority of it impossible to replicate for various reasons. This same crisis is occurring in the field of psychology, causing practitioners to question some of their basic clinical assumptions. An obvious question presents itself: is there a reproducibility crisis in writing center research?
While the widespread realization that a reproducibility crisis exists in psychology and in biomedicine might be recent, the crisis itself is much older. Investors have long known that at least half of published studies that form the basis for drug research and development are not reproducible, scientists from Bayer finding in 2011 that “only in ~20%-25% (of in-house reproducibility efforts) were the relevant published data completely in line with our in-house findings” and that “analysis revealed that the reproducibility of published data did not significantly correlate with journal impact factors, the number of publications on the respective target or the number of independent groups that authored the publications.” In 2012 a researcher from pharma company Amgen, working with a scientist from the MD Anderson Cancer Center in Houston, found that of the 53 ‘landmark’ studies they tried to replicate, “scientific findings were confirmed in only 6 (11%) cases.” This suggests that somewhere between 50% and 90% of studies in some STEM fields or sub-fields may be flawed, and are certainly difficult to use as bases for further study. As the authors of the 2012 study note,
Some non-reproducible preclinical papers had spawned an entire field, with hundreds of secondary publications that expanded on elements of the original observation, but did not actually seek to confirm or falsify its fundamental basis. More troubling, some of the research has triggered a series of clinical studies — suggesting that many patients had subjected themselves to a trial of a regimen or agent that probably wouldn't work.
In psychology the problem is somewhat similar in scope. Almost exactly a year ago, an ambitious project in which researchers replicated 100 research findings in psychology ended with only 39 of the studies showing full reproducibility. It is worth noting that of the 61 studies whose results were not replicated, 24 were classed as having ‘moderately similar’ findings to those of the original experiment, but even in the most charitable mood most (but not all) researchers would admit that widespread non-reproducibility in the sciences remains a serious problem.
There are those, like the lead author of the study that found the 39% reproducibility rate in psychology, who argue that
There is no such thing as exact replication. All replications differ in innumerable ways from original studies. They are conducted in different facilities, in different weather, with different experimenters, with different computers and displays, in different languages, at different points in history, and so on.
There is a valuable cautionary note to be sounded here: it is absolutely true that scientists expect that some percentage of studies, no matter how carefully done, will not be reproducible. The fact that non-reproducibility is occurring is not the ‘crisis,’ and some would argue that the extent to which it is happening in psychology and cell biology doesn’t necessarily constitute a crisis either. Others point out that what is at stake is not simply the exact replication of a previous experiment, but a meaningfully similar repetition of the experiment that achieves the same outcome. Being able to reproduce an experiment doesn’t just involve using the same pipettes as the original team had, it also involves getting predictable results. In the case of cell biology this could mean the difference between providing a meaningful path towards a cure for cancer and providing a path toward an experimental treatment that impacts people’s lives without extending them, and all this begs the obvious question: why are so many studies in disciplines as different as psychology and cell biology not reproducible?
Onlookers note that there are vast sums of money at stake, and very little incentive to reproduce studies in the first place: top tier journals seldom publish studies that repeat previous studies, and almost never publish studies that fail to replicate a previous study’s results. Elizabeth Iorns notes in a Slate article that researchers who pay to reproduce their research might be accused of misusing their grant funding, and Glenn Begley, the Amgen scientist involved in the 2012 study described above, baldly states that the reproducibility crisis “is a systemic problem built on current incentives.” These incentives include publication in academic journals, which in turn leads to job security and financial gain in the form of salaries and the opportunity to seek grant monies, which in turn funds the institution in a self-replicating cycle. It is also true that these systemic, cyclic problems in turn change the ways science is done. Some of the problems noted by researchers include:
triplicate measurements of the same sample instead of independent replications
‘representative images’ used as evidence in cell biology, as opposed to quantifiable image analysis
population selection for ‘correct’ answers in animal studies
shoddy lab practices employed to speed labor performed by people who are severely overworked and often insecurely employed
From all this it seems clear that STEM fields may not be the bastions of objective truth and 'best practice' they sometimes claim to be, and that the ways the academic publishing industry and some academic institutions shape scientific research may not be uniformly positive.
The questions remains: so what? From the perspective of a writing center director, a whole host of research problems might precede non-reproducibility in importance. In fact, the ability to perform research at all is not a given for most writing center directors, and a lack of institutional support for the writing center’s continued existence forces plenty of writing center researchers to eschew research in favor of administrative duties that feel more pressing in the moment. This is unquestionably a bigger problem that any lack of replication in writing center research, if such a lack exists, but the reproducibility crisis in STEM does force us to ask which experimental models we are using in our own research and which methods and methodologies, and whether our means are commensurate with our ends. What relationship to reproducibility should writing center research have? Is our qualitative work too radically ‘located’ in a specific time and place to be repeated? If our research is by definition non-reproducible (as I suspect it is), should we eschew the metric entirely as a measurement of accuracy in writing center research?