Let's talk about replication

On Thursday evening, I came across an online article on the Washington Post that talked about one of the things I took issue with when I wrote a critique of an article that I handed in is this past weekend. If only I had seen it before now! It would have helped my argument immensely. But such is life.

The article (with it's Upworthy-esque title "A shocking statistic about the quality of education research") reports on a newly published academic article that looked into how little research replication is being done in the education sciences.

Complaints of click bait aside, the statistic actually is shocking.

Only 0.13% of education research experiments are replicated.

Of those 0.13% that are replicated? Only 68% are successfully replicated.

And if you take a look at who is doing the replicating, only 54% of the 0.13% are replicated successfully without the original author on the research team.

That's not very good. 54% of 0.13% is....a really small percentage.

Replication is the norm in the "natural" sciences (as stated by education Professor Robert H. Bauernfeind in a magazine article published in 1968, available here if you have access to JSTOR through your library), and as an undergraduate student in zoology, I was constantly reminded that research methods needed to be clear and concise, because if someone couldn't repeat your experiment, well...it wasn't worth all that much.

Of course, replication in behavioural studies is more difficult than say, attempting to reaffirm the basal metabolic rate of Carcinus maenas. Unless of course your experiment goes wrong and your specimens end up eating each other and your results are consistent with the values of recently fed green crabs.

Bad memories. Anyway...

Replicating research in the education sciences can prove to be challenging because even the elements of research that should remain constant from study to study, may be difficult to keep that way. Teachers get new ideas for mixing up their instruction, Learning Management Systems get updated, section sizes in courses are constantly being inflated, not to mention that the demographics in the classroom from semester to semester, let alone year to year, can constantly fluctuate.

But still, shouldn't an effort be made? Wouldn't several studies or trials of your experiment confirming your results build a better case for your hypothesis? I imagine it would be especially beneficial to those in a position to stimulate change at their institution.

What do you think? Is replication important to education research or is the need for novel information too great?

The full report, titled Facts are more important than novelty: Replication in the education sciences, hasn't been published in print yet, but you can access it for free here.