What Is Replication?

Why 64% of Psychology Studies Fail to Replicate

Researchers working on a replication study
Thomas Barwick / Stone / Getty Images

Replication is a term referring to the repetition of a research study, generally with different situations and different subjects, to determine if the basic findings of the original study can be generalized to other participants and circumstances.

Obtaining statistically significant results in a study is great, but when those same procedures and methods can be replicated with different participants with the same results, that is even better.

When studies are replicated and achieve the same or similar results as the original study, it gives greater credence to the findings and means that it is more likely that those results can be generalized to the larger population.

When a study has been replicated with the same results, it gives the study greater reliability. Remember, reliability refers to the consistency of a measure. If a measure is given multiple times to the same or different participants with the same results, it means that the measure has a high level of reliability.

Why might researchers decide to conduct a replication study?

  • These types of studies might be done to determine if the original results possess reliability and validity.
  • A replication study might be done if the original results are important and doing a replication will add to their generalizability and acceptance.
  • Researchers might want to see if the results of the original study might apply to different situations or different populations.

    How Do Scientists Replicate an Experiment?

    When conducting a study or experiment, it is essential to have clearly defined operational definitions. These operational definitions describe all of the variables that a researchers intends to measure. When reporting an experiment, it is also important to accurately and thoroughly describe the methods, materials, and procedures used for collecting data.

    When replicating earlier researchers, experimenters will follow these same procedures but with a different group of participants. If the researchers obtains the same or similar results in these follow-up experiments, it means that the original results are less likely to be merely a fluke.

    So what happens if the original results cannot be reproduced? Does that mean that the experimenters conducted bad research or that, even worse, they lied or fabricated their data? Fabricating results is rare since most scientists have a strong incentive to be honest in order to preserve their professional reputation and scientific standing.

    In most cases, non-replicated research is caused by differences in the participants or in other extraneous variables that might influence the results of an experiment. Sometimes the differences might not be immediately clear and in others researchers might be able to discern which variables might have impacted the results. For example, minor differences in things like the way questions are presented, the weather, or even the time of day the study is conducted might have an unexpected impact on the results of an experiment.

    Researchers might strive to perfectly reproduce the original study, but variations are expected and often impossible to avoid.

    The Reproducibility Project: Why Are the Results of Psychology Experiments So Hard to Replicate?

    In 2015, a group of more than 250 researchers published the results of their five-year effort to replicate 100 different experimental studies previously published in three top psychology journals. The replicators worked closely with the original researchers of each study in order to replicate the experiments as closely as possible.

    The results were less than stellar. Of the 100 experiments in question, 64 percent could not replicate the original results. Of the original studies, 97 percent of the findings were deemed statistically significant. Only 36 percent of the replicated studies were able to obtain statistically significant results.

    As one might expect, these dismal findings caused quite a stir.

    The results of psychology studies are often widely publicized in the popular press, so the suggestion that the majority of such studies might be wrong is problematic. Many people might be left wondering if psychological research is worthwhile or even believable at all.

    So why are psychology results so difficult to replicate? Writing for The Guardian, John Ioannidis suggests that there are a number of reasons why this might happen, but cautions that irreproducibility caused by scientific fraud is rare. Instead, he suggests that the intense competition for research funds and the powerful pressure to obtain significant results is likely a contributing factor. There is little incentive to retest, so many results obtained purely by chance are simply accepted without further research or scrutiny.

    Brain Nosek, project leader of the replication efforts, explained to the site FiveThirtyEight.com that the intention of the project was not to call out the original researchers for deceit or poor experimental design. Failing to replicate the results did not mean the original studies were flawed, he suggested. The goal was to demonstrate that fair and direct replications were possible.

    The project authors suggest that there are three key reasons why the original findings could not be replicated.

    1. The original results were a false positive.
    2. The replicated results were a false negative.
    3. Both studies were correct but differed due to unknown differences in experimental conditions or methodologies.

    The greatest predictor of replication success was the strength of the original results. Stronger results, as measure by the p-values, were more likely to also show an effect upon replication.

    While some might be tempted to look at the results of such replication projects and assume that psychology is rubbish, many suggest that such findings actually help make psychology a stronger science. Human thought and behavior is a remarkably subtle and ever-changing subject to study, so variations are to be expected when observing diverse populations and participants. Some research findings might be wrong, but digging deeper, pointing out the flaws, and designing better experiments helps strengthen the field.

    Related Topics

    More Psychology Definitions: The Psychology Dictionary

    References

    Aschwanden, C. (2015). Psychology is starting to deal with its replication problem. FiveThirtyEight. Retrieved from http://fivethirtyeight.com/datalab/psychology-is-starting-to-deal-with-its-replication-problem/.

    Ionnidis, J. (2015). Psychology experiments are failing the replication test - for good reason. The Guardian. Retrieved from http://www.theguardian.com/commentisfree/2015/aug/28/psychology-experiments-failing-replication-test-findings-science

    Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716. Doi: 10.1126/science.aac4716.

    Continue Reading