Planning and Executing Credible Experiments. Robert J. Moffat

Чтение книги онлайн.

СКАЧАТЬ unsolved problem of classical physics.

Feynman spoke of basic turbulence. Turbulence can be further complicated by heat transfer; yet more complicated by mass transfer; yet more by chemical reactions or combustion; yet more complicated by electromagnetic interactions. Turbulence is key for weather, for breath and blood, for life, for flight, for circulation within celestial stars and their evolution. Turbulence remains unsolved to this day.

      To consider more viewpoints, we include three panels:

       Panel 2.1, “Positive Consequences of the Reproducibility Crisis”

       Panel 2.2, “Invitations to Experimental Research, Insights from Theoreticians”

       Panel 2.3, “Prepublishing Your Experiment Plan”

      This text focuses on experimental strategies, planning, techniques of analysis, and execution. That is our expertise, in addition to thermo‐fluid physics. We have taught experimental planning to students in many fields using draft notes of this text for more than 60 years.

Panel 2.1 Positive Consequences of the Reproducibility Crisis

      As researchers and instructors, we have been promoting experimental repeatability and uncertainty analysis for more than 60 years. When the work of Dr. J.P.A. Ioannidis brought the Reproducibility Crisis in the medical field to public awareness, we welcomed the positive impact it produced.

      Two papers by Dr. Ioannidis in 2005 brought the Reproducibility Crisis to the fore. One was the Journal of the American Medical Association (JAMA) article mentioned in Chapter 1, “Contradicted and Initially Stronger Effects in Highly Cited Clinical Research” (Ioannidis 2005a). The second was “Why Most Published Research Findings Are False” (Ioannidis 2005b).

      The two 2005 articles by Dr. Ioannidis appear to be a watershed moment for science. In various scientific disciplines, researchers have produced guidelines adopted by major publishers.

      Going deeper into the 2005 JAMA article, Dr. Ioannidis chose a notably high criteria for the publications he evaluated. He considered only:

       “All original clinical research studies published in 3 major general clinical journals or high‐impact‐factor specialty journals

       in 1990–2003 and

       cited more than 1000 times in the literature…”

      Dr. Ioannidis then compared these “results of highly cited articles … against subsequent studies of comparable or larger sample size and similar or better controlled designs. The same analysis was also performed comparatively for matched studies that were not so highly cited.”

Although part of the same article, this collection of research studies fared better than those mentioned in our Chapter 1. “Of 49 highly cited original clinical research studies, 45 claimed that the intervention was effective. Of these, 7 (16%) were contradicted by subsequent studies, 7 others (16%) had found effects that were stronger than those of subsequent studies, 20 (44%) were replicated, and 11 (24%) remained largely unchallenged.”⁵

In the same year, Dr. Ioannidis published “Why Most Published Research Findings Are False,” a provocative title. Although the wording appears to encompass all fields, the examples in the article were medical experiments. In order to make his evaluations, he adopted a key metric called the “Positive Predictive Value” (PPV). From this research, Dr. Ioannidis deduced the following “corollaries about the probability that a research finding is indeed true”:

       Corollary 1: The smaller the studies conducted in a scientific field, the less likely the research findings are to be true.

       Corollary 2: The smaller the effect sizes in a scientific field, the less likely the research findings are to be true.

       Corollary 3: The greater the number and the lesser the selection of tested relationships in a scientific field, the less likely the research findings are to be true.

       Corollary 4: The greater the flexibility in designs, definitions, outcomes, and analytical models in a scientific field, the less likely the research findings are to be true.

       Corollary 5: The greater the financial and other interests and prejudices in a scientific field, the less likely the research findings are to be true.

       Corollary 6: The hotter a scientific field (with more scientific teams involved), the less likely the research findings are to be true.

      Be aware. Beware small studies (i) and small effects (ii). Beware explaining indiscriminately (iii). Beware lack of standards (iv). Beware political favors and conflicts of interest (v). Beware fashionable science (vi). The corollaries provide fair warning for all research, and do affect the Nature of Experimental Work.

      Feynman's words suggest that experiments help science to be self‐correcting. However, Ioannidis (2012) gave the warning “Why Science Is Not Necessarily Self‐Correcting” in a more recent paper concerning psychological science.

      Dr. J.P.A. Ioannidis has done more for us than exposing flaws in medical science. He invites better experiments in the biological and medical sciences. He sharpens experimental discernment.

      Dr. Ioannidis is not alone. For example, Simera et al. (2010) report in the European Journal of Clinical Investigation how researchers collaborated and top publishers have agreed to guidelines regarding health research studies. There are many similar groups now.

      Furthermore, there is incentive to resist bad and fraudulent research. RetractionWatch.com keeps track of research that has been retracted by author, publisher, or sponsor.

      Overall, the concerted effort to assure the credibility of experiments keeps us in good company.

Panel 2.2 Selected Invitations to Experimental Research, Insights from Theoreticians

      The desire to better understand how our world works invites us to credible experiments. Beyond measuring, we predict what we expect to measure. Our interest may be our company's factory floor, economics and marketing, agriculture and environment, medicine, information, engineering and technology, climate, or the sciences biology, chemistry, and physics.

Richard Feynman in his lecture “There's Plenty of Room at the Bottom” (1959) invited scientists into a new field which we now call “nanoscience.” The launch of nanoscience has led to innovative products that affect our daily lives. Nanoscience, as a research field, reaches from measuring individual atoms within molecules, to manipulating atoms, to fluid nano‐arrays for medical and pharmaceutical tests and beyond.

      Better Invitation than a Nobel Prize

СКАЧАТЬ