Planning and Executing Credible Experiments. Robert J. Moffat

Чтение книги онлайн.

3 An Overview of Experiment Planning

      My background is mainly in research and development experiments in heat transfer and fluid physics. When I think of planning an experiment, I think about wind tunnels, heat exchangers, temperature, and flow control. I have tried to generalize my experience, but my background certainly colors my outlook.

      Experimental work is expensive. Although costs vary for different fields and situations, the time costs are all similar: all laboratory work runs in real time, an hour for an hour, and there are no short cuts. It is important that experiments be well conceived, well executed, and well documented.

      The purpose of experiments is to produce provably accurate data that answer an agreed‐upon question about the behavior of a system. This is the kind of experiment I wish to discuss. Some of the important ideas will be transferable to other types of experiment, but the main thrust here is to deal with R&D experiments with tangible, numerical objectives.

      The planning of such an experiment is necessarily iterative. The process starts with a tentative plan: a first impression of the goal, a plausible experimental approach, a possible suite of instruments, a tentative set of tests to run. This plan must then be challenged: will it produce the desired information with acceptable accuracy? Then the plan is refined, sometimes by improving the statement of objectives, sometimes by selecting a better approach, sometimes by improving the instrumentation.

3.1 Steps in an Experimental Plan

      The general steps in an experimental program can be summarized in the following outline:

       The Iterative Loop

       The Execution

      Seldom is a successful research experiment designed on a once‐through basis. This is not surprising when you think about the amount of scratch paper usually generated in trying to develop an original analysis. Alternative experimental approaches must be investigated and trade‐offs made between accuracy and convenience, range and speed, etc.

3.2 Iteration and Refinement

      Research echoes sailing into uncharted territory. Sailboats cannot sail directly into the wind. The skipper must tack into the wind, repeatedly aiming right then left, iteratively correcting course.

      This book takes an iterative approach to experiment planning. For example, it may not be clear how to execute some of the steps until later, when their background material has been developed. Some of the steps themselves may not even make sense until the background material has been developed. And some of the background material we ask for won't make sense until the need for it has been established. This circularity is typical of large‐scale projects with interactions: They cannot be studied sequentially, they have to be approached integrating “all at once” and iterating.

      We emphasize the iterative nature of experiments. What seems plausible at first may not prove acceptable later. Sometimes a preliminary “exploratory” experiment is a good investment – to test an approach. It may show that the original concept of the experiment is not a good one. You likely will iterate the experiment itself, as in Figure 1.1. Details of the planning steps and their objectives are dealt with in subsequent chapters.

      An additional issue of risk assessment warrants early treatment because it affects every decision in the planning process.

3.3 Risk Assessment/Risk Abatement

      Consider the following risks:

Before the experiment is authorized to proceed, it must be established to the satisfaction of the reviewers that the motivating question can be answered with the desired accuracy, within the time allowed, and within the allowable budget.

3.4 Questions to Guide Planning of an Experiment

Tables 3.1–3.3 elaborate the contents for the rest of this book – the topics that we feel should be addressed during the planning of an experiment.

      Before some of the above topics can be addressed directly, some background material must be developed on the topology of experiments and the handling of experimental uncertainties.

Table 3.1 Overview of a research experiment plan.