Slides 📊
8.1. Experimental and Sampling Designs
We’ve explored data through descriptive methods, built probability models to understand uncertainty, and learned how sample statistics behave through sampling distributions and the Central Limit Theorem. These tools have equipped us to understand how information flows from populations to samples. But before we can confidently make the reverse journey—inferring from samples back to populations—we need one final, essential piece: understanding how to collect data properly in the first place. This chapter focuses on thoughtful, principled approaches to study design and data collection. Statistical inference is powerful, but it’s only as reliable as the data it’s based on.
Road Map 🧭
Identify the characteristics of a statistical question.
Understand that data contains statistical variation arising from different sources, and that each source is treated differently in an experiment.
Recognize the different sources of data, along with their respective advantages and disadvantages.
Distinguish between observational and experimental studies.
8.1.1. Statistical Questions and the Need for Data
Our journey toward statistical inference begins with recognizing what makes a question statistical in nature.
Statistical vs. Deterministic Questions
Not every question involving data is statistical. If we can predict an outcome with certainty given the inputs—like calculating the area of a rectangle from its length and width—we’re dealing with a deterministic relationship. Statistical questions, by contrast, involve relationships where perfect prediction is impossible, but where we can still identify meaningful patterns and quantify uncertainty by analyzing data.
A data always comes with inherent variability. This variability isn’t always a flaw to be eliminated—it’s often the fundamental characteristic that makes statistical methods both necessary and powerful.
The Nature of Statistical Variation
Statistical variation arises from multiple sources.
Subject differences: Individual units in a study naturally vary in their characteristics, responses, and behaviors.
Measurement errors: Even the most precise instruments introduce some degree of measurement uncertainty.
Random chance: Inherent randomness in natural processes and human behavior.
Ideally, we want the primary source of variation in our data to be random chance, with the influence from other sources minimized through careful design. We aim to reduce measurement errors through precise instruments and standardized procedures, and we handle subject differences through proper randomization and control strategies.
8.1.2. The Spectrum of Data Sources
Before conducting any study, researchers face a fundamental decision: where will their data come from? This choice shapes everything that follows—the types of conclusions that can be drawn, the statistical methods that are appropriate, and the confidence they can place in their results. Understanding the characteristics, advantages, and limitations of different data sources is essential for making informed research decisions.
Anecdotal Data
Anecdotal data represents the most basic form of information—observations from personal experiences, casual reports, or informal accounts shared through news media and social networks. While lacking scientific rigor, anecdotal evidence plays a surprisingly important role in the research ecosystem by providing insights or raising hypotheses to be studied further.
Available Data
The modern research landscape is dominated by an unprecedented availability of existing data. Available data includes any information that has already been collected and can be accessed for research purposes, ranging from government statistics and published study datasets to corporate databases and social media archives.
However, we do not have control over the quality, accuracy, and completeness of an available data, which can impact the reliability of the results and insights derived from it. It is important to asseess the quality of the available data carefully by considering data sources, data collection methods, and data processing techniques used.
Collecting New Data
When available data is insufficient or inappropriate for answering our research question, new data must be collected. Studies involving new data collection can be classified into two major branches:
Observational studies
Experimental studies
In the remainder of this section, we will briefly describe the characteristics of each and highlight their differences. Because experimental studies require greater and more deliberate researcher intervention, their details will be discussed further in Sections 8.2 through 8.5.
8.1.3. Observational Studies
In an observational study, researchers act as careful observers rather than active manipulators. They identify subjects of interest, contact them, and collect measurements, but they do not impose treatments or attempt to influence the study environment. This approach is particularly valuable when interventions would be unethical, impractical, or impossible, or when the research goal is to understand naturally occurring phenomena.
The observational approach follows a systematic process:
Define the research question.
Identify the target population.
Specify variables of interest, including both the primary variables of focus and potential confounding variables that might influence the results.
Design and implement random sampling procedures to obtain a representative sample from the target population.
Observe and measure the variables of interest without intervention.
Apply statistical inference methods to draw conclusions about the broader population.
Strengths and Limitations of Observational Studies
Observational studies excel at documenting naturally occurring relationships and patterns. They allow researchers to study phenomena in realistic settings where all the complex factors that influence outcomes in the real world remain present. This ecological validity makes observational studies particularly valuable for understanding how variables relate in natural environments.
A key limitation, however, is the lack of control over variables and treatment assignments. Without the ability to regulate which factors are present or how they vary, researchers cannot isolate the effects of specific conditions. As a result, the influence of other, uncontrolled factors may be difficult to separate from the patterns of interest.
This limitation does not reduce the value of observational studies; it simply calls for careful interpretation. Patterns observed consistently across multiple well-designed observational studies can still provide strong insights into how phenomena unfold in realistic contexts.
Example 💡: The Case of “Feline High-rise Syndrome” 🐈
Consider the study of “feline high-rise syndrome” by Whitney and Mehlhaff, published in the Journal of the American Veterinary Medical Association in 1987. The researchers wondered: when cats fall from buildings, how does the height of the fall relate to the severity of their injuries? To investigate this question, they identified 132 cats that had been brought to the Animal Medical Center in New York City after falling from multi-story buildings between June and November 1984. For each case, they carefully documented the cat’s injuries, the height from which it fell, and the outcome of treatment.
Their findings were surprising. About 90% of the cats survived their falls with appropriate veterinary care. But more intriguingly, they observed that cats falling from seven stories or higher didn’t sustain significantly more injuries than those falling from lower heights. In fact, cats falling from very high stories (nine floors or more) showed remarkably few limb fractures compared to those falling from intermediate heights.
The researchers proposed what they called the “terminal velocity hypothesis” to explain this pattern. They theorized that cats reach their maximum falling speed after about five stories. Once they achieve this terminal velocity and realize they’re in for a long fall, cats may relax into a “flying squirrel” posture that distributes impact forces more evenly across their body, reducing the likelihood of concentrated injuries.
Why This Had to Be Observational
This study illustrates why observational research is sometimes the only ethical option. Testing the terminal velocity hypothesis experimentally would require deliberately dropping cats from various heights—an approach that would be both unethical and illegal. Even if researchers could design some sort of controlled falling scenario with safety nets or other protections, such an artificial setup would fundamentally change the phenomenon being studied. Instead, the researchers had to rely on cats’ own decisions to fall from high ledges, windowsills, and fire escapes.
8.1.4. Experimental Studies
When ethical and practical constraints allow, experimental studies offer the strongest framework for statistical investigation. In contrast to observational studies, experiments involve deliberate control over one or more variables, including the ability to assign treatments or conditions to subjects according to a planned design. This control enables researchers to minimize the influence of extraneous factors and ensure that differences in outcomes can be more confidently attributed to the conditions under study.
The following sections will expand on this topic in greater detail.
8.1.5. Bringing It All Together
Key Takeaways 📝
Statistical questions require data with inherent variability and seek to quantify relationships among variables.
Data sources vary in quality and appropriateness. Anecdotal data provides inspiration but not evidence; available data offers efficiency but requires careful quality assessment; new data collection provides control but demands resources.
Observational studies are valuable for studying naturally occurring phenomena where intervention is impossible or unethical.
Experimental studies are appropriate when variables and environmental factors can be actively controlled.
The choice between observational and experimental approaches depends on research goals, ethical considerations, and practical constraints.
Study design determines the scope of valid conclusions.
Exercises
Study Classification: For each scenario below, identify whether the research question is statistical or deterministic, and explain your reasoning:
Does the amount of fertilizer applied to tomato plants affect their yield?
What is the area of a circular garden with radius 5 feet?
Are taller basketball players more likely to be successful free-throw shooters?
How much interest will $1000 earn in one year at 3% annual interest?
Observational vs. Experimental: Explain why each research question would likely require an observational rather than experimental approach:
Do people who smoke have higher rates of lung cancer?
Are children from single-parent households more likely to experience academic difficulties?
Does exposure to air pollution affect respiratory health?
Do people who exercise regularly live longer?