8.4. Addressing Potential Flaws in Experimental Design
Understanding the three fundamental principles of experimental design provides the foundation for conducting rigorous research, but implementing these principles perfectly in real-world settings is often impossible. Despite our best efforts to control, randomize, and replicate, various issues can compromise the validity of our conclusions. The difference between a good experiment and a great one often lies not in eliminating all potential problems—which is usually impossible—but in recognizing where problems might arise and taking steps to minimize their impact.
This reality doesn’t diminish the importance of experimental design principles. Rather, it emphasizes why careful planning and critical thinking about potential limitations are essential parts of the research process. Every study involves trade-offs between ideal conditions and practical constraints, and understanding these trade-offs helps us interpret results appropriately and design better studies in the future.
Road Map 🧭
Problem: Even well-designed experiments can suffer from biases and limitations that compromise their validity and generalizability
Tool: Framework for recognizing and minimizing common experimental design issues, including selection bias, measurement bias, confounding, and lack of realism
Pipeline: Understanding these issues helps us design stronger studies and interpret results more critically, improving the quality of Sample → Population inferences
8.4.1. The Nature of Bias in Experimental Design
Bias in experimental design refers to systematic errors that cause our results to deviate from the truth in a consistent direction. Unlike random error, which varies unpredictably around the true value, bias consistently pushes our results away from reality in a systematic way. This distinction is crucial because while we can reduce random error through replication, bias cannot be eliminated simply by collecting more data.
Why Bias is Particularly Dangerous
Bias is insidious because it often goes undetected. Random variation is visible in our data—we can see that individual observations vary around some central tendency. But systematic bias can masquerade as real effects, leading us to conclude that treatments work when they don’t, or that they don’t work when they actually do.
Consider a hypothetical weight-loss study where the treatment group systematically includes more motivated participants. Even if the treatment has no real effect, the treatment group might show better results simply because motivated people are more likely to succeed regardless of the specific intervention they receive. Without recognizing this bias, we might incorrectly conclude that the treatment is effective.
Bias Produces Results That Deviate From Truth
The fundamental problem with bias is that it prevents us from observing what’s actually happening. Instead of measuring the true effect of our treatments, we’re measuring the combined effect of our treatments plus the systematic error introduced by our design flaws. This contamination can completely invalidate our conclusions, making even sophisticated statistical analysis meaningless.
Sources and Stages of Bias
Bias can enter experimental studies at multiple stages:
Design stage: Flawed randomization procedures, inappropriate control groups, or failure to account for important confounding variables
Implementation stage: Systematic differences in how treatments are administered or how different groups are treated
Measurement stage: Inconsistent measurement procedures, biased observers, or instruments that systematically favor certain groups
Analysis stage: Inappropriate statistical methods, selective reporting, or post-hoc changes to hypotheses
The earlier in the process that bias occurs, the more difficult it becomes to detect and correct. This is why careful attention to design principles from the very beginning is so crucial.
8.4.2. Selection Bias: When Groups Are Not Comparable
Selection bias occurs when experimental units are systematically assigned to treatment groups in ways that create fundamental differences between groups that are not due to the treatments themselves. Even when we follow randomization procedures, selection bias can still occur if our randomization is flawed or if certain types of units are more likely to end up in certain groups.
How Selection Bias Manifests
Selection bias violates the fundamental assumption that treatment groups are comparable except for the treatments they receive. When this assumption is violated, observed differences in outcomes might be due to pre-existing differences between groups rather than treatment effects.
Example: Medical Study with Flawed Assignment
Imagine a study testing a new medication where researchers unconsciously assign sicker patients to the treatment group, hoping to help them, while healthier patients end up in the control group. Even if the medication has no effect, the treatment group might show more improvement simply because sicker patients have more room for improvement (regression to the mean). Conversely, a truly effective medication might appear ineffective if the treatment group starts out much sicker than the control group.
Subtle Forms of Selection Bias
Selection bias doesn’t always involve obvious violations of randomization. More subtle forms include:
Volunteer Bias: When certain types of people are more likely to volunteer for specific treatments, creating systematic differences even with proper randomization procedures.
Attrition Bias: When different types of participants drop out of different treatment groups, changing the composition of groups over time.
Self-Selection: In studies where participants have some choice in their treatment assignment, systematic differences in who chooses which treatment can create bias.
Geographic or Temporal Clustering: When treatment assignment is related to location or time in ways that correlate with other important variables.
Preventing Selection Bias
The primary defense against selection bias is rigorous randomization with proper concealment of assignment sequences. Additional strategies include:
Stratified randomization to ensure balance on known important variables
Block randomization to maintain balance throughout the recruitment period
Intention-to-treat analysis to preserve the benefits of randomization even when participants don’t comply perfectly with assigned treatments
Careful monitoring of baseline characteristics to verify that randomization has achieved its intended goal
8.4.3. Measurement Bias: When Observations Are Systematically Distorted
Measurement bias refers to systematic errors in how we collect, record, or process our data. These errors consistently push measurements in one direction, creating apparent differences between groups that don’t reflect true treatment effects.
The Intern Example: A Case Study in Measurement Bias
Consider a medical experiment studying the effects of different treatments on blood chemistry. The study design calls for blood samples to be drawn from participants at regular intervals and analyzed in the laboratory. To ensure consistent procedures, the research team assigns specific personnel to work with specific treatment groups: Intern A always draws blood from Group 1 participants, Intern B works with Group 2, and Intern C handles Group 3.
This arrangement seems logical from a consistency standpoint—each group receives identical treatment from their assigned intern. However, it creates a serious risk for measurement bias. If one intern is less experienced or uses slightly different techniques, their measurements might be systematically different from the others.
How Measurement Error Becomes Bias
In our intern example, suppose Intern A is new and occasionally contaminates blood samples through improper technique. This contamination might systematically alter the laboratory results for Group 1, making their blood chemistry appear different from the other groups. The contamination effect might be subtle enough to go unnoticed during the study, but systematic enough to create apparent treatment differences that have nothing to do with the actual treatments.
The danger is that these systematically biased measurements might be interpreted as evidence of treatment effects. Group 1 might appear to respond differently to their treatment, when in reality the apparent difference is entirely due to measurement problems.
Types of Measurement Bias
Observer Bias: When people collecting data unconsciously favor certain outcomes, particularly when they know which treatment each participant is receiving.
Instrument Bias: When measurement tools systematically favor certain groups, either due to calibration problems or inherent characteristics of the instruments.
Recall Bias: In studies relying on participant reports, systematic differences in how different groups remember or report their experiences.
Social Desirability Bias: When participants alter their responses based on what they think researchers want to hear, particularly if treatment assignment is not blinded.
Preventing Measurement Bias
Blinding: Keep data collectors unaware of treatment assignments whenever possible.
Standardized Protocols: Develop detailed, written procedures for all measurements and ensure all personnel are properly trained.
Random Assignment of Personnel: Instead of assigning specific personnel to specific groups, randomize which personnel work with which participants.
Automated Measurements: Use objective, automated measurement tools when possible to reduce human error and bias.
Quality Control: Implement regular checks on measurement procedures and instrument calibration.
Multiple Observers: Use multiple people to collect the same measurements and check for consistency between observers.
8.4.4. Confounding Bias: The Problem of Unmeasured Influences
Confounding bias occurs when an extraneous variable is related to both our treatment factors and our response variable, but we fail to control for or block this variable in our design. Confounding represents one of the most challenging threats to valid causal inference because it can make treatments appear effective when they’re not, or mask real treatment effects.
Understanding Confounding Variables
A confounding variable (or confounder) has three key characteristics:
It influences the response variable we’re trying to measure
It’s associated with treatment assignment or treatment groups
It’s not part of the causal pathway we’re trying to study
When all three conditions are met, the confounding variable creates an alternative explanation for any observed association between treatment and response.
Example: Exercise and Heart Health
Suppose we want to study whether a new exercise program reduces heart disease risk. We recruit volunteers and randomly assign them to either participate in the exercise program or continue their normal routine. After six months, we find that the exercise group has better cardiovascular health markers.
However, imagine that we failed to account for dietary habits. If people who volunteer for exercise programs also tend to eat healthier diets, then diet becomes a confounding variable. The improved cardiovascular health might be due to:
The exercise program (what we want to conclude)
The healthier diets (confounding)
Both exercise and diet together
Neither (other unmeasured factors)
Without controlling for diet, we cannot determine which explanation is correct.
Why Confounding Is Particularly Challenging
Unmeasured Confounders: We can only control for variables we know about and can measure. Confounding variables that we haven’t identified or cannot measure remain threats to validity.
Complex Relationships: Confounders might interact with treatments in complex ways, making their effects difficult to predict or control.
Multiple Confounders: Real studies often involve multiple potential confounders that might influence results simultaneously.
Residual Confounding: Even when we attempt to control for confounders, our control might be incomplete due to measurement error or oversimplified modeling.
Strategies for Addressing Confounding
Randomization: Proper randomization distributes both known and unknown confounders equally across treatment groups on average, though it may not eliminate confounding in any particular study.
Blocking: When we know important confounding variables in advance, we can use randomized block designs to ensure balance across treatment groups.
Matching: In observational studies or when randomization is not possible, we can match participants on important confounding variables.
Statistical Control: During analysis, we can use statistical methods to adjust for known confounders, though this approach has limitations.
Restriction: We can limit our study to participants who are similar on potential confounding variables, though this may limit generalizability.
Multiple Studies: Different studies with different potential confounders can provide evidence about whether effects are robust across different contexts.
8.4.5. Lack of Realism: The Challenge of External Validity
Lack of realism represents a fundamental tension in experimental design between internal validity (our ability to draw valid causal conclusions within our study) and external validity (our ability to generalize those conclusions to real-world settings). This issue arises when our experimental units, treatments, or study settings fail to adequately represent the conditions we ultimately want to understand.
The Workplace Layoff Study: A Detailed Example
To understand how lack of realism can compromise experimental conclusions, consider a classic example from psychological research. Suppose researchers want to study how layoffs at a workplace affect the morale of workers who remain on the job—a question with obvious practical importance for understanding organizational behavior and employee well-being.
The Ethical Constraint
The most direct approach would be to conduct a true experiment: approach various employers and ask them to randomly lay off some employees so researchers can observe the effects on remaining workers. This approach would provide excellent internal validity—we could establish clear causal relationships between layoffs and morale changes.
However, this approach is completely unethical. Deliberately causing people to lose their jobs for research purposes would cause real harm to participants and their families. No institutional review board would approve such a study, and no ethical researcher would propose it.
The Compromise Solution
Faced with this ethical constraint, researchers might design an alternative study using college students as experimental units. The study design might work as follows:
Recruit college students to participate in a temporary job proofreading textbooks
Create a realistic work environment with multiple students working together
Randomly assign some students to be “laid off” during the study (with their knowledge and consent)
Monitor the remaining students and measure their morale and productivity
Administer psychological surveys to assess the impact of witnessing their colleagues being dismissed
This design maintains the three fundamental principles—it includes control groups (students who don’t witness layoffs), uses randomization (to determine who gets “laid off”), and incorporates replication (multiple students in each condition).
Why This Study Lacks Realism
Despite following proper experimental principles, this study suffers from serious limitations in realism that compromise its external validity:
Different Stakes and Investment
In a real workplace, employees typically depend on their jobs for their livelihood, often supporting families and building careers over many years. Being laid off represents a genuine threat to financial security, career progression, and personal identity. The job is central to their lives and future plans.
For college students proofreading textbooks, the stakes are fundamentally different. The work is temporary, part-time, and supplemental. While losing the job might be disappointing, it doesn’t threaten their primary life goals or financial security. The psychological impact of job loss—for both those laid off and those who remain—is likely to be much less severe.
Different Relationships and Social Dynamics
Real workplace colleagues often develop deep professional and personal relationships over months or years of working together. They share successes, support each other through challenges, and often form friendships that extend beyond work. Witnessing a layoff means seeing people they care about face genuine hardship.
College students working together temporarily for a research study haven’t had time to develop these deep relationships. While they might feel some sympathy for their fellow participants, the emotional impact of seeing someone “laid off” from a temporary research job is likely to be minimal compared to watching long-term colleagues lose their careers.
Different Work Environment and Culture
Established workplaces have complex cultures, hierarchies, and dynamics that develop over time. Employees understand advancement opportunities, have relationships with supervisors, and know the organization’s history and reputation. Layoffs in this context carry implications about company stability, future prospects, and personal career security.
A temporary research environment cannot replicate these complex organizational dynamics. The “workplace” exists only for the duration of the study, has no history or future, and offers no career advancement opportunities.
Different Participant Selection
College students represent a highly specific demographic: typically young, educated, financially supported (at least partially) by others, and in a temporary life phase focused on learning rather than career building. Their psychological responses to workplace dynamics might differ systematically from those of full-time workers with different backgrounds, ages, and life circumstances.
Why These Differences Matter
These limitations don’t just represent minor inconveniences—they fundamentally alter the phenomenon being studied. The psychological mechanisms that drive morale changes in response to layoffs might operate differently or not at all when the stakes are low, relationships are shallow, and participants aren’t truly invested in the work environment.
Results that might not generalize: The study might find no significant effects on student morale, leading researchers to conclude that layoffs don’t affect remaining workers. However, this conclusion might be completely wrong for real workplace settings where the psychological dynamics are more intense.
False confidence in findings: Alternatively, the study might find some measurable effects and researchers might conclude they’ve demonstrated the phenomenon. But the magnitude and nature of effects in real workplaces might be completely different.
Ethical Concerns Even in the Modified Design
Even the modified study design raises ethical questions. If some students aren’t informed that they’ll be “laid off” as part of the experiment, they might experience genuine distress even in this low-stakes environment. Proper informed consent requires that all participants understand what they’re agreeing to, but full disclosure might eliminate the psychological realism the study is trying to achieve.
The Broader Challenge of External Validity
The workplace layoff example illustrates a fundamental challenge in experimental research: the conditions that make experiments internally valid (control, artificial settings, volunteer participants) often make them less representative of real-world conditions.
Laboratory vs. Field Settings
Laboratory experiments offer maximum control but minimum realism. Field experiments offer maximum realism but minimum control. Researchers must choose where to position themselves on this continuum based on their research questions and constraints.
Participant Characteristics
Many studies rely on convenience samples (college students, volunteers, paid participants) who might not represent the broader population of interest. These participants might differ systematically in motivation, demographics, or other characteristics that influence the phenomena being studied.
Artificial Treatments
Experimental treatments are often simplified versions of complex real-world interventions. While this simplification enables clearer causal inference, it might miss important aspects of how interventions work in practice.
Short-term vs. Long-term Effects
Practical constraints often limit studies to short time periods, but many important effects might only emerge over longer periods. A medication study might miss side effects that develop after months of use, or an educational intervention might miss long-term retention effects.
8.4.6. Strategies for Maximizing Realism While Maintaining Validity
While perfect realism is often impossible, researchers can take steps to maximize external validity without completely sacrificing internal validity.
Representative Sampling
Invest in recruiting participants who represent the target population as closely as possible. This might mean recruiting from multiple sites, offering appropriate compensation to encourage participation from diverse groups, or partnering with organizations that serve the population of interest.
Realistic Settings and Procedures
Whenever possible, conduct studies in naturalistic settings that approximate real-world conditions. If laboratory studies are necessary, design procedures that maintain essential features of the real-world phenomenon.
Pilot Testing and Feedback
Before conducting the main study, test procedures with members of the target population and gather feedback about realism and relevance. Participants can often identify aspects of the study that don’t match their real-world experiences.
Multiple Studies with Different Approaches
Use convergent evidence from multiple studies that each address different aspects of the realism-validity trade-off. Laboratory studies might establish causal mechanisms, while field studies demonstrate real-world relevance.
Careful Interpretation and Acknowledgment of Limitations
Be explicit about the limitations of your study and how they might affect the generalizability of results. Discuss which aspects of your findings are most likely to generalize and which might be artifacts of your specific study conditions.
8.4.7. The Goal of Generalization
Understanding experimental design issues ultimately serves a larger purpose: enabling us to design studies whose results can be meaningfully extrapolated beyond the specific sample and conditions we studied. Generalization describes our ability to apply research findings to other populations, environments, or contexts that we care about but didn’t directly study.
The Generalization Challenge
Every study involves specific participants studied under specific conditions at a specific time and place. But the goal of most research is to understand broader principles that apply beyond these specifics. The challenge is determining when and how far we can reasonably extrapolate our findings.
Dimensions of Generalization
Population Generalization: Can findings from our specific sample be applied to the broader population of interest? This depends on how representative our sample is and whether the phenomena we’re studying operate similarly across different demographic groups.
Setting Generalization: Can findings from our specific research environment be applied to real-world settings? This depends on how well our study conditions approximate the conditions where we want to apply our findings.
Treatment Generalization: Can findings from our specific intervention be applied to similar but not identical interventions? This depends on whether we’ve captured the essential active components of the treatment.
Temporal Generalization: Can findings from our specific time period be applied to other time periods? This depends on whether the phenomena we’re studying are stable over time or influenced by changing social, technological, or environmental conditions.
Maximizing Generalizability Through Design
Maximize Representativeness of Experimental Units
The most direct way to improve generalizability is to ensure that study participants represent the population we want to understand. This might involve:
Stratified sampling to ensure representation of important subgroups
Multi-site studies that include participants from different geographic regions or institutional settings
Diverse recruitment strategies that reach participants who might not typically volunteer for research
Appropriate incentives that encourage participation without creating undue inducement
Maximize Similarity to Real-World Conditions
Study procedures should approximate real-world conditions as closely as possible while maintaining experimental control:
Naturalistic settings when feasible, or laboratory settings that preserve essential features of real-world environments
Realistic treatment delivery that matches how interventions would be implemented in practice
Appropriate timeframes that allow effects to develop as they would naturally
Realistic outcome measures that capture changes that matter in real-world contexts
Use Well-Established Measures and Methods
Employing measurement approaches that have been validated across multiple contexts increases confidence that findings will replicate:
Validated instruments with established psychometric properties across different populations
Standardized procedures that can be replicated consistently across different research teams and settings
Multiple measures of key constructs to ensure that findings aren’t dependent on specific measurement approaches
Established analysis methods that follow accepted standards in the field
The Realism-Control Trade-off
In practice, maximizing generalizability often requires trade-offs with internal validity. Studies with perfect control might lack realism, while perfectly realistic studies might lack the control needed for clear causal inference. The optimal balance depends on:
The research question: Some questions prioritize establishing that effects can occur (emphasizing control), while others prioritize understanding how effects work in practice (emphasizing realism)
Existing knowledge: Early research might emphasize control to establish basic causal relationships, while later research might emphasize realism to understand practical applications
Practical constraints: Available resources, ethical considerations, and institutional requirements all influence the feasible range of design options
Building Knowledge Through Multiple Studies
Rather than expecting any single study to achieve perfect generalizability, researchers often build knowledge through programs of research that systematically vary different aspects of studies:
Laboratory studies that establish causal mechanisms under controlled conditions
Field experiments that test whether effects occur in realistic settings
Multi-site trials that examine whether effects replicate across different contexts
Long-term follow-up studies that examine whether effects persist over time
8.4.8. The Simple Rule for Experimental Studies
Given the complexity of experimental design issues, researchers need practical guidance for making design decisions. A simple framework can help navigate the trade-offs and ensure that studies address the most important threats to validity.
Fig. 8.3 Simple Rule for Experimental Studies: A practical framework for design decisions
Control What You Can
The first priority is to identify and control variables that can be controlled without compromising the essential nature of the study. This includes:
Standardizing procedures across all participants and treatment groups
Training personnel to follow consistent protocols
Using validated measurement instruments and calibrating equipment regularly
Controlling the physical environment where possible (lighting, temperature, noise)
Establishing clear inclusion and exclusion criteria for participants
Block What You Can’t Control
For important variables that cannot be controlled but can be measured, use blocking or stratification to ensure balance across treatment groups:
Known demographic variables that influence the response (age, gender, education level)
Baseline measures of the outcome variable or related constructs
Environmental factors that vary systematically (testing location, time of year)
Participant characteristics that affect treatment compliance or response
Randomize to Create Comparable Groups
Use rigorous randomization procedures to distribute unmeasured and unmeasurable factors equally across treatment groups:
Proper randomization procedures with adequate concealment of assignment sequences
Appropriate randomization level (individual vs. cluster randomization)
Stratified or blocked randomization when important grouping variables are known
Verification of randomization success through baseline comparisons
Ensure Sufficient Replication
Plan for adequate sample sizes that provide sufficient power to detect meaningful effects:
Power analysis to determine required sample sizes before beginning data collection
Adequate representation of important subgroups within the sample
Plans for handling missing data and participant dropout
Multiple measurements when possible to increase reliability
Why This Rule Works
This framework works because it systematically addresses the major threats to experimental validity in order of tractability. By controlling what can be controlled, blocking what can be measured but not controlled, and randomizing the rest, researchers can minimize bias while maintaining practical feasibility.
The rule also emphasizes that perfect studies are impossible—the goal is not to eliminate all potential problems but to address them systematically according to their importance and tractability.
8.4.9. Conclusion: Design Is More Important Than Analysis
Perhaps the most important lesson from studying experimental design issues is that the quality of a study is determined primarily by its design rather than its analysis. No amount of sophisticated statistical analysis can compensate for fundamental design flaws, while even simple analytical methods can yield profound insights when applied to well-designed studies.
Design Determines the Scope of Valid Conclusions
The design choices made before any data is collected determine:
Whether causal inferences are justified
Which populations the results apply to
How seriously alternative explanations threaten our conclusions
What types of statistical analysis are appropriate
Why Good Design Is Challenging
Experimental design is challenging because it requires balancing multiple competing considerations:
Scientific rigor vs. practical feasibility
Internal validity vs. external validity
Comprehensive measurement vs. participant burden
Ideal conditions vs. ethical constraints
These trade-offs require judgment, experience, and deep understanding of both the research question and the practical context where the study will be conducted.
The Foundation for Valid Inference
As we move into statistical inference methods in subsequent chapters—confidence intervals, hypothesis testing, and regression analysis—remember that these mathematical tools only work properly when applied to data collected through appropriate experimental designs. The most elegant statistical procedures are meaningless when applied to biased data or inappropriate samples.
Understanding experimental design issues provides the foundation for being both a better researcher (designing studies that produce trustworthy results) and a better consumer of research (evaluating the limitations of studies conducted by others). This critical perspective is essential for evidence-based decision making in science, policy, and practice.
Key Takeaways 📝
Bias systematically distorts results in consistent directions, making it more dangerous than random error because it cannot be reduced through larger sample sizes.
Selection bias occurs when treatment groups differ systematically on characteristics other than treatments, undermining the comparability that randomization is meant to achieve.
Measurement bias arises from systematic errors in data collection procedures, instruments, or personnel that consistently favor certain groups or outcomes.
Confounding bias happens when unmeasured variables influence both treatment assignment and outcomes, creating alternative explanations for observed effects.
Lack of realism represents the trade-off between internal validity (experimental control) and external validity (generalizability to real-world conditions).
Perfect studies are impossible: The goal is not to eliminate all potential problems but to minimize them systematically while maintaining study feasibility.
Design quality determines analysis validity: No statistical method can compensate for fundamental design flaws, making careful experimental planning more important than sophisticated analysis techniques.
The simple rule provides practical guidance: Control what you can, block what you can’t control, randomize to create comparable groups, and ensure sufficient replication.
The journey from understanding probability and sampling distributions to conducting reliable statistical inference requires more than mathematical sophistication—it demands careful attention to how data are collected and what assumptions are met. By understanding common experimental design issues and strategies for addressing them, we prepare ourselves to conduct studies that produce trustworthy evidence and to evaluate critically the research conducted by others.
As we continue through this chapter and move into the methods of statistical inference, keep in mind that every statistical procedure depends on assumptions about how data were collected. Violations of these assumptions—often caused by the design issues we’ve discussed—can invalidate even the most mathematically correct analyses. Good experimental design is not just about following rules; it’s about thinking critically about what could go wrong and taking steps to prevent it.
Exercises
Identifying Types of Bias: For each scenario below, identify the primary type of bias present and explain how it could affect the study conclusions:
A study of a new teaching method where more motivated teachers volunteer to use the new method.
A medical trial where nurses measuring patient recovery know which patients received the experimental treatment.
A study of exercise and mental health that fails to account for participants’ baseline fitness levels.
A psychology experiment using only college students to study workplace stress management.
The Intern Example: Redesign the blood chemistry study to minimize measurement bias while maintaining practical feasibility. Explain your approach and any remaining limitations.
Confounding Variables: A researcher finds that students who take music lessons have higher math scores. Identify three potential confounding variables that could explain this association and explain how each one might work.
Lack of Realism Analysis: Consider the workplace layoff study discussed in this chapter. Suggest three specific modifications to the study design that could increase realism while maintaining ethical standards. Discuss the trade-offs involved in each modification.
External Validity Assessment: You’re reading about a study that tested a new therapy for anxiety using a sample of 100 college students at a private university. The therapy showed significant benefits compared to a control group. Discuss the limitations in generalizing these results to:
Adults with anxiety in the general population
Patients seeking treatment at community mental health centers
People with severe anxiety disorders
Long-term effectiveness of the therapy
Design Improvement: A company wants to test whether flexible work schedules improve employee productivity. Their initial plan is to let employees in one department choose flexible schedules while requiring traditional schedules in another department. Identify the problems with this approach and design a better study using the simple rule framework.
Bias Prevention Strategy: You’re designing a study to test whether a new app helps people stick to exercise routines. For each type of bias discussed in this chapter, explain:
How it might manifest in your study
What specific steps you would take to minimize it
Any remaining limitations you couldn’t completely eliminate
Critical Evaluation: Find a news article reporting on a scientific study (health, education, psychology, etc.). Based on the information provided:
Identify potential experimental design issues that might limit the conclusions
Assess how well the study results might generalize beyond the specific sample and conditions studied
Suggest what additional information you would need to evaluate the study’s quality more thoroughly