8.5. Examples of Experimental Design
The examples we’ll explore in this section demonstrate how researchers identify key features of their research context, match those features to appropriate design frameworks, and implement the designs while addressing practical constraints. Each example builds on the principles and design types we’ve studied, showing how they work together in realistic research scenarios.
8.5.1. Example 1: Instructional Methods Study
Educational researchers want to understand how different instructional methods affect student learning outcomes. The researchers identify three instructional approaches they want to compare:
Self-study from textbook: Students learn independently using written materials
Traditional classroom instruction: Students attend lectures and participate in class discussions
Online instruction: Students complete computer-based modules with interactive content
They recruit a sample of 90 students who will all be learning the same material. The students are randomly assigned to the three groups, with 30 students in each group. After a specified learning period, all students take the same comprehensive exam, and the researchers compare average scores across the three instructional methods.
Design Analysis
Fig. 8.8 Example 1: Instructional Methods
1. What type of experimental design is used?
This is a Completely Randomized Design (CRD). The key evidence is that students are simply randomly assigned to one of the three treatment groups without any prior grouping or blocking. No student characteristics are used to form blocks, and there’s no pairing or matching process.
2. What are the experimental units?
The 90 students serve as the experimental units. Since we’re dealing with human participants, they’re more specifically called subjects.
3. What are the treatments?
The treatments are the application of the three different instructional methods:
Application of self-study learning approach
Application of traditional classroom instruction
Application of online instruction methodology
Note that the traditional classroom instruction could be considered a control group since it represents the standard or conventional approach against which the other methods are being compared.
4. What are the factors of interest and their levels?
There is one factor of interest: the type of instructional method. The instructional method factor has three levels:
Self-study from textbook
Traditional classroom instruction
Online instruction
6. What is the response variable?
The response variable is scores on the final exam. This provides a quantitative measure of how well students learned the material under each instructional approach.
8.5.2. Example 2: Hand Cream Comparison
A cosmetics company wants to compare the effectiveness of two different hand cream formulations. Consumer satisfaction is highly subjective and varies greatly between individuals based on skin type, preferences, and expectations. This variability makes it difficult to detect treatment differences using simple comparisons between different groups of people.
The researchers recruit 40 women to participate in a hand cream comparison study. Half the women (20) use Hand Cream Type 1 for one month, while the other half (20) use Hand Cream Type 2 for one month. All participants stop using any hand cream for two weeks to allow the effects of the first cream to dissipate. The groups switch—those who used Type 1 now use Type 2 for one month, and those who used Type 2 now use Type 1 for one month. At the end of each phase, participants complete satisfaction surveys. The researchers then compare each woman’s satisfaction with Type 1 versus Type 2.
Design Analysis
Fig. 8.9 Example 2: Hand Cream Comparison
1. What type of experimental design is used?
This is a Matched Pairs Design, Type 2 (also called a self-paired design). The key identifying feature is that each participant serves as her own control by trying both treatments.
2. What are the experimental units?
The 40 women are the experimental units. Since they’re human participants, they’re specifically called subjects.
3. What are the factors?
There is one factor: the application of the two different hand cream formulations:
Application of Hand Cream Type 1
Application of Hand Cream Type 2
4. What are the factors of interest and their levels?
There is one factor of interest: the type of hand cream. The hand cream factor has two levels:
Hand Cream Type 1 (Brand 1)
Hand Cream Type 2 (Brand 2)
5. What is the response variable?
The response variable is satisfaction score. This is likely measured through a standardized survey that produces a numerical score representing each participant’s satisfaction with the hand cream they used during that phase.
8.5.3. Example 3: Dairy Cow Nutrition Study
Agricultural researchers want to investigate whether the type of grain mix affects milk production in dairy cows. The researchers plan to compare two different brands of grain mix to see which produces higher milk yields. They have access to 16 dairy cows from different farms, but these cows represent four different breeds. The breed of cow is known to significantly affect baseline milk production, with some breeds naturally producing much more milk than others.
The 16 cows are first separated into four blocks based on their breed, with 4 cows from each breed in each block. Within each breed (block), 2 cows are randomly assigned to receive Grain Mix Brand 1, and the other 2 cows receive Grain Mix Brand 2. After a specified feeding period, milk production is measured for each cow, and the researchers compare average production between the two grain mix brands while accounting for breed differences.
Design Analysis
Fig. 8.10 Example 3: Dairy Cow Nutrition Study
1. What type of experimental design is used?
This is a Randomized Block Design (RBD). The key identifying features are:
Experimental units are first grouped into blocks based on an important characteristic (breed).
Random assignment occurs within each block rather than across all units.
The blocking variable (breed) is not the primary interest but is controlled to improve precision.
2. What are the experimental units?
The 16 dairy cows serve as the experimental units.
3. What are the treatments?
The treatments are feeding the cows different grain mix brands:
Feeding Grain Mix Brand 1
Feeding Grain Mix Brand 2
4. What are the factors of interest and their levels?
The primary factor of interest is the type of grain mix. This has two levels: Brand 1 and Brand 2.
5. What is the response variable?
The response variable is milk production, likely measured as the total volume of milk produced per cow over a specified time period (such as gallons per day or pounds per week).
8.5.4. Bringing It All Together
Key Takeaways 📝
Systematic analysis is essential: Using the six key questions (design type, experimental units, treatments, factors, levels, response) provides a complete understanding of any experimental design.
Context determines design choice: The instructional methods study used CRD because students were relatively homogeneous; the hand cream study used matched pairs because individual differences were large; the dairy study used RBD because breed was a known important factor.
Each design addresses different challenges: CRD handles general variability through randomization; matched pairs controls individual differences through pairing; RBD controls known sources of variability through blocking.
Trade-offs are unavoidable: Every design involves compromises between statistical power, practical feasibility, and experimental control.
Real studies are messier than textbook examples: Practical constraints, ethical considerations, and resource limitations often require modifications to ideal designs.
Exercises
Design Identification: For each scenario below, identify which experimental design would be most appropriate and explain your reasoning:
Testing whether three different fertilizers affect plant growth using 60 plants of the same species.
Comparing two pain medications using 40 patients with chronic pain, where patients try both medications separated by a washout period.
Studying the effect of teaching method on learning outcomes using students from three different grade levels.
Comparing the effectiveness of two workout routines using identical twins as participants.
Blocking Variable Selection: In the dairy cow study, breed was used as the blocking variable. Identify three other variables that might be good blocking variables for this study and explain why each would be appropriate.
Design Modification: Suppose the instructional methods study (Example 1) could be modified to use a randomized block design.
What variable would you use for blocking and why?
How would this change the implementation of the study?
What advantages and disadvantages would this create compared to the original CRD?
Sample Size and Design: Explain why the dairy cow study needed to use RBD rather than matched pairs, even though individual differences between cows are probably quite large.