8.6. Sampling Design
Understanding experimental design principles provides the framework for establishing causal relationships, but even the most carefully designed experiment is only as strong as the sample of participants it studies. How we select experimental units or subjects for our studies fundamentally determines whether our conclusions can be generalized beyond the specific individuals we observe. This critical connection between sampling and inference represents one of the most important—yet often overlooked—aspects of research design.
The transition from experimental design to sampling design marks a shift from internal validity (can we trust the causal conclusions within our study?) to external validity (can we generalize our conclusions to the broader population we care about?). While experimental design principles ensure that our comparisons are fair and unbiased, sampling design principles ensure that our participants represent the population we want to understand.
Road Map 🧭
Problem: How do we select participants for our studies so that results can be generalized to the populations we want to understand?
Tool: Framework for understanding different sampling approaches, from convenience sampling to sophisticated randomized methods
Pipeline: Proper sampling design enables the valid Sample → Population inferences that are the ultimate goal of statistical research
8.6.1. The Foundation of Statistical Inference
Before examining specific sampling methods, it’s essential to understand why sampling design matters so profoundly for statistical inference. The mathematical tools we use for drawing conclusions—confidence intervals, hypothesis tests, regression analysis—all depend on specific assumptions about how our data were collected. When these assumptions are violated, our statistical procedures can produce misleading results, no matter how sophisticated the analysis.
The Connection to IID Assumptions
Most statistical inference procedures assume that our observations are independent and identically distributed (IID). This seemingly abstract mathematical concept has very concrete implications for how we collect data:
Independence means that observing one unit doesn’t influence the probability of observing any other unit, and that the characteristics of one unit don’t affect the characteristics of others in our sample.
Identically distributed means that all units come from the same population with the same underlying probability distribution—they’re all drawn from the same “statistical population” with the same parameters.
When we violate these assumptions through poor sampling design, our statistical inference procedures lose their validity. Standard errors become incorrect, confidence intervals don’t have their stated coverage rates, and hypothesis tests don’t control error rates as intended.
The Population-Sample-Population Cycle
Statistical inference follows a logical cycle that depends entirely on proper sampling design:
Define the target population we want to understand
Draw a representative sample from that population using appropriate methods
Analyze the sample data using statistical procedures
Generalize results back to the target population with known levels of uncertainty
Each step depends on the previous ones. If our sample isn’t representative of our target population (step 2), then our analysis (step 3) and conclusions (step 4) will be invalid, no matter how sophisticated our statistical methods.
8.6.2. The Challenge of Representativeness
The concept of representativeness is central to sampling design, but it’s more complex than it might initially appear. A representative sample is one that accurately reflects the characteristics of the population from which it’s drawn, but achieving representativeness requires careful attention to potential sources of bias.
What Makes a Sample Representative?
A representative sample should:
Include appropriate proportions of different subgroups within the population
Capture the full range of variability present in the population
Avoid systematic exclusion of certain types of individuals
Reflect the diversity of opinions, characteristics, or responses present in the population
Why Representativeness Is Challenging
Several factors make it difficult to achieve truly representative samples:
Population Definition Challenges: Before we can sample representatively, we must clearly define our target population. This seemingly simple task often reveals complex decisions about inclusion and exclusion criteria.
Access and Feasibility Issues: Some members of the population may be much easier to reach than others, creating systematic biases in who ends up in our sample.
Response and Participation Patterns: Even with perfect sampling procedures, certain types of people may be more or less likely to agree to participate, creating post-sampling biases.
Cost and Resource Constraints: Truly representative sampling can be expensive and time-consuming, leading researchers to make compromises that affect representativeness.
Hidden Population Structure: Populations often have complex internal structure that isn’t immediately apparent, making it easy to miss important subgroups or relationships.
8.6.3. Non-Random Sampling Methods: Understanding the Limitations
While we generally prefer randomized sampling methods for statistical inference, non-random sampling approaches are common in practice. Understanding their limitations helps us recognize when they might be appropriate (usually for preliminary investigations) and when they create serious threats to validity.
Convenience Sampling: The Path of Least Resistance
Convenience sampling selects participants based solely on ease of access and availability. This approach is attractive because it’s simple, fast, and inexpensive to implement, making it tempting for researchers with limited resources or tight timelines.
Common Examples of Convenience Sampling
Academic Research: A psychology professor studies decision-making by recruiting students from her own classes. While convenient, this sample only represents college students in that particular major at that specific institution.
Medical Research: A doctor studies the effectiveness of a new treatment by enrolling patients who visit his clinic. This sample may systematically exclude people who can’t afford medical care, live far from the clinic, or prefer different healthcare providers.
Market Research: A company surveys customers who visit their website or respond to email invitations. This approach misses potential customers who don’t engage with the company online.
Political Polling: News outlets conduct “person on the street” interviews in busy downtown areas. Such samples systematically overrepresent people who work downtown, have flexible schedules, and are comfortable talking to reporters.
Why Convenience Sampling Creates Bias
Convenience sampling creates systematic bias because the characteristics that make people easily accessible often correlate with other important variables:
Geographic Clustering: Sampling from easily accessible locations (like college campuses or shopping malls) concentrates the sample in specific geographic areas that may not represent broader populations.
Socioeconomic Bias: People with more flexible schedules, reliable transportation, and discretionary time are more likely to be available for convenience sampling, creating systematic socioeconomic biases.
Demographic Patterns: Age, employment status, family situation, and health status all affect availability and accessibility, leading to systematic under- or over-representation of certain demographic groups.
Behavioral Correlates: The behaviors and attitudes that make people easy to recruit (outgoing personality, willingness to participate in research, comfort with authority figures) may correlate with the very outcomes researchers want to study.
When Convenience Sampling Might Be Appropriate
Despite its limitations, convenience sampling can serve legitimate purposes in certain contexts:
Preliminary Research: When exploring whether a phenomenon exists or developing research methods, convenience samples can provide initial insights at low cost.
Proof of Concept Studies: Testing whether an intervention can work under any circumstances might justify convenience sampling before investing in more expensive representative studies.
Method Development: When developing new measurement instruments or refining research procedures, convenience samples can be adequate for initial testing.
Extreme Case Analysis: When studying rare phenomena or extreme cases, convenience sampling might be the only feasible approach.
Voluntary Response Sampling: Self-Selection Bias
Voluntary response sampling occurs when individuals self-select into the study based on their own willingness or motivation to participate. This approach is common in online surveys, call-in polls, and studies that recruit participants through advertisements or social media.
The Psychology of Voluntary Response
People who volunteer for research studies differ systematically from those who don’t in several important ways:
Strong Opinions: Individuals with extreme views on the topic being studied are much more likely to participate than those with moderate or neutral opinions.
Personal Investment: People who feel personally affected by the research topic are more likely to volunteer, creating samples that overrepresent those with direct stakes in the outcomes.
Altruistic Motivation: Some people participate in research to help others or advance scientific knowledge, but this motivation isn’t randomly distributed across the population.
Time and Resources: Voluntary participation requires discretionary time and often involves some cost (transportation, lost wages, childcare), systematically excluding those without such resources.
Comfort with Research: Some people are more comfortable with research settings, authority figures, or formal procedures, affecting who volunteers.
Media Examples and Their Biases
Television call-in polls provide clear examples of voluntary response bias:
Political Issues: When news programs ask viewers to call in with their opinions on political topics, respondents typically have much stronger views than the general population. The results often show more extreme positions than scientific polls of the same topics.
Product Reviews: Online product reviews suffer from voluntary response bias because people with very positive or very negative experiences are much more likely to write reviews than those with neutral experiences.
Comment Sections: Online comment sections on news articles or social media posts systematically overrepresent people with strong opinions and those comfortable expressing views in public forums.
Why Voluntary Response Fails for Inference
Voluntary response sampling creates several problems for statistical inference:
Unrepresentative Opinions: The sample systematically overrepresents extreme views and underrepresents moderate positions, creating a distorted picture of population sentiment.
Unknown Bias Direction: While we know bias exists, we often can’t determine its direction or magnitude, making it impossible to correct for the bias statistically.
Violation of Random Sampling Assumptions: Statistical inference procedures assume some form of random sampling, but voluntary response samples violate these assumptions in fundamental ways.
Non-Generalizable Results: Results from voluntary response samples can only be generalized to other people who would voluntarily respond under similar circumstances—a much more limited population than researchers typically want to study.
The Limitations of Expert Judgment
Some researchers attempt to improve on simple convenience or voluntary response sampling by using their expertise to construct more balanced samples. While this might seem like an improvement, it introduces different types of bias:
Researcher Bias: Even well-intentioned researchers have unconscious biases that affect their sampling decisions, potentially creating systematic patterns they don’t recognize.
Limited Perspective: Individual researchers can’t anticipate all the factors that might affect representativeness, particularly complex interactions between variables they haven’t considered.
Unmeasurable Bias: Because the sampling process isn’t based on known probabilities, there’s no way to measure or adjust for the bias introduced by human judgment.
False Confidence: Expert-constructed samples might appear more representative than convenience samples, leading to overconfidence in results that are still fundamentally biased.
8.6.4. Random Sampling Methods: The Foundation of Valid Inference
Random sampling methods provide the foundation for valid statistical inference by ensuring that every member of the population has a known, non-zero probability of being included in the sample. This probabilistic foundation enables us to quantify uncertainty and make valid generalizations from samples to populations.
The Philosophy of Randomization in Sampling
Randomization in sampling serves the same fundamental purpose as randomization in experimental design: it removes systematic bias and replaces it with known, manageable random variation. When we can’t control all the factors that might affect who ends up in our sample, randomization ensures that these factors balance out across many possible samples.
Key Properties of Random Sampling
Known Selection Probabilities: For every member of the population, we can calculate the probability that they’ll be included in our sample. This probabilistic foundation enables statistical inference.
Unbiased Selection: The sampling process doesn’t systematically favor any particular type of person or outcome. Any biases that remain are due to random chance rather than systematic factors.
Quantifiable Uncertainty: Because we understand the probabilistic mechanism that generated our sample, we can calculate the uncertainty associated with our estimates and test results.
Reproducible Methods: Random sampling procedures can be described precisely and replicated by other researchers, enabling scientific verification of results.
Independence: When properly implemented, random sampling ensures that the selection of one unit doesn’t influence the probability of selecting any other unit (or provides a close approximation to independence).
Simple Random Sampling: The Gold Standard
Simple Random Sampling (SRS) represents the conceptual foundation for most statistical inference procedures. In SRS, every possible unit in the population has exactly the same probability of being selected, and every possible sample of a given size has exactly the same probability of being chosen.
Definition and Properties
A simple random sample of size \(n\) from a population of size \(N\) is selected such that:
Every individual in the population has probability \(\frac{n}{N}\) of being included in the sample
Every possible sample of size \(n\) has probability \(\frac{1}{\binom{N}{n}}\) of being selected
The selection of each unit is independent of the selection of every other unit (approximately, when sampling without replacement from large populations)
Implementation Procedure
Implementing SRS requires a systematic approach:
Step 1: Define the Target Population
This crucial first step often reveals complexities not initially apparent. Questions to address include:
Who exactly belongs to the population of interest?
What are the geographic, temporal, or other boundaries of the population?
How do we handle people who move in or out of the population during the study?
What do we do about people who are temporarily unavailable or difficult to reach?
Step 2: Create a Sampling Frame
The sampling frame is a complete list of all units in the target population, with each unit uniquely identified. This is often the most challenging practical aspect of SRS:
Ideal Requirements: The sampling frame should include every member of the target population exactly once, with current and accurate contact information for each unit.
Practical Challenges: Real sampling frames often have problems like incomplete coverage (missing some population members), overcoverage (including people who shouldn’t be in the population), duplication (the same person listed multiple times), or outdated information.
Common Sampling Frames: Telephone directories, voter registration lists, institutional enrollment records, membership databases, and government records all serve as sampling frames, each with specific strengths and limitations.
Step 3: Assign Unique Labels
Each unit in the sampling frame receives a unique identifier. This might be:
Sequential numbering (1, 2, 3, …, N)
Existing ID numbers (Social Security numbers, student ID numbers, account numbers)
Systematic codes that preserve anonymity while maintaining uniqueness
Step 4: Generate Random Sample
Use a truly random process to select which units will be included:
Random Number Generation: Use computer random number generators, random number tables, or other chance mechanisms to select unit labels.
Without Replacement: Typically, we sample without replacement to ensure each person appears in the sample at most once.
Sample Size Determination: The sample size should be determined based on statistical power analysis, precision requirements, and resource constraints.
Fig. 8.7 Simple Random Sampling Process: From population enumeration to final sample selection
Mathematical Foundation
The mathematical properties of SRS provide the foundation for statistical inference:
Probability of Selection: Each unit has probability \(\frac{n}{N}\) of being selected.
Sample Probability: The probability of obtaining any specific sample of size \(n\) is \(\frac{1}{\binom{N}{n}}\).
Independence Approximation: When \(N\) is large relative to \(n\), sampling without replacement behaves approximately like sampling with replacement, justifying independence assumptions.
Unbiased Estimation: Sample statistics computed from SRS are unbiased estimators of corresponding population parameters.
Implementation in R
R provides built-in functions for implementing simple random sampling:
# Method 1: Sample from a vector of IDs
population_ids <- c("ID001", "ID002", "ID003", ..., "ID9804")
sample_ids <- sample(population_ids, size = 5, replace = FALSE)
# Method 2: Sample indices from a dataset
N <- nrow(dataset) # Population size
n <- 5 # Sample size
sample_indices <- sample(N, size = n, replace = FALSE)
sampled_data <- dataset[sample_indices, ]
# Method 3: Using sampling weights (for unequal probability sampling)
weights <- c(0.1, 0.2, 0.1, 0.3, 0.3) # Must sum to 1
sample_ids <- sample(population_ids, size = 5, replace = FALSE, prob = weights)
Why SRS is “Simple” in Name Only
Despite its name, simple random sampling is often far from simple to implement in practice:
Sampling Frame Challenges: Creating a complete, accurate sampling frame can be extremely difficult for large or dynamic populations.
Cost and Logistics: Contacting randomly selected individuals who might be scattered across large geographic areas can be expensive and time-consuming.
Response Rates: Even with perfect random selection, non-response can introduce bias if certain types of people are systematically less likely to participate.
Rare Populations: When studying rare characteristics or small subgroups, SRS might require enormous sample sizes to capture enough individuals of interest.
Advantages of Simple Random Sampling
Statistical Validity: SRS satisfies the assumptions required for most statistical inference procedures, enabling valid confidence intervals, hypothesis tests, and other analyses.
Unbiased Estimation: Sample statistics provide unbiased estimates of population parameters, meaning they’re correct on average across all possible samples.
Quantifiable Precision: We can calculate exact standard errors and confidence intervals, providing precise measures of uncertainty.
Broad Applicability: SRS works for any population and any characteristic, without requiring advance knowledge of population structure.
Scientific Credibility: Results from well-executed SRS studies are generally accepted as valid by the scientific community and policymakers.
Limitations and Practical Challenges
Cost and Efficiency: SRS can be expensive and inefficient, particularly when the population is geographically dispersed or difficult to access.
Rare Subgroups: Important subgroups might be so rare that they rarely appear in random samples of feasible size.
Sampling Frame Problems: The quality of SRS depends entirely on the quality of the sampling frame, which may be incomplete, outdated, or biased.
Non-Response Issues: Random sampling design can’t control whether selected individuals actually participate, and non-response can introduce serious biases.
Practical Implementation: The ideal of SRS often must be compromised due to practical constraints, potentially affecting the validity of inference procedures.
8.6.5. Stratified Random Sampling: Balancing Representation and Efficiency
When populations contain important subgroups that differ substantially from each other, stratified random sampling provides a method for ensuring adequate representation of all subgroups while potentially improving the precision of population estimates.
The Motivation for Stratification
Simple random sampling works well when the population is relatively homogeneous, but many populations contain distinct subgroups that differ systematically in characteristics relevant to the study. In such cases, SRS faces several potential problems:
Subgroup Representation: Small but important subgroups might be severely underrepresented or even completely missing from random samples.
Precision Loss: If subgroups vary dramatically in the characteristics being studied, combining them in a single analysis can reduce statistical precision.
Administrative Needs: Policy makers and practitioners often need separate information about different subgroups, not just overall population averages.
Efficiency Concerns: Some subgroups might be much more expensive or difficult to sample than others, suggesting that unequal sampling rates might be more efficient.
How Stratified Sampling Works
Stratified sampling addresses these issues by dividing the population into strata (subgroups) based on characteristics known before sampling, then conducting separate random samples within each stratum.
Step 1: Define Strata
Strata should be:
Homogeneous within: Units within each stratum should be as similar as possible with respect to the characteristics being studied.
Heterogeneous between: Different strata should differ substantially from each other on the characteristics of interest.
Based on available information: Stratification variables must be known for the entire population before sampling begins.
Relevant to the research question: Stratification variables should be related to the outcomes being studied.
Manageable in number: Too many strata can create logistical complications and require very large sample sizes.
Step 2: Determine Sample Allocation
Once strata are defined, researchers must decide how many units to sample from each stratum. This allocation decision significantly affects both the cost and precision of the resulting estimates.
Step 3: Conduct Simple Random Sampling Within Strata
Within each stratum, conduct independent simple random samples of the predetermined sizes.
Step 4: Combine Strata Results
Analyze data from each stratum separately and then combine results to produce population estimates, properly accounting for the stratified sampling design.
Fig. 8.8 Stratified Random Sampling: From population stratification to combined analysis
Allocation Methods: Deciding How Much to Sample from Each Stratum
The choice of allocation method significantly affects both the cost and precision of stratified sampling. Different allocation strategies optimize different objectives.
Uniform Allocation: Equal Representation
Uniform allocation samples the same number of units from each stratum, regardless of stratum size. This approach ensures equal precision for estimates within each stratum and is particularly useful when:
The primary goal is comparing subgroups rather than estimating overall population parameters
All strata are considered equally important for policy or scientific purposes
Stratum sizes are roughly similar
The cost of sampling is similar across strata
Implementation: If we want a total sample size of \(n\) with \(M\) strata, we sample \(\frac{n}{M}\) units from each stratum (rounding as necessary to get integer values).
Advantages: Simple to implement, ensures adequate representation of all subgroups, provides equal precision for subgroup estimates.
Disadvantages: May be inefficient for estimating overall population parameters, oversamples small strata and undersamples large strata relative to their population proportions.
Proportional Allocation: Maintaining Population Structure
Proportional allocation samples from each stratum in proportion to the stratum’s size in the population. This approach maintains the population’s natural structure in the sample.
Implementation: For stratum \(i\) with \(m_i\) units in a population of size \(N = \sum_{i=1}^M m_i\), sample size is:
where \(n\) is the total desired sample size.
Advantages: - Provides unbiased estimates of population parameters with relatively simple analysis - Maintains representativeness across subgroups - Often more cost-effective than uniform allocation - Results can be analyzed as if they came from simple random sampling
Disadvantages: - Small strata may have very small sample sizes, limiting the precision of subgroup estimates - May not be optimal for detecting differences between subgroups - Doesn’t account for different levels of variability within strata
Example: In a population with 60% urban residents and 40% rural residents, a proportionally allocated sample of 1000 would include 600 urban and 400 rural residents.
Variation Allocation: Optimizing for Precision
Variation allocation (also called optimal allocation) determines sample sizes based on the variability within each stratum. Strata with higher variability receive larger sample sizes because they require more observations to achieve the same level of precision.
Implementation: Sample size for stratum \(i\) is proportional to the standard deviation within that stratum:
where \(s_i\) is the standard deviation of the variable of interest in stratum \(i\).
Advantages: - Minimizes the standard error of population estimates for a given total sample size - Allocates resources efficiently by focusing sampling effort where variability is highest - Can substantially improve precision compared to proportional allocation when strata have very different variability levels
Disadvantages: - Requires advance knowledge of variability within each stratum, typically from pilot studies or previous research - May result in very unequal sample sizes across strata - Optimization is specific to one variable; if multiple variables are important, optimal allocation for one may be poor for others - More complex to implement and analyze
When to Use Variation Allocation: This approach is most valuable when: - Precise population estimates are the primary goal - Previous data or pilot studies provide reliable estimates of within-stratum variability - Strata differ substantially in their variability - The cost of additional sampling is high enough to justify the complexity
Optimal Allocation: Balancing Multiple Objectives
Optimal allocation extends variation allocation by incorporating cost considerations along with precision objectives. This approach recognizes that sampling costs often vary dramatically across strata.
Implementation: The optimal allocation balances precision gains against cost differences:
where \(c_i\) is the cost of sampling one unit from stratum \(i\).
Cost Considerations: Costs might vary across strata due to: - Geographic dispersion (rural areas might be more expensive to reach) - Accessibility (some populations require special recruitment efforts) - Response rates (groups with lower response rates effectively cost more per completed interview) - Language or cultural barriers (requiring specialized staff or procedures) - Institutional requirements (some settings require additional permissions or procedures)
Advanced Optimization: In practice, optimal allocation often involves more complex optimization that considers: - Multiple variables of interest with different importance weights - Non-response patterns that vary across strata - Budget constraints that limit total sampling effort - Minimum sample size requirements for subgroup analysis - Political or administrative requirements for subgroup representation
Advantages of Stratified Sampling
Guaranteed Subgroup Representation: Unlike SRS, stratified sampling ensures that all important subgroups are represented in the sample with adequate sample sizes for meaningful analysis.
Improved Precision: When strata are more homogeneous than the overall population, stratified sampling typically provides more precise estimates than SRS of the same size.
Separate Subgroup Analysis: Stratified designs naturally provide data for analyzing subgroups separately, meeting the needs of researchers and policymakers who need subgroup-specific information.
Flexible Resource Allocation: Different allocation strategies allow researchers to optimize for different objectives (precision, cost, subgroup representation) depending on study goals.
Reduced Sampling Costs: When strata are geographically or administratively clustered, stratified sampling can reduce travel and coordination costs.
Administrative Benefits: Organizations often prefer stratified designs because they ensure representation of all important constituencies or organizational units.
Implementation Challenges and Considerations
Stratification Variable Selection: Choosing appropriate stratification variables requires balancing statistical efficiency with practical feasibility:
Variables should be strongly related to the outcomes of interest
Information must be available for the entire population before sampling
Too many stratification variables create too many strata with small sample sizes
Variables should create strata of reasonable minimum sizes
Boundary Issues: Deciding exactly how to define strata can be challenging: - Where to set cutpoints for continuous variables (age groups, income levels) - How to handle units that could belong to multiple strata - What to do about units with missing information on stratification variables - How to handle units that change strata during the study period
Analysis Complexity: Stratified sampling requires more complex analysis procedures: - Need to account for stratification in calculating standard errors and confidence intervals - Must use appropriate statistical software that handles survey design features - May need specialized expertise for proper analysis - Results presentation becomes more complex when reporting subgroup estimates
Sample Size Planning: Determining appropriate sample sizes for stratified designs requires more complex calculations: - Must consider precision requirements for both overall estimates and subgroup estimates - Need to balance competing objectives when subgroup precision conflicts with overall precision - Must account for different allocation strategies in power analyses - Should consider potential non-response patterns that might vary across strata
Quality Control: Stratified sampling requires additional quality control measures: - Ensuring proper implementation of sampling procedures within each stratum - Monitoring response rates and data quality across strata - Checking that stratification was implemented correctly - Verifying that analysis properly accounts for the stratified design
When Stratified Sampling is Most Valuable
Stratified sampling provides the greatest benefits when:
Subgroups Differ Substantially: When important subgroups have very different characteristics, stratification can greatly improve precision and ensure adequate representation.
Subgroup Analysis is Important: When researchers need reliable estimates for specific subgroups, not just overall population averages.
Cost Varies Across Subgroups: When some subgroups are much more expensive or difficult to reach than others, stratification allows for efficient resource allocation.
Administrative Structure Exists: When the population naturally divides into administrative or geographic units that can serve as strata.
Previous Information Available: When prior research or administrative data provides information about subgroup characteristics that can guide stratification and allocation decisions.
Stratified sampling represents a sophisticated approach to sampling design that can substantially improve both the efficiency and validity of research studies. However, it requires careful planning, implementation, and analysis to realize these benefits. When properly executed, stratified sampling provides a powerful tool for ensuring that research results are both statistically precise and representative of all important population subgroups.
8.6.6. Bringing It All Together
Understanding sampling design provides the crucial link between data collection and statistical inference. While experimental design principles ensure that our studies can establish causal relationships, sampling design principles ensure that those relationships generalize to the populations we want to understand. Together, these design principles create the foundation for reliable, actionable research findings.
As we move toward actual statistical inference methods in subsequent chapters, remember that every confidence interval, hypothesis test, and regression analysis depends on assumptions about how the data were collected. Poor sampling design can invalidate even the most sophisticated statistical analysis, while good sampling design enables simple statistical methods to produce profound insights about populations and processes we care about.
TAKEAWAY BOX IS TOO LONG!!
Key Takeaways 📝
Sampling design determines inferential validity: The methods used to select participants fundamentally determine whether statistical inference procedures are valid and whether results can be generalized.
Non-random sampling creates systematic bias: Convenience and voluntary response sampling are simple and cheap but introduce biases that cannot be measured or corrected, limiting their use to preliminary investigations.
Random sampling enables valid inference: Methods like simple random sampling and stratified random sampling provide the probabilistic foundation required for statistical inference procedures.
Simple random sampling is the gold standard: SRS gives every population member equal selection probability and satisfies the IID assumptions required for most statistical procedures.
Stratified sampling improves efficiency and representation: When populations contain distinct subgroups, stratified sampling can improve precision while ensuring adequate subgroup representation.
Population Definition Challenge: You want to study “social media usage among teenagers” but need to define your target population precisely.
What specific decisions must you make about age boundaries, geographic scope, and inclusion criteria?
How might different population definitions affect your sampling approach?
What sampling frame could you realistically use for this population?
What groups might be systematically excluded from common sampling frames?
Sample Size and Precision: Explain why stratified sampling with proportional allocation often provides more precise estimates than simple random sampling of the same total size. Use a concrete example to illustrate your explanation.
Real-World Implementation: Choose a research question that interests you and design a complete sampling plan that includes:
Clear definition of the target population
Identification of an appropriate sampling frame
Choice between simple random sampling and stratified sampling with justification
If using stratified sampling, specification of strata and allocation method
Discussion of likely implementation challenges and how you would address them
Voluntary Response Analysis: A local newspaper publishes an online survey asking readers about their support for a proposed tax increase. The survey receives 2,847 responses, with 73% opposing the tax.
What type of sampling method is this?
Why might these results not represent the views of all local residents?
What specific types of people might be overrepresented in this sample?
How might the results differ if the same question were asked in a scientific poll using random sampling?
Sampling Frame Evaluation: A researcher wants to study the health behaviors of adults in a mid-sized city and is considering three possible sampling frames:
Telephone directory listings
Voter registration records
Driver’s license records
For each sampling frame, identify what groups would likely be underrepresented or overrepresented, and discuss how these biases might affect a study of health behaviors.Allocation strategies serve different objectives**: Uniform allocation ensures equal subgroup representation, proportional allocation maintains population structure, and variation allocation optimizes precision.
Implementation challenges are substantial: Both SRS and stratified sampling face practical challenges including sampling frame construction, cost management, and non-response handling.
Design complexity must match study objectives: The choice between different sampling methods should depend on research goals, resource constraints, and the structure of the target population.
Exercises
Sampling Method Classification: For each scenario below, identify the sampling method being used and explain its major strengths and limitations:
A political pollster calls every 50th person in the phone directory.
A news website asks visitors to vote in an online poll about a current issue.
A health researcher obtains a list of all registered patients at area hospitals and randomly selects 500 for a nutrition study.
A marketing company recruits participants for a focus group by approaching shoppers at a mall entrance.
Simple Random Sampling Implementation: You want to conduct a simple random sample of 100 students from a university with 25,000 enrolled students.
Describe the complete procedure you would use, including how you would obtain a sampling frame.
What practical challenges might you encounter?
How would you handle students who are selected but don’t respond?
Write R code to select the sample assuming you have a dataset called student_roster.
Stratified Sampling Design: A state education department wants to study teacher job satisfaction using a sample of 600 teachers. The state has 120 elementary schools, 80 middle schools, and 60 high schools.
Explain why stratified sampling might be preferable to simple random sampling for this study.
Calculate sample sizes for each stratum using proportional allocation.
Calculate sample sizes using uniform allocation.
Discuss the advantages and disadvantages of each allocation method for this study.
Bias in Non-Random Sampling: A researcher wants to study exercise habits among adults and recruits participants by posting flyers at gyms, health food stores, and medical clinics.
What type of sampling method is this?
Identify at least three specific ways this sampling method might create bias.
Explain why the bias cannot be corrected through statistical analysis.
Suggest an alternative sampling approach that would reduce these biases.
Allocation Method Comparison: A market research company is studying smartphone usage across three age groups: 18-30 (40% of population), 31-50 (35% of population), and 51+ (25% of population). Previous research suggests that usage variability is highest in the 18-30 group and lowest in the 51+ group.
Calculate sample sizes for each age group using proportional allocation (total n = 900).
Explain when you might prefer uniform allocation instead.
Describe how variation allocation would differ from proportional allocation.
What additional information would you need to implement optimal allocation?