Complex survey analysis
What question does this help answer?
What proportion of the target population has an outcome, or how do population means and associations differ, when the sample used unequal selection probabilities, stratification or clustering?
When to use it
Use design-aware methods for probability surveys such as NHANES and for your own multistage samples. Obtain the release-specific documentation before choosing weights or combining waves; an interview, examination and laboratory subsample may represent different sets of eligible participants.
When not to use it
Do not attach arbitrary weights to a convenience survey and assume it becomes representative. A genuinely simple random sample can use simpler methods. Analyses seeking causal effects still need a separate identification strategy; sampling weights and treatment-propensity weights serve different purposes.
How it works in plain language
Weights describe each respondent's contribution to the target population. Strata and primary sampling units describe the sampling structure used to estimate uncertainty. Declare that structure in survey-aware software, then estimate the specified proportion, mean or regression. Follow the survey provider's choice of Taylor linearization or replicate-weight variance estimation.
What a complex survey design looks like
Key assumptions and design requirements
- Name the target population, survey release, outcome and time period before choosing the analytic sample.
- Use the documented weight appropriate to all included measurements; retain the corresponding strata and cluster identifiers.
- Check extreme weights, sparse domains, design degrees of freedom and the provider's reliability criteria; participant count alone does not establish precision.
- Distinguish unit nonresponse from missing items. State the missingness assumption and justify complete-case analysis, imputation or another method rather than assuming the survey weight resolves all missingness.
Subgroups are domains of the original sample
For an age-defined subgroup, declare the complete eligible survey design first and use the software's domain or subpopulation method. In R this can mean subsetting the survey design object; in Stata it can mean the survey subpopulation option. Filtering the raw data first may discard information needed for variance estimation.
Clinical example
A team wants the prevalence of elevated blood pressure among adults aged 40–64 in a specified NHANES release. They confirm the examination-weight rules for that release, retain the design variables, flag the age domain and calculate a weighted prevalence with a design-based confidence interval. They report the unweighted respondent count separately from the population estimate. This is a proposed workflow, not an analysis that has been run.
What to watch out for
A sampling weight alone does not account for strata and clusters, and deleting everyone outside a subgroup before declaring the survey design can give incorrect uncertainty. Survey weights do not remove all nonresponse bias or establish causality.
Before using this in your workspace
Record the dataset release, weight variable, strata, sampling units, domain definition and missing-data plan with the relevant aim. If the design variables or documentation are unavailable, resolve that limitation before presenting population estimates as survey-valid.
- Important considerations when analyzing health survey data collected using a complex sample design (opens in a new tab) · Am J Public Health · doi 10.2105/AJPH.2013.301515
- National health and nutrition examination survey: analytic guidelines, 1999-2010 (opens in a new tab) · Vital Health Stat 2
- NHANES Tutorials: Variance Estimation (opens in a new tab) · CDC / National Center for Health Statistics