Where this lesson starts

HSCI 230 taught the vocabulary of validity. Its Lessons 7 to 11 defined selection bias, information bias and confounding, named their common forms, and showed how each appears on a causal diagram. In a course on designing research, the vocabulary leads to two practical questions: which decisions in a study protocol prevent each bias, and how the protocol can show, with numbers, how large the bias that remains could be. A protocol often gathers the answers in a subsection titled "threats to validity and mitigation", and this lesson supplies the tools that subsection uses.

Selection bias is the first threat a protocol plans for. Its direction and size can be computed from sampling fractions, it can be bounded with assumed fractions when the true ones are unknown, and it is kept small by decisions about the sampling frame, response and retention. A one-screen recap of all three threats, and a short diagnostic check on the difference between a confounder, a mediator and a collider, come first.

Recap: validity and the three threats

A study is internally valid when the association it reports, apart from random error, equals the association in the source population, the population that actually produced the participants. It is externally valid when that association also holds in a target population beyond the source. Bias is a systematic difference between the estimate and the truth, and a larger sample does not remove it. Three threats account for most bias in observational research, and HSCI 230 treated each of them in depth. Table 7.1 is the whole recap. Where a definition is unfamiliar, the HSCI 230 lesson named in the third column is the place to review it.

Table 7.1. The threats to internal validity, how each arises, where HSCI 230 teaches it, and what this lesson adds.

The pre-course survey for this course found that most students could define these threats, and that about one student in four still described a variable on the causal pathway as a confounder. The check below targets that confusion. It is ungraded and takes two or three minutes.

The four items turn on the direction of the arrows in a causal diagram. A confounder is a common cause of the exposure and the outcome and is controlled in the protocol, a mediator lies on the path between them and is left out of the adjustment set for a total effect, and a collider is a common effect, such as completing the survey in the third item, on which an analysis of completers conditions. The next part sets out the protocol decisions that address each threat.

Bias is decided when the protocol is written

Most of the decisions that determine whether a study is biased are made before any data are collected. The sampling frame and the recruitment method decide who can enter the study. The instruments, their mode of administration and the timing of measurement decide how much misclassification there will be. The list of variables to be measured decides which confounders can ever be controlled, because an analysis can adjust only for what was recorded. Restriction and matching are design choices, and stratification and adjustment, although carried out in the analysis, must be planned in the protocol so that the necessary variables are collected and the analysis is not chosen after the results are seen. Table 7.2 pairs each threat with the protocol decisions that prevent it and the analysis steps that check or correct it.

Table 7.2. Protocol decisions and analysis steps for each threat to validity.

The protocol therefore carries a plan for each threat, often written in a subsection titled threats to validity and mitigation. Each section of this lesson ends with the decisions it contributes to that subsection, and Section 4 shows a complete example.

The first threat in such a plan is selection bias. The next part shows how its direction and size can be computed from the sampling fractions of the source population.

Quantifying selection bias with sampling fractions

HSCI 230 described selection bias as conditioning on a common effect. When both the exposure and the outcome influence who takes part, restricting the analysis to participants creates an association between them, or distorts one that exists. Figure 7.1 shows the structure in general and in the study that serves as this lesson's running example, the Campus Connection Study, a worked example that this lesson shares with Lesson 8. That study is a survey of loneliness and depressive symptoms among students aged 18 to 29 at five public post-secondary institutions in British Columbia.

The diagram shows that selection bias is possible. Sampling fractions show how large it is and in which direction it runs. Imagine the source population arranged in the usual two-by-two table, with exposure in the columns and disease in the rows. Upper-case letters denote counts in the source population and lower-case letters denote the counts that end up in the study. Table 7.3 sets out the source population in this form.

Table 7.3. The source population as a two-by-two table of exposure and disease, with counts in upper-case letters.

Each cell has a sampling fraction, the proportion of that cell of the source population that entered the study and stayed in it to the analysis. Equation 7.1 defines the four fractions, each as the study count divided by the source count for one cell.

Equation 7.2 combines the four fractions into a single quantity, the odds ratio of the sampling fractions (OR_sf). The observed odds ratio equals the source-population odds ratio multiplied by the odds ratio of the sampling fractions:

Three consequences follow. When OR_sf equals 1, the observed odds ratio is unbiased, even when the four fractions differ from each other. When OR_sf is above 1, the observed odds ratio is larger than the true one, which for a harmful exposure is a bias away from the null value of 1. When OR_sf is below 1, the observed odds ratio is smaller, which for a harmful exposure is a bias toward the null. The same information can be expressed as sampling odds: the selection odds of cases to non-cases among the exposed (sf₁₁ ÷ sf₂₁) and among the unexposed (sf₁₂ ÷ sf₂₂). There is no selection bias in the odds ratio when these two odds are equal. The result applies exactly to the odds ratio. For a risk ratio it holds approximately when the outcome is rare.

HSCI 341 Lesson 2 Section 3 called selection that depends on the outcome the formal criterion for selection bias, and that criterion holds for a descriptive estimate such as a prevalence, which is biased whenever cases and non-cases take part at different rates. For an odds ratio the criterion is an odds ratio of the sampling fractions different from 1, which Worked Example 7.1 computes: selection that depends on the outcome biases the odds ratio only when it depends on the outcome differently in the exposed and the unexposed.

A low response rate is therefore not, by itself, evidence of selection bias. A meta-analysis of studies that measured both found only a weak relationship between survey response rates and the size of non-response bias (Groves & Peytcheva, 2008). What matters is whether response depends on the exposure and the outcome jointly, which is what OR_sf measures.

Worked Example 7.1 applies Equation 7.2 to non-response in a source population of 10,000 people, first when non-response depends on exposure alone and then when it also depends on disease. Calculator 7.1, which follows it, applies Equation 7.2 to any four sampling fractions and an observed odds ratio; its presets include the daycare study of Worked Example 7.2 and the two Campus Connection cases of Worked Example 7.3.

Bounding the bias when the fractions are unknown

The true sampling fractions are almost never known, because the source population is not observed cell by cell. The formula is still useful as a sensitivity analysis. The investigator assigns plausible fractions to the four cells, computes OR_sf, and divides the observed odds ratio by it to see what the estimate would have been without the selection. Assigning one set of values is a deterministic analysis; drawing the fractions from distributions and repeating the calculation many times is a probabilistic one, which Section 4 describes as part of quantitative bias analysis. Worked Example 7.2 applies a deterministic analysis to a case-control study of daycare attendance and childhood respiratory disease, and the daycare preset of Calculator 7.1 reproduces it.

Selection bias can be corrected fully in only two situations. In the first, the factors that drive selection are measured, precede both exposure and outcome, and have a known distribution in the source population, so that they can be handled like confounders, for example by weighting. In the second, the investigator can identify and measure a variable that carries the whole dependence of selection on exposure and outcome (sometimes called a bias breaker) and can obtain unbiased estimates of its distribution in the source population. Both situations require information collected by design, which is why the protocol decisions below include what to record as well as how to recruit.

Protocol decisions that prevent selection bias

The aim of the design is to keep OR_sf close to 1: selection may depend on exposure or on outcome, but it should not depend on their combination. The decisions fall into four groups.

The four tabs below take the groups in turn: the sampling frame, response, retention, and what to record. The Response tab opens with Box 7.1, which recalls the strategies for raising response from HSCI 341 Lesson 3.

Worked Example 7.3 applies the four groups of decisions to the Campus Connection Study and then uses Equation 7.2 to bound the selection bias that its expected response rate of 15% leaves room for.

Section 1 has shown that selection biases an odds ratio only when participation depends jointly on the exposure and the outcome, that the odds ratio of the sampling fractions measures the size and direction of that bias, and that a protocol keeps it small through its sampling frame, recruitment message, contact schedule, retention procedures and records of the people who do not take part. The key takeaways below summarize these points. The reflection that follows applies the sampling fractions to a study that recruits through social media, and the knowledge check closes the section.

From the test to the study

HSCI 341 Lesson 5 described a measurement in terms of its sensitivity (Se), the probability that a person who truly has a characteristic is classified as having it, and its specificity (Sp), the probability that a person who truly lacks it is classified as lacking it. HSCI 230 Lesson 9 showed the consequence for a study: misclassification that is the same in the groups being compared (non-differential) tends to pull a measure of association toward the null, and misclassification that differs between them (differential) can push it either way. Both statements can be made quantitative, and each calculation corresponds to a protocol decision: the table a study would observe, the correction of an observed table, the validation sub-study that supplies the sensitivity and specificity a correction needs, and a sample size large enough that misclassification does not leave the study underpowered.

Computing the misclassified table

The first calculation shows what non-differential misclassification of a binary exposure does to the two-by-two table a study observes. Equation 7.3 states the condition that defines non-differential error.

Misclassification of exposure is non-differential when the sensitivity and specificity of the exposure measurement are the same among cases and non-cases:

Each observed cell is then a mixture of correctly classified people and people who belong in the other exposure column. Among the cases, the observed exposed count contains the truly exposed cases who were correctly classified and the truly unexposed cases who were wrongly classified as exposed. The same mixing happens among the non-cases. Table 7.4 gives each observed count as a mixture of two true counts, weighted by Se_E and Sp_E.

Table 7.4. Observed counts under non-differential exposure misclassification, in terms of the true counts, Se_E and Sp_E.

Because the mixing blends the exposed and unexposed columns in the same proportions for cases and non-cases, the two columns become more alike and the odds ratio moves toward 1, provided Se_E + Sp_E is greater than 1. Misclassifying exposure moves people between the exposure columns and leaves the case and non-case totals unchanged, which is the property that makes back-correction possible. Error rates that look modest, of 10% to 20%, can attenuate an association considerably. Worked Example 7.4 shows the size of that attenuation in a study whose true table is known, and Calculator 7.2 reproduces the example, with a second preset for better measurement, so that other values of Se_E and Sp_E can be tried.

Misclassification of the outcome

Outcome misclassification has an asymmetry that matters for design. Suppose the true risk of disease is 0.10 among the exposed and 0.05 among the unexposed, a risk ratio of 2.0. If the outcome measure misses some cases (sensitivity 0.80) but never labels a healthy person as a case (specificity 1), both observed risks are multiplied by 0.80, giving 0.08 and 0.04, and the risk ratio stays at 2.0. If instead the measure finds every case (sensitivity 1) but wrongly labels 5% of healthy people as cases (specificity 0.95), the observed risks become 0.145 and 0.0975, and the risk ratio falls to 1.49. False positives add roughly the same number of spurious cases to each group, which dilutes a ratio, and the dilution is greatest when the outcome is rare. A protocol therefore favours a highly specific outcome definition, for example a confirmed diagnosis or a validated case-finding algorithm, even at some cost in sensitivity.

In a case-control study the same logic applies to the case definition: false-positive cases bias the odds ratio toward the null, so diagnoses are confirmed before enrolment. Sensitivity and specificity estimated in the general population cannot be used directly to correct a case-control study, because sampling cases and controls at different rates changes the proportions of true and false positives among those enrolled.

Differential misclassification

When the accuracy of the exposure measurement differs between cases and non-cases, or the accuracy of the outcome measurement differs between exposed and unexposed people, the misclassification is differential:

Equation 7.4 states the condition for differential exposure misclassification. The resulting bias can run in either direction and its size cannot be predicted without knowing the error rates in each group. Recall bias, in which cases search their memories for exposures more thoroughly than controls, is the familiar example, and HSCI 230 Lesson 9 described observer and detection bias as others. Differential error cannot be corrected by a single pair of sensitivity and specificity values, so the protocol concentrates on preventing it. Figure 7.2 contrasts the two kinds of misclassification, and Table 7.5 lists the protocol decisions that prevent differential error.

Table 7.5. Protocol decisions that prevent differential misclassification.

The second decision in Table 7.5 holds the mode of administration constant. Box 7.2 gives background on modes of administration and on mode effects, and explains why a protocol uses one mode for every participant.

Non-differential misclassification of a binary exposure moves the odds ratio toward the null by an amount that Table 7.4 allows the analyst to compute, while differential misclassification has to be prevented by design. The calculation in Table 7.4 can also be run in reverse, which the next part shows.

Back-correcting an observed table

When the sensitivity and specificity of the exposure measurement are known, or can be assumed, the true table can be recovered from the observed one. Because misclassifying exposure leaves the non-case total m₀ unchanged, the true count of exposed non-cases is:

Equation 7.5 recovers the exposed non-cases. The same reasoning applied to the case total m₁ gives Equation 7.6 for the exposed cases.

The unexposed counts follow by subtraction from the totals. Applied to the observed table of Worked Example 7.4 with the true values Se_E = 0.80 and Sp_E = 0.90, the formulas return the original 90, 70, 210 and 630, and the odds ratio of 3.86. Applied to the same observed table with Se_E = 0.90 and Sp_E = 0.95, they return an odds ratio of 3.03. The corrected estimate depends heavily on the assumed error rates, and for some combinations the formulas give a zero or negative cell, which shows that the assumed values are incompatible with the data. For these reasons the correction is reported over a range of plausible values, and those values come from a validation study in a population like the study population. Calculator 7.3 contains both versions of the example.

A back-corrected table is only as accurate as the sensitivity and specificity used to compute it. The next part describes how a protocol plans the validation sub-study that supplies those values.

Planning a validation sub-study

A validation sub-study measures a subset of participants twice: once with the study's instrument and once with a reference standard that is accepted as correct or close to it. It supplies the sensitivity and specificity (or, for a continuous measure, the relationship between measured and true values) that a correction needs. Two directions of estimation should be kept apart. A validation estimate runs from truth to measurement, for example the probability of being classified as exposed among those who are truly exposed, which is the sensitivity. A correction runs from measurement to truth, for example the probability of being truly exposed among those classified as exposed, which is a predictive value and depends on prevalence as well as on the error rates. Equation 7.7 sets the two directions side by side.

A protocol plans the sub-study in five steps. The accordion below sets out each step, from the choice of a reference standard to the use of the results.

Worked Example 7.5 applies the five steps to the PHQ-2 in the Campus Connection Study, including the sample size calculation of the fourth step.

Regression calibration in brief

For a continuous exposure measured with error, such as dietary intake from a food-frequency questionnaire, the analogue of back-correction is regression calibration (Rosner, Willett & Spiegelman, 1989). In the validation sub-study, the reference measurement is regressed on the study measurement (and on any other covariates in the main model). The fitted equation is then used to predict a calibrated exposure value for every participant in the main study, and the outcome model is fitted with the calibrated values in place of the measured ones. The slope moves away from the attenuated value toward the true association, and its standard error must be adjusted for the uncertainty in the calibration step, for example by bootstrapping. The method assumes that the measurement error is non-differential and that the calibration equation transfers from the sub-study to the whole sample. A protocol that plans regression calibration therefore plans an internal validation sub-study with both measurements, and HSCI 410 provides the regression skills to carry it out.

Validation and correction address the bias that misclassification causes. Misclassification also reduces power, and the last part of this section turns to the sample size a protocol needs when its outcome measure is imperfect.

Inflating the sample size for misclassification

Sample size formulas assume that the proportions entered into them are the proportions the study will observe. With an imperfect outcome measure, the observed proportions are:

With non-differential error the difference p₁′ − p₂′ is smaller than p₁ − p₂, so a study sized with the true proportions would be underpowered. Box 7.3 recalls how HSCI 341 Lesson 2 computed the sample size for comparing two proportions.

Misclassification enters through the last of those inputs, so the protocol computes the observed proportions first and enters them into the two-proportion formula. Worked Example 7.6 applies Equation 7.8 to a cohort comparison and to the Campus Connection Study, and Calculator 7.4 reproduces both cases, with a third preset for a perfect test.

Two further inflation factors appear in protocols. The first is the design effect from unequal weighting. When respondents carry weights that vary, because strata were sampled at different rates or because the weights were adjusted for non-response as in the Campus Connection Study, the effective sample size is smaller than the number of respondents. Kish's approximation gives this design effect as 1 + CV², where CV is the coefficient of variation of the weights, and the required sample size is multiplied by it. The second is the variance inflation factor, 1 ÷ (1 − R²), where R² is the proportion of the variance of the exposure explained by the other covariates in a regression model. It is the factor in the multicollinearity formula of HSCI 230 Lesson 11 Section 2, and it applies only when a regression-adjusted analysis, taught in HSCI 410, is planned: adjustment for covariates that are correlated with the exposure inflates the variance of the exposure coefficient by this factor, so the sample size is multiplied by it.

Section 2 has shown that misclassification affects a study in two ways. It biases the estimate, which the protocol limits by measuring in the same way in every group and assesses with back-correction and a validation sub-study, and it reduces power, which the protocol offsets by sizing the study with the observed proportions. The key takeaways below summarize these calculations. The reflection that follows applies the sample size inflation and the planning of a validation sub-study to a screen for generalized anxiety, and the knowledge check closes the section.

Confounding as a design problem

HSCI 230 Lessons 7 and 11 defined confounding, showed it on causal diagrams, and illustrated Simpson's paradox, in which an association reverses when a third variable is taken into account. In a protocol, the control of confounding is a sequence of decisions. The investigator first decides which variables are confounders, using a causal diagram. For each one, the protocol then states whether it will be controlled by restriction or matching at the design stage, or measured and controlled by stratification or adjustment in the analysis. The stratified analysis that this lesson carries out by hand, the Mantel-Haenszel estimator, completes the sequence, together with two phenomena that can be mistaken for confounding in a stratified table: effect modification and the non-collapsibility of the odds ratio.

Confounders in brief

A confounder is a common cause of the exposure and the outcome, or a proxy for such a cause, that is not on the causal pathway from the exposure to the outcome. This is the definition the diagnostic check in Section 1 used. The three classical conditions follow from it. A confounder is associated with the outcome independently of the exposure, because it causes the outcome or is a proxy for a cause. It is associated with the exposure in the source population, because it causes the exposure or shares a cause with it. And it is not caused by the exposure, so it does not lie on the path from exposure to outcome, and it is not a consequence of the outcome. A mediator satisfies the first two conditions and fails the third. The three conditions are necessary but not sufficient. A variable measured before the exposure that is caused by two unmeasured factors, one that also causes the exposure and one that also causes the outcome, meets all three, yet it is a collider, and adjusting for it opens a non-causal path between the exposure and the outcome (M-bias). The causal diagram, which shows the direction of each arrow, settles the question. Two further distinctions affect what a protocol states. A population confounder is a variable known from previous research to confound the association in the target population, and the protocol plans to control it whatever the study data show. A sample confounder appears to confound in the study data only, and the protocol states in advance how such variables will be judged, so that the adjustment set is not built from every variable that happens to change the estimate.

Worked Example 7.7 introduces a study of carriage of Streptococcus pneumoniae and childhood respiratory disease in which a measured variable, RSV infection, meets these conditions.

In the STREP example, RSV is associated with both the exposure and the outcome, and the crude odds ratio overstates the association found within each RSV stratum. Once a protocol has identified such a confounder, it has to decide how to control it, and the next part sets out the four approaches available.

Four ways to control a confounder

A protocol chooses, for each confounder, one of four approaches or a combination of them. The first two act on who is studied; the last two act on how the data are analyzed and require that the confounder be measured well. Table 7.6 compares the four approaches by how each works, what it costs and what the protocol states about it.

Table 7.6. Four approaches to controlling a confounder.

Restriction

Restriction prevents confounding by admitting only one level of the confounder. Restricting a study of cervical cancer to women removes confounding by gender completely. Restricting a study to people aged 60 to 69 reduces confounding by age without removing it, because age still varies within the band. When the confounder is binary, admitting the low-risk level is usually preferred, because the exposure effect is then easier to detect and effect modification by the restricted variable is avoided. In the STREP example, restricting to children without RSV gives an odds ratio of 3.21. That estimate is free of confounding by RSV and describes only children without RSV, which is the cost of restriction: the protocol trades the breadth of its conclusions for their internal validity.

Matching

Matching is the second of the two design-stage approaches in Table 7.6. Box 7.4 recalls how HSCI 230 introduced matching and overmatching.

Matching makes the distribution of a confounder the same in the groups being compared. Its consequences differ by design. The three tabs below describe matching in cohort studies, in case-control studies and in matched pairs, and the third tab gives the matched odds ratio and McNemar's test in Equation 7.9.

Restriction and matching act on who is studied, and both depend on knowing which variables are confounders, since matching on a variable that is associated with the exposure but is not a confounder is overmatching. The causal diagram supplies that knowledge, and the next part shows how it determines the adjustment set.

Choosing the adjustment set from a causal diagram

A causal diagram, or directed acyclic graph (DAG), records the investigator's assumptions about what causes what. HSCI 230 Lesson 7 taught how to draw one and how conditioning on a collider or a mediator biases an estimate. For a protocol, the diagram has a specific job: it determines the adjustment set, the variables that must be measured and controlled to estimate the effect of the exposure, using the backdoor criterion (Greenland, Pearl & Robins, 1999).

1. Draw the diagram from subject-matter knowledge before seeing the study data, including unmeasured variables.
2. Set aside every arrow that leaves the exposure, and every variable that the exposure causes.
3. List the remaining paths that connect the exposure to the outcome. Each one that begins with an arrow into the exposure is a backdoor path.
4. Choose a set of measured variables that blocks every backdoor path, without including colliders on those paths (unless the path is also blocked by another variable in the set) and without including descendants of the exposure.

Figure 7.3 presents the backdoor criterion as a story in seven scenes, from the causal front door to the closing of the backdoor by conditioning on the confounder. Figure 7.4 then applies the four steps to a simplified causal diagram for the Campus Connection Study.

In this simplified diagram there are two backdoor paths from loneliness to depressive symptoms, one through financial strain and one through gender, and the adjustment set {financial strain, gender} blocks both. Perceived social support is caused by loneliness, so it is excluded: adjusting for it would estimate only the part of the effect that does not pass through support. Completing the survey is a collider; it cannot be adjusted away, and Section 1 dealt with it as selection bias. Living arrangement affects loneliness and, in this diagram, affects depressive symptoms only through loneliness, so it opens no backdoor path and is not needed for control of confounding. If the investigators believed that living arrangement also affected depressive symptoms directly, they would draw that arrow and add the variable to the set. The protocol's real diagram contains more variables than this one. Its value lies in making each of these judgements visible to reviewers before the data are collected.

Change-in-estimate as a supporting check

The change-in-estimate approach compares the crude odds ratio (OR_c) with the odds ratio adjusted for a candidate variable (OR_a). By the common convention, a change of more than 10 percent is taken as evidence that the variable confounds the association, and a protocol that uses another threshold names it in advance as a choice. Three rules keep the check honest. The crude value is the baseline. For ratio measures the change is computed on the log scale, (ln OR_c − ln OR_a) ÷ ln OR_c, because a halving and a doubling are changes of the same size on that scale. For a logistic model this is the percentage change in the exposure coefficient, because the coefficient is the log odds ratio, and it is the form that HSCI 410 Lesson 3 uses. And the check is applied only to variables that the diagram allows to be confounders, because adjusting for a mediator or a collider also changes the estimate. In the STREP example the Mantel-Haenszel estimate in Worked Example 7.8 is 3.36, and the change from the crude value is (ln 5.44 − ln 3.36) ÷ ln 5.44 = 28% on the log scale, so RSV meets the criterion and the diagram supports it as a confounder. A protocol names the diagram as the primary basis for the adjustment set, states the threshold it will use for variables whose role is uncertain, and fixes both before the analysis.

The change-in-estimate check needs an adjusted estimate to compare with the crude one. For one or two categorical confounders that estimate usually comes from a stratified analysis, which the next part carries out by hand.

Stratified analysis with the Mantel-Haenszel estimator

The Mantel-Haenszel procedure stratifies the data by the levels of the confounder, computes the odds ratio within each stratum, and pools the strata into one adjusted estimate with a test of whether that estimate differs from 1 (Mantel & Haenszel, 1959). In practice it is accompanied by a test of whether the stratum-specific values are similar. It is the analysis that a protocol usually plans when it has one or two categorical confounders. Equation 7.10 gives the odds ratio within one stratum, Equation 7.11 pools the strata into the Mantel-Haenszel estimate, Equation 7.12 tests whether the stratum-specific values are similar, and Equation 7.13 tests whether the pooled estimate differs from 1.

Worked Example 7.8 applies Equation 7.11 to the two RSV strata of Worked Example 7.7. Calculator 7.5 reproduces the calculation, reports the homogeneity test of Equation 7.12, and has a second preset in which the strata differ.

Interactive 7.1 simulates a study of 2,000 people in which a third variable, C, raises the risk of the outcome. Its sliders change the strength of confounding and of effect modification, so that the crude, stratum-specific and Mantel-Haenszel odds ratios can be compared as each changes.

Effect modification

Effect modification is present when the association between exposure and outcome differs across levels of a third variable. It is a property of the causal system to be described, while confounding is a bias to be removed. Whether it is present depends on the scale of measurement. The three cards below define interaction on the additive scale and on the multiplicative scale, and the two terms for a joint effect that is greater or smaller than the sum or the product of the separate effects.

On the multiplicative scale, effect modification appears in a stratified analysis as stratum-specific odds ratios that differ. Box 7.5 states what a protocol does in that case.

Non-collapsibility of the odds ratio

The odds ratio has a property that can mimic confounding. Even when a third variable is unrelated to the exposure, so that it cannot confound, the crude odds ratio can differ from the stratum-specific odds ratios whenever that variable affects the outcome and the outcome is common (Greenland, Robins & Pearl, 1999). Suppose 100 exposed and 100 unexposed people are studied in one stratum, with 80 and 50 cases, and 90 exposed and 90 unexposed in a second stratum, with 30 and 10 cases. The exposure is equally common in both strata, so the stratifying variable is unrelated to it. The odds ratio is 4.0 in each stratum and the Mantel-Haenszel estimate is 4.0, but the crude odds ratio from the combined table (110 of 190 exposed and 60 of 190 unexposed are cases) is 2.98. The change-in-estimate rule would flag a confounder where there is none. Two checks separate the explanations. The first asks whether the third variable is associated with the exposure, which it must be to confound (here it is not). The second uses the risk ratio, which is collapsible: without confounding, its crude value is a weighted average of the stratum-specific values. Here the stratum risk ratios are 1.60 and 3.00 and the crude risk ratio is 1.83, which lies between them. When the outcome is rare the odds ratio approximates the risk ratio and non-collapsibility is negligible.

Section 3 has set out the decisions a protocol makes about confounding. The protocol identifies confounders with a causal diagram, controls each one by restriction, matching, stratification or adjustment, and separates confounding from effect modification and from the non-collapsibility of the odds ratio. The key takeaways below summarize these decisions. The reflection that follows asks for the adjustment set in a cohort study of first-year students, and the knowledge check closes the section.

From measured to unmeasured confounders

Sections 1 to 3 dealt with biases that a protocol can prevent or correct with information it collects. Some confounders cannot be measured at all, and three computations show how much such a confounder could change a result: external adjustment, a planned sensitivity analysis and the E-value. A short orientation to the model-based methods that later courses teach comes first. The threats to validity and mitigation subsection of a protocol, with a complete example for the Campus Connection Study, then brings the four sections together.

Methods that scale to many confounders: an orientation

Stratification works well for one or two confounders. With five confounders of three levels each there are 243 strata, many of them nearly empty, and the Mantel-Haenszel estimator loses most of its information. Model-based methods summarize the confounders more efficiently. Later courses teach them. A protocol for a study with one or two confounders often plans a stratified analysis and names the model-based method that a fuller analysis would use. Table 7.7 summarizes what each method does and where it is taught, and it completes, at the level of orientation, the confounding-control methods that HSCI 341 Lesson 1 Section 4 previewed: propensity scores, regression and difference-in-differences.

Table 7.7. Model-based methods for many confounders: what each does, its key assumption and where it is taught.

In the STREP example, logistic regression of CRD on STREP and RSV gives an odds ratio of 3.35, and stratification on the propensity score gives 3.36, identical to the Mantel-Haenszel estimate because with a single binary confounder the propensity score takes only two values. The methods differ in how they handle many covariates. Except for instrumental variables, and difference-in-differences for confounders that are stable over time, they share one limitation: they adjust only for what was measured.

A protocol therefore also needs a way to reason about the confounders it could not measure, which the next part takes up.

When a confounder was never measured

Every protocol leaves some confounders unmeasured, because they were unknown, too costly or too sensitive to ask about, or because the data source does not record them. The protocol cannot remove this residual confounding, but it can name the likely candidates, state the direction of the bias each would cause, and plan a calculation that shows how strong a confounder would have to be to change the conclusion.

External adjustment

When a confounder U was not measured in the study but its prevalence and its association with the disease are known from other sources, the observed estimate can be divided by a bias factor (Schlesselman, 1978):

The formula is exact for risk ratios when the confounder is binary and its risk ratio with the disease is the same among the exposed and the unexposed; it is approximate for odds ratios, and the approximation is best when the disease is rare. In a case-control study the two prevalences describe the source population, which the controls represent. Worked Example 7.9 applies Equation 7.14 to the STREP data as if RSV had not been recorded, and Calculator 7.6 reproduces the example, with a second preset for a strong confounder that is far more common among the exposed.

External adjustment gives one corrected estimate for one set of outside values, and the agreement in Worked Example 7.9 depends on the accuracy of those values. A protocol therefore also plans for the possibility that they are wrong, which is the subject of the next part.

Planning a sensitivity analysis

When no single set of external values is trustworthy, the protocol plans a sensitivity analysis that varies the bias parameters over plausible ranges and reports how the estimate changes. The same logic applies to all three threats. For selection bias the parameters are the sampling fractions of Section 1, for misclassification they are the sensitivity and specificity of Section 2, and for an unmeasured confounder they are the prevalences and the strength used in external adjustment. Quantitative bias analysis is the general name for these methods (Lash, Fox & Fink, 2009). It ranges from simple analyses with one set of fixed values, through grids of fixed values, to probabilistic analyses that draw the parameters from probability distributions and repeat the correction many times. HSCI 230 Lesson 12 interprets sensitivity analyses that published studies report; a protocol plans one in five steps.

1. Name each threat to be analyzed and the direction in which it would move the estimate.
2. Define its bias parameters.
3. State where plausible values will come from: published studies, national surveys, pilot data or the study's own validation sub-study.
4. Choose the form of the analysis: a grid of fixed values (deterministic) or distributions (probabilistic).
5. State how the results will be reported, including the values at which the conclusion would change.

Worked Example 7.10 applies these steps to an unmeasured confounder of the STREP and CRD association, with a grid of fixed values for the prevalence and the strength that enter the bias factor of Equation 7.14.

The E-value

VanderWeele and Ding (2017) summarized this kind of analysis in a single number. The E-value is the minimum strength of association, on the risk-ratio scale, that an unmeasured confounder would need to have with both the exposure and the outcome, beyond the measured confounders, to explain away an observed association completely. Equation 7.15 gives the E-value for a risk ratio above 1.

For a risk ratio of 2.0, E = 2.0 + √(2.0 × 1.0) = 3.41, so a confounder would need a risk-ratio association of at least 3.4 with both the exposure and the outcome to explain the result away. An odds ratio or hazard ratio can be treated as a risk ratio when the outcome is rare; when the outcome is common (above about 15%), the square root of the odds ratio approximates the risk ratio (VanderWeele & Ding, 2017). In a protocol the E-value is a planning tool. The investigator computes it for the effect the study is designed to detect, compares it with the strength that the named unmeasured confounders could plausibly have, and states in advance that it will be reported for the estimate and for the confidence limit closer to 1. Worked Example 7.11 carries out this planning step for the Campus Connection Study, and Calculator 7.7 reproduces it, with presets for two other estimates.

External adjustment, a planned grid and the E-value give a protocol three ways to show how much an unmeasured confounder could change a result. The last part of this lesson places these results, with those of Sections 1 to 3, in the protocol subsection on threats to validity and mitigation.

Writing the threats to validity and mitigation subsection

The subsection brings the four sections of this lesson together for one study. It usually sits in the methods of a protocol, after the analysis plan and sample size. A reviewer reading it should be able to see that each threat has been identified for this particular study, that its likely direction has been reasoned out, and that the protocol contains specific steps to prevent it and to measure what remains. A strong subsection is specific to the study, quantitative where the tools of this lesson allow, consistent with the causal diagram, sampling plan and sample size elsewhere in the protocol, and candid about the threats that cannot be removed. Table 7.8 lists what the subsection states for each threat and the tools from this lesson that supply it.

Table 7.8. The elements of a threats to validity and mitigation subsection and the tools from this lesson that supply them.

The subsection is usually organized by threat (selection, information, confounding, and any design-specific or temporal threat), in about half a page to a page. It does not need to repeat the full sample size or analysis plan; it refers to them. Missing data cut across the threats: unit non-response and loss to follow-up are selection problems, while item missingness affects information and the measured confounders, so the subsection states the missingness it expects, the mechanism it assumes and the method it will use. Box 7.6 recalls the missing data mechanisms that HSCI 230 Lesson 11 described.

Worked Example 7.12 gives the complete subsection for the Campus Connection Study. It draws on the selection decisions and bound of Worked Example 7.3, the size of the validation sub-study in Worked Example 7.5 and the E-value of Worked Example 7.11, and it adds a plan for missing data.

The note that follows summarizes how a subsection like Worked Example 7.12 is drafted.

Section 4 has shown how a protocol reasons about confounders it cannot measure, and how the threats to validity and mitigation subsection gathers the tools of all four sections into one part of the protocol. The key takeaways below summarize these points. The reflection that follows plans a sensitivity analysis for an unmeasured confounder with the bias factor and the E-value, and the knowledge check closes the section.