Kiran Kumar
Controlling other factors to observe primary effect
In the last post, we walked through experimental design for an online checkout flow: a team wants to know whether a redesigned checkout reduces friction and increases the rate of completed purchases compared to the existing flow. We framed the hypothesis, picked a primary metric (completed purchases per visitor who reached checkout), and set up a random assignment between the old flow (control) and the new flow (treatment). That gets us from 0 to 1, an experiment exists. What it does not automatically give us is a clean read on the primary effect, because plenty of other factors are moving around at the same time as the experiment and some of them can masquerade as the effect we are trying to measure. This post covers how to identify those factors and control for them, so that the difference we observe between treatment and control can be attributed to the checkout redesign and not to something else riding along with it.
- Hypothesis
- Measurement variables
- Latent or unobservable factors
- Experimental design (0 to 1)
- Controlling other factors to observe primary effect. (discussed here)
- Collection and analysis of data for pattern discovery
- Modeling of patterns for prediction
- Generalizing or scaling the experiment (1 to n)
- Establishing a baseline
- Monitoring through controls and baselines
- Ethics and governance
What is a confounding variable
A confounding variable (or confound, a term we introduced briefly in the measurement variables post) is a factor that is associated with both the treatment assignment and the outcome, in a way that is not part of the causal path we care about. If a confound is unevenly distributed between the treatment and control groups, the comparison of outcomes between groups no longer isolates the effect of the treatment. Some of the difference we observe is real signal from the treatment, and some of it is contamination from the confound, and a simple comparison of means cannot tell the two apart.
Formally, if $T$ is the treatment indicator (new checkout flow versus old), $Y$ is the outcome (completed purchase), and $Z$ is some other factor, $Z$ is a confound with respect to the effect of $T$ on $Y$ when $Z$ influences $Y$ and $Z$ is not independent of $T$ in the data we observe.
$$ \text{naive estimate} = E[Y \mid T = 1] - E[Y \mid T = 0] $$
$$ \text{true effect} = E[Y(1) - Y(0)] $$
The naive estimate equals the true effect only when $T$ is independent of every other factor that affects $Y$, which is exactly what proper randomization is supposed to guarantee. When that independence breaks, the naive estimate can be biased upward, biased downward, or in the worst case have the wrong sign entirely, a phenomenon closely related to what statisticians call Simpson's paradox when a confound is a categorical variable with several levels.
A confound is not simply "any other variable that affects the outcome". Plenty of variables affect completed purchases (price sensitivity, product category, time of day) without being confounds. A variable only becomes a confound for a specific comparison when it is also unevenly distributed across the groups being compared. The same variable can be harmless in a well randomized experiment and dangerous in an observational study or a poorly randomized one.
Common confounds in online experiments
Online experiments share a family of usual suspects. It helps to build a checklist of these before an experiment launches, because most of them can be addressed by design if caught early, and only patched after the fact if caught late.
Device type
Mobile and desktop visitors tend to have different baseline purchase completion rates, different session lengths, and different tolerance for multi step forms. If a new checkout flow happens to render better on desktop and the treatment group randomly ends up with a few more desktop visitors than the control group by chance, some of the lift we measure is really a device effect, not a flow effect.
Traffic source
Visitors arriving from paid ads often look different from visitors arriving through organic search or a direct visit. Paid traffic can include more first time, lower intent visitors picked up by broad targeting, while organic visitors have often already decided to buy and are searching for the brand by name. If a marketing campaign happens to ramp up or wind down mid experiment, and that shift is not distributed evenly between treatment and control, traffic source becomes a confound.
Day of week and seasonality
Purchase behavior on weekends differs from weekdays, and both differ around paydays, holidays, and promotional events. An experiment that runs treatment more heavily in one calendar window than control, even briefly during a ramp up period, risks folding day of week effects and seasonal effects into the treatment estimate.
New versus returning users
Returning users already trust the brand, may have saved payment methods, and complete checkout at a higher rate regardless of the flow they see. New users are unfamiliar with the site and more sensitive to friction, which is often exactly what a checkout redesign is trying to fix. If the mix of new and returning users differs between groups, this alone can produce a lift or a loss that has nothing to do with the redesign.
None of these four are exotic. They show up in almost every consumer facing online experiment, which is why a data scientist should treat this checklist as a default starting point and then add domain specific confounds on top of it (in a checkout experiment specifically, saved card status, cart value, and coupon usage are worth considering too).
Techniques to control for confounds
There is a useful order of preference among the techniques below. Broadly, anything done before the experiment runs (by design) is more reliable than anything done after the data is collected (by analysis), and we will come back to why in the next section.
Randomization at the right unit
The first and strongest defense against confounding is randomization itself, and getting the unit of randomization right matters as much as randomizing at all. For the checkout experiment, the natural unit is the visitor (or better, a persistent user identifier), not the individual session or page view. If we randomized at the session level instead, the same visitor could see the old flow on one visit and the new flow on another, and any effect from getting used to a flow, or from a partial cart carried across sessions, would leak between the two conditions. Randomizing consistently at the visitor level, using a stable hash of a user or cookie identifier, ensures that whatever is true about a given visitor (their device, their loyalty, their price sensitivity) is fixed within a condition rather than smeared across both.
Randomization at a large enough sample also has a second useful property beyond eliminating confounds in expectation: on average, across many repeated experiments, confounds are balanced between groups, but in any single run there is still a chance of imbalance. That is exactly the gap that the remaining techniques exist to close.
Stratified sampling or blocking
Stratification (also called blocking) means splitting the population into groups based on a known confound before randomizing, and then randomizing treatment and control independently within each group. For the checkout experiment, we could stratify by device type, first forming a mobile stratum and a desktop stratum, then randomly assigning visitors to treatment or control separately within each stratum in a fixed ratio. This guarantees, by construction rather than by chance, that the treatment and control groups have the same proportion of mobile and desktop visitors. Traffic source is a good second stratification variable: block by device type and traffic source jointly (mobile paid, mobile organic, desktop paid, desktop organic) if there is enough volume in each cell to keep the statistical power of the experiment reasonable.
Stratification is strictly better than plain randomization when the stratifying variable is known in advance and predictive of the outcome, because it removes a source of chance imbalance entirely rather than merely making it unlikely.
Holding a variable constant by restricting the population
Sometimes the simplest control is to not let a confound vary at all. Instead of stratifying by traffic source, we could restrict the experiment to organic and direct visitors only, and exclude paid campaign traffic for the duration of the test. This trades generality for cleanliness: the estimate we get applies to organic and direct visitors, and we would need a separate test (or a stratified design) to know whether the effect holds for paid traffic too. Restriction is a reasonable choice when a confound is hard to measure reliably (traffic source attribution is often noisy) or when its distribution is expected to shift unpredictably during the test window, such as a marketing campaign that might launch mid experiment without the experimentation team knowing in advance.
Adjusting for a covariate after the fact
If a confound was not controlled for at design time, it is sometimes possible to statistically adjust for it during analysis, provided it was measured and recorded for every visitor. A common approach is to fit a regression of the outcome on both the treatment indicator and the covariate:
$$ Y = \beta_0 + \beta_1 T + \beta_2 Z + \epsilon $$
where $Z$ is the confound (say, a new versus returning indicator). $\beta_1$ then estimates the treatment effect holding $Z$ fixed, which corrects for imbalance in $Z$ between the groups as long as the relationship between $Z$ and $Y$ is modeled correctly and there is enough overlap in $Z$ across both groups. This same idea extends to matching methods and propensity score adjustment when there are several confounds to account for at once, techniques that trace back to Rosenbaum and Rubin's work on propensity scores in observational studies [[1]].
Controlling by design versus controlling in analysis
It is worth being explicit about the distinction, because the two are often confused as interchangeable when they are not.
Controlling by design happens before or during data collection. Randomizing at the right unit, stratifying the sample, and restricting the population are all design time controls. They change how the data is generated, so that the confound cannot create an imbalance between groups in the first place.
Controlling in analysis happens after the data has already been collected. Statistical adjustment, matching, and propensity weighting are all analysis time controls. They do not change how the data was generated, they attempt to remove the effect of an imbalance that already exists in the collected data.
Design time control is generally preferable for a few concrete reasons.
First, analysis time adjustment only works for confounds we thought to measure. A stratified design protects against imbalance in a variable whether or not we later remember to include it in a regression, but a regression adjustment can only correct for a variable that is sitting in the dataset. Any confound we failed to record is simply invisible to analysis time methods and remains a threat.
Second, analysis time adjustment depends on getting the model right. If the true relationship between $Z$ and $Y$ is nonlinear and we fit a linear adjustment, or if there are interactions between $Z$ and the treatment that the model does not capture, the adjustment can be incomplete or even introduce new bias. A stratified design makes no such modeling assumption, the balance is exact by construction.
Third, analysis time adjustment can run into a lack of overlap. If nearly all mobile visitors ended up in the treatment group and nearly all desktop visitors ended up in control, a regression is being asked to extrapolate into regions of the data it has barely seen, and the resulting estimate is unreliable no matter how sound the model looks on paper. A stratified design prevents this scenario from happening at all.
The practical implication is not that analysis time controls are useless, they are a valuable safety net and are sometimes the only option, particularly for confounds discovered late or for observational data where randomization was never possible to begin with. But the right posture is to spend effort up front identifying likely confounds and designing them out of the experiment, and to treat analysis time adjustment as a second line of defense rather than the primary plan.
Worked example: checking the checkout experiment
Returning to the checkout experiment from the last post: the primary metric is the completed purchase rate among visitors who reach the checkout page, comparing the new flow (treatment) against the existing flow (control). Let's work through device type and traffic source as concrete confounds and how each would be controlled.
Device type as a confound
Suppose historical data shows desktop visitors complete checkout at roughly 38 percent while mobile visitors complete at roughly 24 percent, a large and well known gap driven by smaller screens, slower typing, and less trust in entering payment details on a phone. If we simply randomize visitors one to one into treatment and control without any further care, in expectation the two groups will have similar device mixes and this is not a problem. The risk shows up in two more specific situations: a genuinely unlucky random split that happens to tilt one group toward mobile, which is more likely to matter when the sample size during a ramp up period is still small, or a rollout mechanism that assigns treatment based on something correlated with device, for example if the new flow was enabled first for a subset of users on a newer app version that mobile users are more likely to have installed.
Control by design: stratify the random assignment by device type. Split the incoming visitor stream into a mobile bucket and a desktop bucket first, then randomize treatment and control independently within each bucket at the same ratio, say 50/50 in both. This guarantees the treatment and control groups have matching device mixes regardless of how the randomization happens to fall, and it also lets us report the treatment effect separately for mobile and desktop, which is useful information on its own since a checkout redesign is quite likely to help one device type more than the other.
Control in analysis (if this was missed): if the experiment already ran without stratification and we discover after the fact that the groups ended up with a mild device imbalance, say 54 percent mobile in treatment against 48 percent mobile in control, we can add a device indicator as a covariate in the regression estimating the treatment effect, or report the effect separately within each device stratum after the fact (a post hoc stratification, sometimes called poststratification). This is a reasonable patch, but note it only works because device type was actually being logged for every visitor. If it had not been recorded, this fix would not be available at all.
Traffic source as a confound
Suppose the marketing team runs a paid ad campaign that ramps up in the middle of the two week experiment window, driving a surge of paid search visitors. If the experiment's traffic splitting mechanism assigns new visitors to treatment and control based on when they first arrive (a common implementation detail, for example round robin assignment by timestamp rather than a stable per visitor hash), then a traffic surge concentrated in time could end up disproportionately represented in whichever condition happens to be "due" during that surge. Paid search visitors, being lower intent on average, complete checkout less often than organic visitors regardless of which flow they see, so this would bias the comparison.
Control by design: two changes address this together. First, assign treatment and control using a stable hash of a persistent visitor identifier rather than time of arrival, so that the mechanism generating group assignment has no relationship to when a traffic surge occurs. Second, stratify by traffic source the same way as device type, splitting into paid and organic buckets and randomizing independently within each, so that even a genuine surge in paid traffic affects both the treatment and control groups by the same proportion.
Control in analysis (if this was missed): include traffic source as a covariate alongside device type in the regression, or restrict the analysis to the subset of the experiment window before the campaign ramped up if that window alone has enough data to be conclusive. Both are weaker than having stratified from the start, the covariate adjustment depends on the model being correctly specified, and restricting the window throws away data and reduces statistical power.
Put together, a defensible version of this experiment stratifies on device type and traffic source jointly at the point of assignment (mobile paid, mobile organic, desktop paid, desktop organic), randomizes visitors to treatment or control independently within each of the four cells using a stable per visitor hash, and still logs both variables for every visitor so that a regression adjustment or subgroup breakdown is available as a check even though it should no longer be doing the heavy lifting.
A note on new versus returning users
New versus returning status deserves one additional comment because it behaves a little differently from device type and traffic source. A returning user's status is fixed before the experiment starts, so stratifying on it works exactly the same way as the other two variables. But a returning user in the treatment group who becomes accustomed to the new flow, and returns again later in the experiment window, is still the same user seen twice. This is exactly why randomization at the visitor level, discussed above, matters so much here specifically: without a stable identifier tying repeat visits to the same person and the same condition, a returning user could show up as new data in the opposite condition on a later visit, which corrupts the new versus returning stratification itself. Getting the unit of randomization right is a prerequisite for stratification to work as intended, not an independent, separable step.
With the primary effect isolated from these confounds, the next question is what to do with the data the experiment produces, which is where collection and analysis for pattern discovery picks up in the next post.
References
[1] Rosenbaum, P.R. and Rubin, D.B. 1983. The central role of the propensity score in observational studies for causal effects. Biometrika. 70, 1 (1983), 41 to 55.