Kiran Kumar
Collection and analysis of data for pattern discovery
In the last two posts, the process moved from picking measurement variables to an experimental design meant to isolate the effect of one variable at a time, holding everything else steady enough to trust the comparison. That kind of design works when a data scientist already knows roughly what question is being asked. This post covers a different situation: data that nobody designed an experiment around at all. It arrives because a system logs it, an app instruments it, or a sensor records it continuously, and the task is to look at it with no single predetermined question in mind and find patterns worth studying further.
This is the point in the series where the process branches away from the tightly controlled comparison of the last two posts and toward open ended exploration. The two modes are not competitors. Exploration is usually what surfaces the candidate hypothesis, and controlled experimentation, or at least a disciplined observational study, is usually what confirms whether the candidate holds up. Getting the order backwards, treating a pattern found by browsing a dashboard as if it were already proven, is one of the more common failure modes in applied data science, and a good chunk of this post is about why that happens and how to guard against it.
- Hypothesis
- Measurement variables
- Latent or unobservable factors
- Experimental design (0 to 1)
- Collection and analysis of data for pattern discovery (discussed here)
- Modeling of patterns for prediction
- Generalizing or scaling the experiment (1 to n)
- Establishing a baseline
- Monitoring through controls and baselines
- Ethics and governance
Designed data versus found data
It helps to name the distinction plainly before going further, because the two kinds of data behave differently and deserve different levels of trust.
Designed data is collected to answer a specific question that was decided in advance. The experimenter chose what to measure, when to measure it, and often who or what to measure it on, precisely because those choices bear on one hypothesis.
Found data (sometimes called observational or incidental data) is collected because a system happens to produce it as a side effect of running. Nobody decided in advance which question it would answer. Server logs, click streams, sensor telemetry, and transaction records all fall into this category.
Found data is attractive because it is often abundant and cheap relative to running a new experiment. It is also attractive because it reflects what actually happened in production, at scale, with real users or real conditions, rather than a constructed scenario. The tradeoff is that nobody controlled for anything. The data was not generated to make one comparison clean, it was generated because a system was doing its job, and whatever biases exist in how the system does its job are baked directly into the data.
A data scientist coming from the tightly designed world of the last two posts has to change posture when working with found data. The question is no longer only "does this variable move the outcome" but first "can I trust that this dataset represents what I think it represents." That second question is where most of the real work in this stage of the process lives.
Common pitfalls in passive collection
Three pitfalls come up often enough in practice that they are worth naming individually, since each one distorts a discovered pattern in a different way.
Survivorship bias
Survivorship bias occurs when the data available for analysis only contains the cases that made it through some earlier filter, and the filter itself is correlated with the outcome being studied. The canonical illustration is the returning World War II bombers examined for bullet damage. Engineers wanted to know where to add armor, and the naive answer was to armor the spots with the most bullet holes on returning planes. Abraham Wald pointed out this reasoning was inverted: the planes in the sample had survived despite that damage, so the armor belonged on the spots with no holes, the spots where a hit meant the plane never came back at all [[1]]. Any dataset built only from survivors, whether they are planes, customers who did not churn, or transactions that did not get flagged, tells you what survivors look like, not what determines survival.
Selection bias in what gets logged
A related but distinct problem is that logging systems rarely capture everything, and the decision about what to instrument is itself a choice made under constraints, usually engineering priorities or storage cost, that have nothing to do with the analysis someone will eventually want to run. If an event is logged only when a particular code path executes, and that code path is more likely to execute under certain conditions, the resulting dataset systematically overrepresents those conditions. This is easy to miss because the absence of a record does not announce itself the way an obviously wrong value would. A data scientist has to actively ask what could have happened that would never show up as a row at all.
Missing data that is not missing at random
Statisticians distinguish a few flavors of missingness [[2]]. Data missing completely at random has a missingness pattern unrelated to anything, observed or not. That case is convenient and, in found data, rare. Data missing at random has a missingness pattern that depends only on other observed variables, so it can in principle be corrected for once those variables are accounted for. The dangerous case is data not missing at random, where the very fact that a value is missing is related to the unobserved value itself. A sensor that drops readings specifically under the conditions being studied, or a customer who does not fill out a satisfaction survey specifically because they were unhappy, both produce missingness that is entangled with the thing the analysis is trying to measure. Treating that kind of gap as if it were random and simply dropping the incomplete rows quietly changes what question the remaining data answers.
The basics of exploratory data analysis
Exploratory data analysis, a term and a practice largely credited to John Tukey [[3]], is the discipline of looking at data before trusting any conclusion drawn from it. The point is not to produce a polished chart for a stakeholder, it is to build an honest sense of what the data actually contains, warts included, before a pattern gets promoted to something worth acting on.
Distributions first
Before comparing groups or fitting anything, look at the distribution of each variable in isolation. A histogram or a simple set of quantiles answers questions a summary statistic alone cannot: is the variable unimodal or does it show multiple clusters, is it heavily skewed, does it have a hard floor or ceiling that suggests truncation or clipping somewhere upstream. A mean and a standard deviation computed on a bimodal distribution describes a value that may not correspond to any real observation.
Looking for outliers
An outlier deserves investigation before it is either trusted or discarded. Some outliers are genuine, rare but real events that carry information. Others are artifacts: a unit conversion error, a sentinel value like $-1$ or $9999$ used by an upstream system to mean "unknown" and never filtered out, or a duplicate record inflating a count. The mistake to avoid in either direction is a reflexive one, neither deleting every extreme value on sight nor including it uncritically because it is present in the data. The right move is to ask what process generated that value.
Checking data quality before trusting a pattern
A short checklist that tends to catch a large share of problems before they propagate into a discovered pattern:
Do timestamps fall in a sane range, and is the timezone consistent across every source that was joined together?
Do categorical fields have a stable, exhaustive set of values, or does a new category silently appear over time, suggesting an upstream schema change?
Is the volume of records per time period stable, or are there unexplained drops and spikes that might indicate an outage in the logging pipeline rather than an outage in the real world?
Does the geographic or demographic coverage of the data match the coverage of the population it claims to represent?
None of this produces an interesting finding by itself, and it is tempting to rush past it toward the more rewarding work of finding a pattern. Skipping it is exactly how a data quality artifact gets mistaken for a discovery.
Pattern discovery generates hypotheses, it does not confirm them
The single most important framing for this stage of the process is this: a pattern found by exploring found data is a candidate, not a conclusion. Exploratory analysis, almost by construction, involves looking at many variables, many slices, and many possible relationships. Some fraction of what turns up will look striking purely by chance, and the more slices examined, the larger that fraction becomes. This is the same multiple comparisons problem that shows up in formal hypothesis testing, just informal and often invisible because nobody wrote down in advance how many comparisons were being made.
$$ P(\text{at least one spurious pattern found}) = 1 - (1 - \alpha)^{k} $$
where $\alpha$ is the chance any single comparison looks interesting purely by chance and $k$ is the number of slices examined. Even a small $\alpha$ compounds quickly once $k$ grows into the dozens, which is routine when exploring by time of day, day of week, geography, device type, and every combination of those at once.
The discipline this implies is not to avoid exploration, exploration is exactly how interesting candidates get found in the first place, but to be honest about what stage of the process a pattern is in. A pattern discovered by looking at a dashboard needs to be restated as a specific hypothesis, ideally checked against a fresh slice of data the exploration did not touch, before it earns the confidence to inform a decision. That restatement, turning an interesting shape in a chart into hypothesis driven exploration with a falsifiable claim attached to it, is the subject of the very next post in this series.
Worked example: ride hailing demand patterns
Consider a ride sharing company with a fleet of drivers matched to riders through an app. Every trip generates a log record: request time, pickup location, dropoff location, driver assignment latency, trip duration, and a stream of GPS pings while the trip is in progress. None of this was collected to answer one question. It is a byproduct of running the service. The data science team wants to understand demand: when and where do ride requests spike, so that driver supply can be positioned ahead of need.
Framing the exploration
The team starts, appropriately, with distributions. A histogram of requests by hour of day shows the expected commute peaks, a morning bump and a larger evening bump, with a smaller late night bump on weekends. Requests by day of week show weekday and weekend shapes differ enough that averaging across all seven days would blur two distinct patterns together. So far this is unsurprising and confirms the data behaves the way intuition would predict, which is itself a useful sanity check before trusting anything less obvious.
The team then slices by neighborhood, and this is where a pattern starts to look genuinely interesting: request volume in a handful of dense urban neighborhoods appears to be lower than population and foot traffic data would predict, particularly in the evening peak. The tempting conclusion is a demand insight, perhaps that riders in those neighborhoods prefer walking or transit at that hour, which would argue against repositioning drivers there.
A data quality issue that would have biased the pattern
Before accepting that conclusion, the checklist from the previous section earns its keep. Someone on the team checks whether trip volume correlates with anything about how the data itself was captured, not just the underlying event. It turns out GPS pings from the driver app drop out more frequently in exactly those dense urban neighborhoods, because tall buildings on both sides of a street degrade satellite signal, a well known effect sometimes called the urban canyon problem. When a trip loses enough consecutive GPS pings, an upstream data pipeline step, built for an unrelated purpose, silently discards the trip record rather than logging it with incomplete location data.
The practical consequence is that trips are not merely mislocated in those neighborhoods, a fraction of them never appear in the dataset at all. This is a textbook case of data not missing at random: the probability a trip record is missing is directly tied to the neighborhood, which is also the variable the exploration is trying to draw a conclusion about. The apparent drop in evening demand in those neighborhoods was not a rider behavior pattern, it was a logging artifact wearing the costume of a demand pattern. Had the team promoted this straight from the exploratory dashboard into a driver repositioning recommendation, the fix would have pulled drivers away from neighborhoods that in reality needed more supply, not less.
Catching this required treating the pattern the way this post recommends: as a candidate, not a conclusion. Cross referencing trip volume against an independent signal, in this case cell tower or wifi based location pings the app also collects as a fallback, showed request volume in those neighborhoods was in line with expectations once the dropped records were accounted for. The corrected data still contained a real pattern worth exploring further, evening demand did vary meaningfully by neighborhood, just not in the direction or magnitude the raw, GPS gap riddled dataset had suggested.
What carries forward
The corrected neighborhood level demand differences are exactly the kind of output this stage of the process is meant to produce: a set of candidate patterns, now checked for the more obvious ways the data collection process itself could have generated them, ready to be restated as specific, falsifiable hypotheses. Whether evening demand in a given neighborhood is driven by commute patterns, event schedules, or weather is a question for hypothesis driven exploration, examined next in this series, rather than something this exploratory pass was ever positioned to answer on its own.
References
[1] Wald, A. 1943. A method of estimating plane vulnerability based on damage of survivors. Statistical Research Group, Columbia University. Reprinted in Mangel, M. and Samaniego, F.J. 1984. Abraham Wald's work on aircraft survivability. Journal of the American Statistical Association. 79, 386 (1984), 259 to 267.
[2] Rubin, D.B. 1976. Inference and missing data. Biometrika. 63, 3 (1976), 581 to 592.
[3] Tukey, J.W. 1977. Exploratory Data Analysis. Addison-Wesley.