1  The Data Pipeline

In this mini and the following one, we’ll discuss several pieces of how to work with large datasets in practice:

  1. Storing data, using tools like relational databases
  2. Extracting data for a specific task, using SQL or Spark
  3. Using data to fit models, with neural networks or Spark ML

But first, we must look at the entire process end-to-end: the data pipeline, following the data from when it arrives at the company to its storage, use, and reuse. As statisticians, we often gloss over most steps of the pipeline, and start our analysis with a nicely curated CSV containing all the relevant information. But in practice, someone must do the curation—and more importantly, someone must design the data pipeline, and think about the life of the data from beginning to end. What data should be collected, where should it be stored, and how can it be put to use?

As we discuss data pipeline design, let’s consider a motivating example.

Example 1.1 (HamsterCard) Credit card companies work with large volumes of data: data about their customers, data about transactions made with their cards, data for marketing, data for fraud detection, and so on. There are many credit cards for specific audiences, like cards with rewards for travel, cards with special discounts for gas or groceries, and even luxury cards that come in titanium and help you flaunt your wealth.

So let’s consider the HamsterCard, a credit card for people who like hamsters. Benefits include hamster-themed card designs, 5% cash-back rewards at pet supply stores, and a quarterly magazine for hamster aficionados.

HamsterCard, the card for hamster lovers. Generated by Bing Image Creator.

To successfully operate the HamsterCard, the company must do many things:

  1. Find potential customers to market HamsterCards to, through postal mail, email, and online advertisements
  2. Detect and block fraudulent transactions
  3. Offer special promotions and coupons to current customers based on their interests
  4. Set and update credit limits for customers based on their creditworthiness and the company’s willingness to take financial risks
  5. Track the productivity of customer support staff who answer phone calls and online messages about HamsterCard, and identify high and low performers for management

All of these will involve data.

1.1 ETL

The first part of the process is to get the data. This is not as easy as it sounds. The data you need comes from many sources in many different formats, and you must work to get what you need.

The usual acronym for this process is ETL: extract, transform, and load. You must extract the relevant data from various sources, transform it to be consistent and match whatever schema you use to store it, and load it into your choice of database or storage system. This might happen repeatedly as you obtain updated data, or when you decide you need to change what data is extracted and stored.

Exercise 1.1 (Lead generation for HamsterCard) Credit card companies need to market their cards to prospective customers, since each customer is a potential source of revenue. That means finding new people who don’t have the card but could be interested in getting one. Prospective customers are called leads, and companies try to acquire leads so they can send them advertisements and convince them to sign up.

One of HamsterCard’s partner companies generating leads. Liberty Meadows by Frank Cho.

HamsterCard acquires leads in several ways:

  1. They buy customer lists from partner banks, with customer names, addresses, and basic financial information. These are provided as tab-delimited text files.
  2. They buy email lists from marketing companies (that themselves bought the email addresses from other online services). The lists include email addresses, variables giving interests and demographic information learned or inferred by the marketing companies, and perhaps information about other products that email is known to be associated with. The lists are given as CSVs.
  3. They buy subscriber lists from Hamsters Monthly, Hamster Fancy, and other top newsletters and magazines. These lists give email addresses, subscription dates, and click-through rates measuring how many times the subscriber clicked on links in the email newsletters. The lists are in a shared SQL database that HamsterFacts has read-only access to.
  4. They buy subscription lists from Hamster Facts, a text message service that sends hourly hamster facts texts. The subscription list has phone numbers, but no other information about the subscribers. The list is provided as a text file with one phone number per line.

Suppose you’ve been assigned to build a database of leads by combining these sources, so the marketing team can use them to send emails, postcards, and texts.

For each step in the extract, transform, and load process, what decisions do you need to make? What problems might occur?

We can see that ETL can be very complicated in its own right. Data always seems to come in the wrong format, it’s frequently updated, and people want to use it for many different things. ETL involves designing a core storage system for the parts you want to keep, and writing all the necessary code to load data into that storage system. For this class, that storage system will often be a SQL database.

1.2 Feature engineering

When the ETL process is done, hopefully you have all the data you need in a convenient format. Now you need to solve some business questions using your data, and since you’re a statistician, you’ll probably have to fit some models.

The variables we use in our models don’t always match the variables stored in our databases. Some differences are trivial: a categorical variable might be turned into a one-hot encoded vector, for instance. But some differences are more substantial.

Feature engineering refers to the process of manipulating data to extract new features (predictors) that can be used in fitted models. Let’s consider an example.

Example 1.2 (Fraud detection for HamsterCard) Credit card fraud is a major problem, and every credit card issuer must check transactions for signs of fraud so fraudulent transactions can be rejected or reversed. HamsterCard thus has an automated system that must make classifications: when each transaction is received, it gets features about that transaction, and it must classify it as “fraudulent”, “questionable”, or “valid”. Fraudulent transactions will be blocked, questionable ones may get checked by sending a text or email to the cardholder, and valid transactions will go through.

For each transaction, HamsterCard receives the following information:

  • The date and time
  • The vendor (such as the store the card is being used at)
  • The geographic location of the purchase, if in person, or a code indicating if the purchase was made online or over the phone
  • The card number
  • The purchase amount

With only these features, a classifier would have a very hard time. So before giving the information to the classifier, HamsterCard’s system looks up additional information, such as

  • the number of times this customer has made purchases with the same vendor previously
  • the average amount this customer spends per purchase
  • the number of past fraudulent transactions for this customer

Getting this information may require pulling data from multiple databases inside HamsterCard, but it would make the fraud detector much more accurate.

Part of being a data scientist in a large company, then, is feature engineering: identifying data that may be relevant for a statistical problem, and pulling out interesting features from it. These features may not be stored in the original data but instead are calculated from it. The challenge is knowing what to calculate, and this is why a good data scientist must understand their data sources and the problems they are trying to solve.

In this course, a lot of feature engineering will involve writing SQL queries to fetch appropriate data from our database.

Exercise 1.2 (Feature engineering for fraud) Suggest three additional features that HamsterCard may want to extract from its data sources to assist in the fraud detection task in Example 1.2.

1.3 Model building

You’ll spend plenty of time on this in your other classes, so we won’t discuss it here.

1.4 Model deployment

Once you’ve built a model, it needs to be put to use. Sometimes this means you use the model to answer some research questions, write a report, and give a presentation about what you’ve learned, since the project goals were to answer some questions. This is what you’ve been learning to do in other courses, like when you write data analysis reports for 36-617.

But in other cases, the model needs to be used in a product. Your model for recommending movies to streaming service subscribers has to be used to make recommendations; your model for fraud detection needs to be used to detect fraud in new data. This usually means applying your model to new data beyond the training dataset, and that means trouble.

1.4.1 Training

A model put into production use may be used for many months or years. Over that time, additional training data may be collected. It may be useful to update the model with the new data as it’s collected, so the model is always trained with the largest and most up-to-date possible dataset.

That implies you must choose: How often should the model be updated?

Exercise 1.3 (Re-training a HamsterCard fraud detector) Consider HamsterCard’s fraud detector in Example 1.2. Each day, HamsterCard processes millions of new transactions, and in principle it could feed these transactions into its fraud detection model to retrain it.

What reasons might there be to retrain the model as quickly as possible, say every day?

Why might it be a problem to use the previous day’s data to train the fraud detector for the next day?

Now, it’s important not to automate this too much. If you retrain the model automatically every day or week, then immediately start using the new model, you may run into problems. What if the new data causes the model to perform worse? What if a batch of unusual data arrives and biases the model? You’re going to need a process to check the model every time it is updated, and only put it to use if it passes the checks. In fact, we’ll need to monitor the model’s performance even if we’re not updating it, as we’ll see in Section 1.5.

1.4.2 Prediction

There are different ways to put a predictive model into production.

One approach is batch prediction. Predictions are made in batches, say every week or every month, and in large quantity. That batch of predictions is then used to do various things until the next batch is produced. The predictions are produced by some automated system that runs periodically, and perhaps we can afford a slow system: if a batch takes a full day to produce, that’s fine if we’re only doing it once a month.

The opposite is online prediction: predictions are made in real time as new observations come in. The predictions are immediately used in the product, and so it is often important for the predictions to be made very quickly. We can’t afford a slow, complicated model that takes many minutes to look up necessary data and make its prediction; we might deliberately use a simpler but less accurate model because we do not have time for a more accurate but slower model.

The word “latency” is often used to describe the time between something being requested and it being delivered, and so we could define a model’s latency as the time between providing an observation and receiving a prediction for that observation. In batch predictions, we can tolerate high latency; in online predictions, we may need the latency to be as small as possible.

Exercise 1.4 In Example 1.1, we listed four tasks HamsterCard must do with data. Each of these would involve a model (or several).

For each task, identify if predictions would be made in batches or online, and how much latency can be tolerated.

1.5 Monitoring

We statisticians tend to think of models as static objects. We spend time carefully constructing a model, then we fit the final version, and then we use it. But models used for prediction on new data must face the real world, and the real world sucks.

1.5.1 Distribution shift

Ever since you first learned to fit regression models, you were taught not to extrapolate beyond the range of your data. But whenever you use a model trained on the past to predict observations received in the present, you are implicitly extrapolating: you are assuming that the same relationship between \(X\) and \(Y\) is present now as was present when you fit the model (Hume 1748). You are also assuming the population distribution of \(X\) is roughly the same as it was when you trained the model.

Let’s examine each of these in turn.

Definition 1.1 (Concept drift) Regression models typically try to approximate \(\E[Y \mid X = x]\), where \(Y \in \R\) and \(x \in \R^p\); classifiers try to approximate \(\Pr(Y = k \mid X = x)\) for a class \(k\), or try to pick the class with the highest probability. When we train models using a sample, we are hoping that the estimate we obtain from that sample is a good approximation in the population.

But what if the true relationship \(\E[Y \mid X = x]\) changes? Our model is now an approximation to the wrong relationship. Depending on the change, our predictions could become much less accurate. The same applies for a change in the true classification probability \(\Pr(Y = k \mid X = x)\).

This problem is called concept drift.

Another possibility is that the distribution of \(X\) may change even while \(\E[Y \mid X = x]\) stays the same.

Definition 1.2 (Covariate shift) Suppose we obtained our training sample from some distribution \(F_\text{train}\), but over time, new observations begin to come from some different distribution \(F_\text{new}\).

This problem is called covariate shift.

This is much more akin to the extrapolation you’re familiar with. If \(F_\text{new}\) produces values of \(X\) that are very different from those drawn from \(F_\text{train}\), then your model will have to extrapolate beyond the range of \(X\) it was trained on. This problem is more severe when \(X\) is multidimensional (i.e. you are predicting using many covariates), because a new distribution could produce combinations of features your model was never trained on, even if individually they are within the same range.

Both concept drift and covariate shift can cause the performance of a predictive model to decline over time.

Example 1.3 (Predicting house prices) Consider a model that predicts a house’s value using features like its size, age, number of bedrooms, and so on—the kind of model that sites like Zillow and Redfin use to estimate values of houses that are not for sale.

If house prices suddenly go up in an area due to increased demand, this can cause concept drift: you have the same houses with the same \(X\) values, but the value \(Y\) is suddenly much higher. If you do not update your model, it will continue predicting the old prices.

Similarly, if preferences change and people suddenly hate large, spacious homes because of how much time they have to spend sweeping the floors, this would be concept drift.

On the other hand, if the model were trained on an area with mostly single-family homes, but then a developer came in and built hundreds of units of townhomes and condos, that could cause covariate shift: the new housing units are very different than those the model was trained on, and the model will have to extrapolate to predict their values.

Exercise 1.5 Give examples of how concept drift and covariate shift could affect HamsterCard’s fraud detection system.

1.5.2 Feedback loops

We statisticians are used to think of our models as being separate from the world being modeled. The models live in our computer; the data comes from the world out there. Nothing we do in R or Python affects the real world.

But if you’re building a product that uses your models to make decisions that do something in the real world—send advertising, block a user, purchase a stock, or whatever—those decisions may affect the world and affect future data you collect.

A simple example: Many police departments keep extensive data on reported crimes and arrests, and use this data to predict locations where high crime or arrest rates are expected. They then send additional police patrols to those areas. If those patrols are more likely to conduct traffic stops, search suspicious pedestrians, and issue tickets for loitering, they are creating more reported events in the data—and hence contribute to the area being predicted to have more events in the future. Police may be sent to areas based on their own work in those areas, not because they have an intrinsically higher crime rate than other areas. This is a feedback loop: the model’s output is creating data that will affect its outputs.

Exercise 1.6 Give an example of how HamsterCard’s marketing models—which choose potential customers to target with advertising—could create a feedback loop.

Preventing feedback loops can be difficult. After all, the point of the model is often to make changes in the real world. There is no one strategy to prevent them, but in particular situations we may be able to.

For example, recommender systems can easily have feedback. If you recommend movies to viewers based on what you think they’ll like, but estimate whether they’ll like movies based on how many other people watch and enjoy them, you may tend to make popular movies more popular while rarely recommending movies that people haven’t viewed yet. This creates a “rich get richer” situation: nobody watches unknown movies, so you don’t know if anyone will like them, so you don’t recommend them to anyone. This can be avoided by deliberately recommending a mix of movies highly rated by your algorithm and less well-known movies with high uncertainty.

1.5.3 Tracking performance

Together, these problems suggest an important fact: We cannot simply train a model, deploy it, and never look back. We must periodically check that the model is still performing well and serving its intended purpose.

What you monitor depends on what is available to you. If you eventually observe outcomes for your predictions, you can measure their accuracy or success rate and track it over time. You can also track metrics about the typical inputs to your model, the distribution of covariates, or other features relevant to your prediction.

Exercise 1.7 Give some examples of metrics HamsterCard might track for its marketing models.