Design better prediction models with context-aware data engineering
In this post, I would like to share with you my approach to optimize input data for deep learning models. I have successfully applied it in my work as a data scientist and data engineer. On the basis of practical examples you will learn how to enrich model input data with contextual information. This will enable you to design more robust and and accurate deep learning models.
Deep learning models are so powerful, because they are very good at integrating contextual information. We can boost performance of neural networks by adding several context to dimensions to the raw data. We can achieve this with some clever data engineering.
When you develop a new predictive deep learning algorithm you likely aim to choose a model architecture that perfectly fits your specific use case. Based on input data and the actual prediction task, there are plenty of recipes that you may have already in mind: Are you going to classify images? Then you’ll go most probably for convolutional neural networks. Is it about forecasting time series or analyzing text? LSTM networks could be a promising option, then. Normally the decision about the right model architecture is mainly driven by the type of data that flows into the model.
In that way, finding the right input data structure (i.e. defining the model’s input layer) is one of the most crucial steps in model design. I usually invest much more development time in engineering the shape of input data than in anything else. To be clear: We don’t we have to deal with the structure of the raw data as a given and just find the right model for it. The ability of neural networks to take care of feature engineering and feature selection internally (“end to end modeling”) does not free us from optimizing the structure of the input data. We should serve the data in a way that the model can make best sense out of it and take the most informed decision (i.e. the most precise prediction). The secret ingredient here is contextual information. We should enrich our raw data with as much context as possible.
What is Context?
But what do I mean specifically by context? Let’s take an example. Marie is a data scientist starting a new job to develop a sales forecasting system for a beverage retail company. Her task in a nutshell: Given a specific shop and a specific product (lemonade, orange juice, beer…), her model should predict the number of future sales of the product in the specific shop. Forecasting would be applied to thousands different products offered in hundreds of different shops. So far so good. On Marie’s first day, she goes to sales department, where the forecasting job is already performed, albeit manually by Peter, a very experienced sales accountant. Her goal is to understand on what basis this domain expert decides whether a particular product will be more or less in demand in the future. As a good data scientist, Marie anticipates that Peter’s years of experience will be very helpful in defining what data might be valuable to the model. To find out, Marie asks him two questions:
“What data do you look at to figure out how many bottles of a particular brand of lemonade we will sell next month in our store in Berlin? How do you interpret that data?”
Peter replies: “We look in the first step on lemonade sales in Berlin store over time”. He draws the following chart to illustrate his strategy:
We see a continuous curve with recurring peaks in July/August (summertime in Berlin). It makes intuitive sense that more product is sold in the summer, when temperatures are high and people are more in the mood for refreshments. On a smaller time scale (ca one month) we see noisy up and down of sales within the range of ca. 10 items, which is likely due to unpredictable events (random noise).
Peter: “When I see a repetitive pattern with increased sales in summer and decreased sales in winter. I assume that this will also happen in the future and estimate sales accordingly.” That sounds very plausible.
Peter interprets sales data in the context of time, where the distance of two data points is defined by their time difference. If the data would not be ordered in time context, it would be harder to interpret. For example, if we look just look at sales distribution in a histogram plot, time context is lost and our best future sales estimate would be some aggregate such as the median of all values.
Context arises when data is ordered in some way.
It goes without saying that you should feed your sales prediction model with historic sales data in the right time order to preserve the context that comes “for free” from your database. Deep learning models are so powerful, because they are very good at integrating contextual information, similar to our brain (or Peters brain, in this case).
Did you ever wonder why deep learning works so well for image classification and image object detection? Because ordinary images come already with lots of “natural” context: Images are basically data points of light intensity ordered in two context dimensions: namely the spatial distance in x and the spatial distance in y. Movies (image time series) complement time as a third context dimension.
Because context is so beneficial for prediction, we can boost our model performance by adding more context dimensions despite from the ones that are already included in the raw data. We achieve this with some clever data engineering as described in the next part.
We should serve the data in a way that the model can make best sense out of it and take the most informed decision. I usually invest much more development time in engineering the shape of input data than in anything else.
Designing Context-Enriched Data
Back to the meeting with Marie and Peter. Knowing that real data in most cases doesn’t look as nice as in the plot above, Marie slightly modifies the chart:
Marie: “What if the last sales data point is above the usual noise level? That could be a real effect. Maybe there’s a successful marketing campaign going on for the product. Maybe the recipe has been changed and it tastes much better now. In these cases, the effect is lasting, and sales will remain at the same high level in the future. Or maybe it is just an outlier due to a random event. For example, a school class visiting Berlin happened to enter the store and all the children bought a bottle of this lemonade brand. In this case, the increase is not stable, but just noise. How would you decide whether it is a real effect or not?”
You can see Peters scratching his head before he answers: “In this case, I’m looking at the sales of stores that are similar to the one in Berlin. For example, our stores in Hamburg and Munich. These stores are comparable because they are also located in major German cities. I wouldn’t consider stores in the countryside because I expect different customers with different tastes and preferences there.”
He adds sales curves from the other stores with two potential scenarios. “If I see the sales increase just in Berlin, I consider it as noise. However, if I see lemonade sales going up also in Hamburg and Munich I expect it to be a stable effect.”
So, in difficult situations, Peter takes more data into account to make better informed decisions. He adds a new data dimension in the context of different shops. As we learned above, context arises when the data is ordered in some way. To create a shop context we first have to define a distance measure to order data from different shops accordingly. Peter for example discriminates the shops according to the size of the cities where they are located.
With a some SQL and Numpy hacking, we can provide a similar context to our model. We first have to get population sizes of all cities where shops from our company are located. Then we measure the distance between all shops in terms of difference in population. Finally, we assemble all sales data in a 2D Matrix, where the first dimension is time and the second dimension is our shop distance measure.
The sales matrix provides a good overview of recent lemonade sales and the resulting patterns can be interpreted intuitively. Look at the lower left data point of the sales matrix: This is the most recent sales number for Berlin. The bright spot is likely an outlier, as the steep sales increase is not reproduced in similar stores (Hamburg for example). In contrast, the sales spike in July is reproduced by the similar stores.
We always need a distance measure to create context.
Translating Peters statement into mathematical terms, lemoande sales can be modeled as a function of population size of the cities where the product is offered. We always have to think very carefully about the right distance measure when adding new context dimensions. It depends, on what factors the entity we want to predict is affected. The influencing factors are totally dependent on the product and distance measures must be tailored accordingly. If you look at beer sales in Germany for example, you will see that consumers are likely to buy products from local breweries (You find ca. 1300 different was originally published in Towards Data Science on Medium, where people are continuing the conversation by highlighting and responding to this story.
SOCIAL SHARE CARD GENERATOR