A Data Pipeline for 1 million movies and 10 million streaming links

Feb 2023: I wanted to see all scores for movies + tv shows and where to stream them on one page but couldn't find an aggregator that included all sources that were relevant for me.

Mar 2023: So, I built an MVP that grabbed scores on the fly and . It eclipses similar orchestration engines easily - at least for my needs.

Fast forward to today and after 12 months of continuous data munching, I want to share how the pipeline works in detail. You'll learn how to build a complex system that grabs data from many different sources, normalizes data and combines it into an optimized format for querying.

Pics or didn't happen!

The block in the center contains a script like this (simplified):

def main():

    return tmdb_extract_daily_dump_data()



def tmdb_extract_daily_dump_data():

    print("Checking TMDB for latest daily dumps")

    init_mongodb()



    daily_dump_infos = get_daily_dump_infos()

    for daily_dump_info in daily_dump_infos:

        download_zip_and_store_in_db(daily_dump_info)



    close_mongodb()

    return [info.to_mongo() for info in daily_dump_infos]



[...]

The following beast is also a flow (remember, this is only one of the green dots):

)

Let's break this one down:

1. Get the next prioritized movie or tv show (see next section)


2. Get up-to-date data from TMDB
   


3. Scrape IMDb, Metacritic and Rotten Tomatoes for current scores


4. Scrape TV Tropes for... tropes


5. 
   Huggingface API to gather DNA data (will explain below)


6. Store high dimensional vectors for DNA data
   


7. 
   Store relational data for movies, shows and streaming links

Each of those steps are more or less complex and involve using async processes.

Where do you start? Priority Queue

To determine which titles to pick next there are two lanes that are processed in parallel. This is another area where Windmill shines. Parallelization and orchestration works flawlessly with their architecture.

The two lanes to pick the next item are:

Lane 1: Flows for each data source separately

First of all, titles that don't have any data attached will be selected for each data source. That means if the Metacritic pipeline has a movie that wasn't scraped yet, it will be selected next. This makes sure that every title was processed at least once, including new ones.

Once every title has attached data, the pipeline selects those with the least recent data.

Here is example of such a flow run, here with an error because the rate limit was hit:

and

Windmill Editor

Windmill offers an in-browser IDE-like editor experience with linting, auto-formatting, an AI assistant and even collaborative editing (last one is a paid feature). The best thing is this button though:

It allows two things:

1. Filter by DNA values (using relational data)


2. Search by similarity (using vector data)

Examples:

There will be dedicated blog post about the DNA with many more details in the future.

Deeper Dive into the Data Pipeline

To fully understand how the data pipeline works, here is a breakdown what happens for each data source:

1. Once a day, a MongoDB collection is updated with all required input data

For each data source there is an ìnit flow that prepares a MongoDB collection with all required data. For IMDb, that's just the imdb_id. For Rotten Tomatoes, the title and release_year are required. That's because the ID is unknown and we need to guess the correct URL based on the name.

2. Continuously fetch data and write it into the MongoDB collection

Based on the priority selection explained above, items in the prepared collections are updated with the data that is fetched. Each data source has their own collection which gets more and more complete over time.

3. Once a day, various flows collect the data from the MongoDB collections and write them into Postgres

There is a flow for movies, one for tv shows and another one for streaming links. They collect all necessary data from various collections and store them in their respective Postgres tables, which are then queried by the web application.

Here is an excerpt of the copy movies flow and script:

Here is an excerpt of all schedules that are defined for GoodWatch:

Key Takeaways

Windmill is a great asset in any developer's toolkit for automating tasks. It's been an invaluable productivity booster for me, allowing me to focus on the flow structure and business logic while outsourcing the heavy lifting of task orchestration, error handling, retries and caching.

Handling large volumes of data is still challenging, and optimizing the pipeline is an ongoing process - but I'm really happy with how everything has turned out so far.

Okay, okay. That's enough

Thought so. Just let me link a few resources and we're finished:

Did you know that GoodWatch is open-source? You can take a look at all scripts and flow definitions in this repository: https://github.com/alp82/goodwatch-monorepo/tree/main/goodwatch-flows/windmill/f

Let me know if you have any questions.