Fetching Instagram Data at Scale in Python: From One Request to Async

I collect Instagram data for <私が収集しているもの・用途 例: hashtag research for my content projects>, and at some point "just call the API in a loop" stopped being fast enough. Here's how I went from a single request to concurrent fetching in Python, using HikerAPI — a REST API for Instagram data (auth is one header, from $0.001/request, 100 free requests to start).

The starting point: one request

Everything begins with a plain synchronous call:

import requests



headers = {"x-access-key": "YOUR_KEY"}

r = requests.get(

    "https://api.hikerapi.com/v2/hashtag/medias/top",

    params={"name": "photography"},

    headers=headers,

)

print(r.json())

This returns the top posts for a hashtag as JSON. Simple. But if you have <私のボリューム 例: a few hundred hashtags or usernames> to go through, a sequential loop means each request waits for the previous one to finish. At ~0.5–1s per round trip, hundreds of items turn into many minutes of waiting for what is mostly network idle time.

Option 1: threads (smallest change to your code)

Since this is I/O-bound work, ThreadPoolExecutor gets you most of the win with almost no rewrite:

from concurrent.futures import ThreadPoolExecutor

import requests



headers = {"x-access-key": "YOUR_KEY"}

HASHTAGS = ["photography", "travel", "food"]  # your list here



def fetch_hashtag(name):

    r = requests.get(

        "https://api.hikerapi.com/v2/hashtag/medias/top",

        params={"name": name},

        headers=headers,

        timeout=30,

    )

    r.raise_for_status()

    return name, r.json()



with ThreadPoolExecutor(max_workers=10) as pool:

    for name, data in pool.map(fetch_hashtag, HASHTAGS):

        print(name, len(data))

Ten workers roughly means ten requests in flight at once — a 10x wall-clock speedup on a big list.

Option 2: asyncio + aiohttp (better for large volumes)

For bigger batches I switched to asyncio, mainly because a semaphore makes concurrency control explicit:

import asyncio

import aiohttp



HEADERS = {"x-access-key": "YOUR_KEY"}

CONCURRENCY = 10



async def fetch_hashtag(session, sem, name):

    async with sem:

        async with session.get(

            "https://api.hikerapi.com/v2/hashtag/medias/top",

            params={"name": name},

        ) as r:

            r.raise_for_status()

            return name, await r.json()



async def main(hashtags):

    sem = asyncio.Semaphore(CONCURRENCY)

    async with aiohttp.ClientSession(headers=HEADERS) as session:

        tasks = [fetch_hashtag(session, sem, h) for h in hashtags]

        return await asyncio.gather(*tasks)



results = asyncio.run(main(["photography", "travel", "food"]))

The semaphore is the important part: without it, gather fires everything at once, which brings us to…

Rate limits: don't hammer the API

A few things I do to stay polite and keep runs stable:

• 
  Cap concurrency. I stay around 10 in-flight requests. More than that mostly increases error rates, not throughput.


• 
  Back off on 429/5xx. Retry with exponential backoff instead of failing the whole batch:
  

async def fetch_with_retry(session, sem, name, retries=3):

    for attempt in range(retries):

        try:

            return await fetch_hashtag(session, sem, name)

        except aiohttp.ClientResponseError as e:

            if e.status in (429, 500, 502, 503) and attempt < retries - 1:

                await asyncio.sleep(2 ** attempt)  # 1s, 2s, 4s

                continue

            raise

• 
  Save as you go. At $0.001/request the cost of a batch is small, but re-running a half-failed batch is still wasted money. I write each result to disk (JSONL) as it arrives, so a crash only re-fetches what's missing.

Takeaway

• One-off calls: plain requests is fine.


• Hundreds of items: ThreadPoolExecutor, 10 workers, done in minutes.


• Ongoing/large collection: asyncio + semaphore + retry, writing results incrementally.

The nice thing about having Instagram data behind a plain REST API is that all the usual Python concurrency patterns just work — no session juggling or scraper babysitting.

How do you handle batch collection in your projects — threads or asyncio? And what concurrency do you find APIs generally tolerate?

Disclosure: this post is part of HikerAPI's reward program (rewarded in API credits). The code and workflow are my own.